Multi-label sentiment classification model training method, related device and medium

Abstract

The present disclosure provides a multi-label sentiment classification model training method, related devices and media. The method includes: inputting sample texts in the initial sample text set into multiple multi-label sentiment classification models to obtain multiple maximum probability sentiment classifications corresponding to each multi-label sentiment classification model; based on the first dominant sentiment classification label and the comparison with each maximum probability sentiment classification, screening out matching sample texts; inputting matching sample texts into the multi-label sentiment classification model to obtain a first positive discrimination probability; calculating a first loss function based on the first sentiment classification label and the first positive discrimination probability; training the multi-label sentiment classification model based on the first loss function; replacing the initial sample text set with the matching sample text set, repeating the above steps, and finally obtaining a trained multi-label sentiment classification model. The present disclosure can improve the accuracy of sentiment classification performed by the multi-label sentiment classification model. The present disclosure can be applied to scenarios such as data processing and artificial intelligence.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a multi-label sentiment classification model training method, related devices and media. Background Art

[0002] At present, the emotions reflected in text can be classified through artificial intelligence. Multi-label text classification refers to the simultaneous classification of different emotional aspects of the same text. For example, it can identify whether the same text contains the emotion of happiness, whether it contains the emotion of love, whether it contains the emotion of surprise, etc. It can even identify whether the same text contains a high-intensity emotion of happiness, whether it contains a medium-intensity emotion of surprise, etc.

[0003] In the prior art, there are many methods for multi-label sentiment classification of texts using large models. However, these methods have low sentiment classification accuracy of the trained models due to inaccurate labeling of training samples and a large amount of noise. Summary of the invention

[0004] The disclosed embodiments provide a sentiment classification model training method, a sentiment classification method, a related device and a medium, which can improve the accuracy of sentiment classification performed by a multi-label sentiment classification model.

[0005] According to one aspect of the present disclosure, a multi-label sentiment classification model training method is provided, comprising:

[0006] Inputting sample texts in the initial sample text set into a plurality of the multi-label sentiment classification models to obtain the maximum probability sentiment classification corresponding to each of the multi-label sentiment classification models;

[0007] Based on the comparison between the first dominant sentiment classification label of the sample text and the maximum probability sentiment classification corresponding to each of the multi-label sentiment classification models, matching sample texts are screened out from the initial sample text set, and the screened matching sample texts constitute a matching sample text set;

[0008] Inputting the matching sample text into each of the multi-label sentiment classification models to obtain a first positive discriminant probability corresponding to each of the sentiment classifications;

[0009] Based on the multiple first sentiment classification labels of each of the sentiment classifications and the first positive discriminant probability, calculating a first loss function corresponding to each of the multi-label sentiment classification models;

[0010] For each of the multi-label sentiment classification models, training the multi-label sentiment classification model based on the first loss function corresponding to the multi-label sentiment classification model;

[0011] The initial sample text set is replaced by the matching sample text set, and the step of inputting the sample texts in the initial sample text set into the multiple multi-label sentiment classification models is returned to until the first loss function satisfies a predetermined condition, and then a trained multi-label sentiment classification model is determined from the multiple multi-label sentiment classification models.

[0012] According to one aspect of the present disclosure, a text multi-label sentiment classification method is provided, comprising:

[0013] Get the target text;

[0014] Inputting the target text into a multi-label sentiment classification model, obtaining a first sentiment classification result for each sentiment classification, wherein the first sentiment classification result is used to indicate whether the target text belongs to the sentiment classification;

[0015] The multi-label sentiment classification model is trained by the following process: inputting sample texts in the initial sample text set into each of the multi-label sentiment classification models to obtain the maximum probability sentiment classification corresponding to each of the multi-label sentiment classification models; based on the comparison between the first dominant sentiment classification label of the sample text and the maximum probability sentiment classification corresponding to each of the multi-label sentiment classification models, screening out matching sample texts from the initial sample text set, and the screened matching sample texts constitute a matching sample text set; inputting the matching sample texts into the multi-label sentiment classification model to obtain the first positive discrimination probability corresponding to each sentiment classification; based on each The first sentiment classification label of the sentiment classification, and the first positive discriminant probability, calculate the first loss function corresponding to each of the multi-label sentiment classification models; for each of the multi-label sentiment classification models, train the multi-label sentiment classification model based on the first loss function corresponding to the multi-label sentiment classification model; replace the initial sample text set with the matching sample text set, and return to the step of inputting the sample text in the initial sample text set into multiple multi-label sentiment classification models, until the first loss function satisfies a predetermined condition, and then determine the trained multi-label sentiment classification model from the multiple multi-label sentiment classification models.

[0016] According to one aspect of the present disclosure, a multi-label sentiment classification model training device is provided, comprising:

[0017] A first input unit is used to input the sample text in the initial sample text set into multiple multi-label sentiment classification models to obtain the maximum probability sentiment classification corresponding to each of the multi-label sentiment classification models;

[0018] A cleaning unit is used to screen out matching sample texts from the initial sample text set based on a comparison between the first dominant sentiment classification label of the sample text and the maximum probability sentiment classification corresponding to each multi-label sentiment classification model, and the screened matching sample texts constitute a matching sample text set;

[0019] A second input unit, used to input the matching sample text into each of the multi-label sentiment classification models to obtain a first positive discriminant probability corresponding to each sentiment classification;

[0020] A calculation unit, for calculating a first emotion classification label for each emotion classification and the first forward discriminant probability, and a first loss function corresponding to each multi-label emotion classification model;

[0021] A first training unit is used to train a multi-label sentiment classification model for each of the multi-label sentiment classification models based on a first loss function corresponding to the multi-label sentiment classification model;

[0022] A return unit is used to replace the initial sample text set with the matching sample text set, and return to the step of inputting the sample texts in the initial sample text set into multiple multi-label sentiment classification models until the first loss function meets the predetermined condition, and then determine the trained multi-label sentiment classification model from the multiple multi-label sentiment classification models.

[0023] According to one aspect of the present disclosure, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the multi-label sentiment classification model training method as described above or the text multi-label sentiment classification method as described above is implemented.

[0024] According to one aspect of the present disclosure, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the multi-label sentiment classification model training method or the text multi-label sentiment classification method is implemented.

[0025] According to one aspect of the present disclosure, a computer program product is provided, which includes a computer program, and the computer program is read and executed by a processor of a computer device, so that the computer device executes the multi-label sentiment classification model training method as described above or the text multi-label sentiment classification method as described above.

[0026] The disclosed embodiment trains a multi-label sentiment classification model through multiple rounds of iterations. Each round of iteration includes a process of screening sample texts and adjusting the parameters of the multi-label sentiment classification model using the screened matching sample texts. Specifically, in each round of iteration, the sample texts screened out from the initial sample text set after the previous round of iterations are input into multiple multi-label sentiment classification models. Multiple multi-label sentiment classification models obtain the maximum probability sentiment classifications judged by each. According to the matching of the multiple maximum probability sentiment classifications and the dominant sentiment classification labels of the sample texts, it can be determined which sample texts are accurate and screened out as matching sample texts. In this way, the quality of the samples is improved and the accuracy of the model training is improved. Then, the screened matching sample texts are input into the multi-label sentiment classification model, and the first loss function can be calculated based on the first sentiment classification label of each sentiment classification and the first positive discrimination probability of the sentiment classification judged by the multi-label sentiment classification model, and the parameters of the multi-label sentiment classification model are adjusted based on the first loss function. In each round of iteration, the model training is performed after removing a part of the inaccurate samples, which improves the effect of the model training. And through multiple rounds of iterations, we can gradually test and remove some samples that are believed to be poor in each round. Through this multiple rounds of continuous sample screening and training, the training accuracy is improved, and the classification accuracy of the trained multi-label sentiment classification model is improved.

[0027] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0029] Figure 1 It is a system architecture diagram of a multi-label sentiment classification model training method and a text multi-label sentiment classification method according to an embodiment of the present disclosure;

[0030] Figure 2A-2C is a schematic diagram of an interface in which an embodiment of the present disclosure is applied to a scenario of reading a script in a script reading application;

[0031] Figure 3A-3D is a schematic diagram of an interface in which an embodiment of the present disclosure is applied to a scenario of evaluating a script;

[0032] Figure 4A-4E It is a schematic diagram of an interface in which an embodiment of the present disclosure is applied to a scenario of re-creating a script;

[0033] Figure 5 is a flowchart of a multi-label sentiment classification model training method according to an embodiment of the present disclosure;

[0034] Figure 6 is a schematic diagram of removing sample text according to an embodiment of the present disclosure;

[0035] Figure 7 is a flow chart for determining a trained multi-label sentiment classification model according to an embodiment of the present disclosure;

[0036] Figure 8 is a schematic diagram of determining a trained multi-label sentiment classification model according to an embodiment of the present disclosure;

[0037] Fig. 9A is an overall structural diagram of a multi-label sentiment classification model training method according to an embodiment of the present disclosure;

[0038] Fig. 9B is a schematic diagram of iterative multi-model learning and data cleaning according to an embodiment of the present disclosure;

[0039] Fig. 9C is a schematic diagram of iterative multi-model learning and data cleaning according to another embodiment of the present disclosure;

[0040] Fig.10 According to one embodiment of the present disclosure Figure 5 Flowchart of step 510 in;

[0041] Fig.11A is a schematic diagram of specific components of sentiment classification according to an embodiment of the present disclosure;

[0042] Fig. 11B is a schematic diagram of classifying sample text to obtain the maximum probability sentiment classification according to an embodiment of the present disclosure;

[0043] Fig. 12A is a schematic diagram of specific components of sentiment classification according to another embodiment of the present disclosure;

[0044] Fig. 12B is a schematic diagram of classifying sample text to obtain the maximum probability sentiment classification according to another embodiment of the present disclosure;

[0045] Fig.13 According to one embodiment of the present disclosure Fig.10 Flowchart of step 1020 in FIG.

[0046] Fig.14is a schematic diagram of determining the maximum probability emotion classification based on a discrimination threshold according to an embodiment of the present disclosure;

[0047] Fig.15 is a schematic diagram of obtaining a discrimination threshold for sentiment classification according to an embodiment of the present disclosure;

[0048] Fig.16 is a flowchart of a multi-label sentiment classification model training method according to another embodiment of the present disclosure;

[0049] Fig.17A According to another embodiment of the present disclosure Fig. 9A The local structure diagram in ;

[0050] Fig. 17B is a schematic diagram of dividing a matching sample text set according to an embodiment of the present disclosure;

[0051] Fig.18 According to one embodiment of the present disclosure Fig.16 Flowchart of step 1610 in FIG.

[0052] Fig.19 is a schematic diagram of label division of multiple first emotion classification labels according to an embodiment of the present disclosure;

[0053] Fig. 20 According to one embodiment of the present disclosure Fig.16 Flowchart of step 1630 in;

[0054] Fig.21 According to one embodiment of the present disclosure Fig.16 Flowchart of step 1640 in;

[0055] Fig. 22 According to one embodiment of the present disclosure Fig.21 Flowchart of step 2110 in;

[0056] Fig.23 According to one embodiment of the present disclosure Fig.21 Flowchart of step 2120 in;

[0057] Fig.24 According to one embodiment of the present disclosure Fig.21 Flowchart of step 2130 in;

[0058] Fig.25 According to one embodiment of the present disclosure Fig.21 Flowchart of step 2140 in;

[0059] Fig.26 According to one embodiment of the present disclosure Fig.21Flowchart of step 2150 in;

[0060] Fig. 27 According to one embodiment of the present disclosure Fig.21 Flowchart of step 2160 in;

[0061] Fig.28 is a flowchart of a text multi-label sentiment classification method according to an embodiment of the present disclosure;

[0062] Fig.29 According to one embodiment of the present disclosure Fig.28 Flowchart of step 2820 in;

[0063] Fig. 30A It is an overall structural diagram of a text multi-label sentiment classification method according to an embodiment of the present disclosure;

[0064] Fig. 30B is a schematic diagram of inputting a high-order sentiment feature vector into a classification layer according to an embodiment of the present disclosure;

[0065] Fig.31 According to one embodiment of the present disclosure Fig.29 Flowchart of step 2910 in;

[0066] Fig.32 is a schematic diagram of converting a target text into a target text vector according to an embodiment of the present disclosure;

[0067] Fig.33 According to one embodiment of the present disclosure Fig.29 Flowchart of step 2920 in;

[0068] Fig.34 is a schematic diagram of the structure of a basic learning model according to an embodiment of the present disclosure;

[0069] Fig.35A is a schematic diagram of determining a basic semantic feature vector based on a basic learning model according to an embodiment of the present disclosure;

[0070] Fig.35B is a schematic diagram of determining an output sub-vector based on a first transformation sub-model according to an embodiment of the present disclosure;

[0071] Fig.35C is a schematic diagram of the structure of a multi-head attention model of an embodiment of the present disclosure;

[0072] Fig.36 is a schematic diagram of determining a basic semantic feature vector based on a basic learning model according to another embodiment of the present disclosure;

[0073] Fig.37 According to one embodiment of the present disclosure Fig.29 Flowchart of step 2930 in;

[0074] Fig.38 is a schematic diagram of the structure of an emotion detail learning model according to an embodiment of the present disclosure;

[0075] Fig.39 is a schematic diagram of determining a high-order emotion feature vector based on an emotion detail learning model according to an embodiment of the present disclosure;

[0076] Fig.40 is a flowchart of a text multi-label sentiment classification method according to another embodiment of the present disclosure;

[0077] Fig.41 is a schematic diagram of a second sentiment classification result according to an embodiment of the present disclosure;

[0078] Fig.42 is a schematic diagram of determining a third emotion classification result according to an embodiment of the present disclosure;

[0079] Fig.43 is a module diagram of a multi-label sentiment classification model training device according to an embodiment of the present disclosure;

[0080] Fig.44 is a module diagram of a text multi-label sentiment classification device according to an embodiment of the present disclosure;

[0081] Fig.45 It is a terminal structure diagram for executing a multi-label sentiment classification model training method and a text multi-label sentiment classification method according to an embodiment of the present disclosure;

[0082] Fig.46 It is a server structure diagram for executing a multi-label sentiment classification model training method and a text multi-label sentiment classification method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0084] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0085] Embedding layer: A word encoding technology specifically used in the field of natural language processing, the purpose is to reduce the waste of resources. It uses low-dimensional vectors to represent the encoding, which can express the relationship between words through neural network optimization.

[0086] Deep neural network: It is a multi-layer neural network. Deep neural network uses the output features of the previous layer as the input of the next layer for feature learning. After layer-by-layer feature mapping, the features of the existing spatial samples are mapped to another feature space, so as to learn to have better feature expression for the existing input. Deep neural network has multiple nonlinear mapping feature transformations and can fit highly complex functions.

[0087] Gradient descent algorithm: A method for minimizing the loss function. It calculates the gradient of the loss function at the current position and updates the parameters in the direction of gradient descent to find the optimal solution that minimizes the value of the loss function.

[0088] System architecture and scenario description of the application of the embodiments of the present disclosure

[0089] Figure 1 It is a system architecture diagram used by the multi-label sentiment classification model training method and the text multi-label sentiment classification method according to the embodiment of the present disclosure. It includes a terminal 140, the Internet 130, a gateway 120, and a server 110.

[0090] The terminal 140 is a device used to perform sentiment classification model training tasks and sentiment classification tasks. The terminal 140 can also be a device that sends sentiment classification model training requests and sentiment classification requests to the server 110 so that the server 110 can perform sentiment classification model training tasks and sentiment classification tasks. It includes various forms such as desktop computers, laptops, tablet computers, PDAs (personal digital assistants), mobile phones, vehicle-mounted terminals, smart home appliances, etc. In addition, it can be a single device or a collection of multiple devices. For example, multiple devices are connected through a local area network and share a display device to work together to form a terminal. The terminal 140 can also communicate with the Internet 130 in a wired or wireless manner to exchange data.

[0091] The gateway 120 is also called an internetwork connector or a protocol converter. The gateway 120 realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway 120 is a translator between two systems that use different communication protocols, data formats or languages, or even have completely different architectures. At the same time, the gateway 120 can also provide filtering and security functions. The message sent by the terminal 140 to the server 110 must be sent to the corresponding server 110 through the gateway 120. The message sent by the server 110 to the terminal 140 must also be sent to the corresponding terminal 140 through the gateway 120.

[0092] The server 110 refers to a computer system that can provide sentiment classification model training services and sentiment classification services to the terminal 140. Compared with the terminal 140, the server 110 has higher requirements in terms of stability, security, performance, etc. The server 110 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc. The server 110 can also communicate with the Internet 130 in a wired or wireless manner to exchange data.

[0093] The embodiments of the present disclosure can be applied in various scenarios, such as Figure 2A-2C The scene of reading a script in the script reading application shown, Figure 3A-3D The scene showing the evaluation of the script, Figure 4A-4E The scenes shown are re-creations of the script, etc.

[0094] (I) Scenarios for reading scripts in script reading applications

[0095] Emotional analysis of the characters based on the scene descriptions and lines in the script can assist in understanding the script. For example, in a scene, when the heroine is more sad, it means that she has encountered a setback, which indicates that the character has reached a certain low point. This scene is a key scene or needs to be understood. Another example is that in a scene, the heroine has strong love and other sweet emotions, which indicates that the character in this scene has reached an emotional peak. Emotional analysis before and after this scene can depict the character's change process from the low point to the peak, so as to grasp the emotional development of each character in the script.

[0096] like Figure 2A As shown, terminal 140 is a desktop computer. The script reading application can be used to read multiple scripts. After selecting the script "XX Biography" to read, the "Details" option and the "Catalog" option of "XX Biography" are displayed. Among them, the catalog of "XX Biography" includes multiple episodes, and the current page displays episodes 1 to 8. The title of episode 1 is "Daily Life", the title of episode 2 is "Chatting", and the title of episode 3 is "Go Take Photos". The titles of episodes 4 to 8 are replaced by "XX". After the subject triggers "Go Take Photos", the interface of the script reading application is as follows Figure 2B shown.

[0097] exist Figure 2B In the example, the script reading application opens the reading interface and displays the text content of episode 3. In the text content, there are lines from characters A, B, C, and D. The subject may need to infer the emotions of each character from their lines. After the subject triggers the "emotion annotation" option, the interface of the script reading application is as follows: Figure 2C shown.

[0098] It should be understood that after the "emotion tagging" option is triggered, the terminal 140 or the server 110 can provide an emotion classification service to obtain the emotion of each character's lines.

[0099] exist Figure 2C In, with Figure 2B The difference is that the script reading application not only displays the text content of episode 3 in the reading interface, but also displays the emotions of each character's lines. At the beginning, the emotion of character A is faith, the emotion of character B is surprise, the emotion of character C is doubt, and the emotion of character D is worry. As the plot develops, the emotion of character A is period, the emotion of character B is joy, the emotion of character C is faith, and the emotion of character D is nothing. It can be seen that in episode 3, the emotion of character A changes from "faith" to "expectation", the emotion of character B changes from "surprise" to "joy", the emotion of character C changes from "doubt" to "faith", and the emotion of character D changes from "worry" to "nothing". Through emotion annotation, you can grasp the emotional development of each character in the script.

[0100] (II) Scenarios for evaluating scripts

[0101] Based on the emotional development of the characters in the script, it is possible to demonstrate and evaluate whether the emotional experience is attractive enough, which is helpful for the script reviewer to make a quick evaluation of the script. It is also possible to evaluate the emotional expression and plot development of the script by analyzing the emotional concentration in the script, thereby providing a more accurate script evaluation to determine whether it needs to be invested in the subsequent filming of the film and television drama.

[0102] like Figure 3A As shown, Figure 2A The difference is that the "emotional development" option and the "emotional trend" option are also shown. After the subject triggers the "emotional development" option, the interface of the script reading application is as follows Figure 3B shown.

[0103] exist Figure 3B In, with Figure 3A The difference is that the emotional development of each episode is also shown. The emotional development of episode 1 is "none", the emotional development of episode 2 is "none→joy", and the emotional development of episode 3 is "joy→surprise→expectation". The emotions of episodes 4 to 8 are replaced by "XX".

[0104] Figure 3BThe emotional development of each episode is roughly shown in the figure. Considering that an episode includes multiple scenes and multiple characters, a more fine-grained emotional analysis is needed, that is, emotional trend analysis of multiple scenes in each episode and emotional trend analysis of multiple episodes. Emotional trend refers to the degree of positive or negative of each emotional classification. For example, the emotional levels of "joy" with high, medium, and low intensity are 3, 2, and 1 respectively. For another example, the emotional levels of "concern" with high, medium, and low intensity are -3, -2, and -1. After the subject triggers the "emotional trend" option, the interface of the script reading application is as follows Figure 3C or Figure 3D shown.

[0105] exist Figure 3C There are 10 scenes in episode XX. Among them, the emotional level of scene 1 is -1, the emotional level of scene 2 is -3, the emotional level of scene 3 is -2, the emotional level of scene 4 is 1, the emotional level of scene 5 is 1, the emotional level of scene 6 is 3, the emotional level of scene 7 is -1, the emotional level of scene 8 is -2, the emotional level of scene 9 is -3, and the emotional level of scene 10 is -3. Based on the emotional levels of the 10 scenes, the emotional trend can be drawn, and the emotional trend of episode XX can be seen more intuitively.

[0106] It should be understood that an episode contains multiple scenes, and the emotional level of the episode can be determined based on the emotional levels of the multiple scenes. The emotional level that appears most frequently in the multiple scenes can be determined as the emotional level of the episode. Figure 3C The emotional level of the XXth episode is -3. The emotional level of multiple episodes can also be determined as the emotional level of the episode by weighted average. Figure 3C The sentiment rating for episode XX is shown as -1.

[0107] exist Figure 3D There are 12 episodes of "XX Biography". Among them, the emotional level of episode 1 is -1, the emotional level of episode 2 is 2, the emotional level of episode 3 is -2, the emotional level of episode 4 is 1, the emotional level of episode 5 is -3, the emotional level of episode 6 is -1, the emotional level of episode 7 is 1, the emotional level of episode 8 is -2, the emotional level of episode 9 is -3, the emotional level of episode 10 is -2, the emotional level of episode 11 is 1, and the emotional level of episode 12 is 3. Based on the emotional levels of the 12 episodes, the emotional trend can be drawn, and the emotional trend of the episodes of "XX Biography" can be seen more intuitively.

[0108] Based on the emotional development of the XX episode and the emotional development of each episode, we can evaluate "XX Biography". For example, the attractiveness evaluation level of "XX Biography" is medium.

[0109] (III) Scenes of Re-creation of the Script

[0110] Writers and screenwriters can use emotional development to evaluate whether the emotional expression of the script they created meets their expectations. Adjusting the arrangement of the plot based on emotional development can ensure that there are interesting dramatic emotions in each episode or every few scenes, and can also grasp the emotional climax of the overall creation.

[0111] like Figure 4A As shown, Figure 4A and Figure 2A The difference is that, because the object triggers the "Details" option, the interface of the script reading application also shows the details of "XX Biography". Specifically, the attractiveness evaluation level of "XX Biography" is medium, the emotional diversity is low, and the emotional conflict is medium.

[0112] like Figure 4B As shown, after viewing the above details, the subject (such as a writer or screenwriter) chooses to trigger the "script re-creation" option.

[0113] like Figure 4C As shown, after the subject triggers the "script re-creation" option, the script reading application interface displays the emotional development trend of each episode, and displays a prompt message "The emotional development trend of each episode is as follows, please select the episode to re-create". After viewing the emotional development trend of each episode, the subject chooses to re-create the 6th episode.

[0114] like Figure 4D As shown, after the subject selects Episode 6, the interface of the script reading application displays the content and emotion annotation of the original Episode 6, and the content and emotion annotation of the new Episode 6. After the creation is completed, the "Confirm Creation Completed" option can be triggered.

[0115] like Figure 4E As shown, after the subject triggers the "Confirm Creation Completed" option, the screen of the script reading application about details is returned. Since the script has been re-created, the attractiveness evaluation level of "XX Biography" is high, the emotional diversity is high, and the emotional conflict is high.

[0116] In the prior art, there are many methods for multi-label sentiment classification of texts using large models. However, these methods have low sentiment classification accuracy of the trained models due to inaccurate training sample annotations and a large amount of noise. If the sentiment classification of the model is inaccurate, the accuracy of the above-mentioned character sentiment analysis, script evaluation, and script re-creation will all be low. Therefore, it is extremely necessary to improve the accuracy of the sentiment classification model.

[0117] General description of the disclosed embodiments

[0118] According to an embodiment of the present disclosure, a multi-label sentiment classification model training method is provided.

[0119] The multi-label sentiment classification model training method refers to a method for training a multi-label sentiment classification model. The multi-label sentiment classification model refers to a model used to perform multiple sentiment classifications on a text and output multiple labels of whether it belongs to a sentiment classification. Sentiment classification refers to a type of emotion, such as joy, surprise, and expectation. For example, assume that the text is "The dog is cute" and assume that the sentiment classification includes joy, surprise, and expectation, a total of 3 types. The multi-label sentiment classification model performs sentiment classification on "The dog is cute" and outputs (joy, 1), (surprise, 1), (expectation, 0), a total of 3 labels of whether it belongs to the sentiment classification. Among them, (joy, 1) means that the label belonging to joy is 1, that is, the sentiment classification of "The dog is cute" includes joy. (Surprise, 1) means that the label belonging to surprise is 1, that is, the sentiment classification of "The dog is cute" includes surprise. (Expectation, 0) means that the label belonging to expectation is 0, that is, the sentiment classification of "The dog is cute" does not include expectation.

[0120] The multi-label sentiment classification model training method of the present disclosure embodiment can be executed by an electronic device, which can be Figure 1 The terminal 140 or the server 110 shown, that is, the method can be executed by the terminal 140 or the server 110 alone, or can be executed jointly by the terminal 140 and the server 110.

[0121] like Figure 5 As shown, according to one embodiment of the present disclosure, a multi-label sentiment classification model training method includes:

[0122] Step 510: input the sample texts in the initial sample text set into multiple multi-label sentiment classification models to obtain the maximum probability sentiment classification corresponding to each multi-label sentiment classification model;

[0123] Step 520: based on the comparison between the first dominant sentiment classification label of the sample text and the maximum probability sentiment classification corresponding to each multi-label sentiment classification model, select matching sample texts from the initial sample text set, and the selected matching sample texts constitute the matching sample text set;

[0124] Step 530: input the matching sample text into each multi-label sentiment classification model to obtain a first positive discriminant probability corresponding to each sentiment classification;

[0125] Step 540: Calculate a first loss function corresponding to each multi-label sentiment classification model based on the first sentiment classification label of each sentiment classification and the first positive discriminant probability;

[0126] Step 550: for each multi-label sentiment classification model, training the multi-label sentiment classification model based on a first loss function corresponding to the multi-label sentiment classification model;

[0127] Step 560, replace the initial sample text set with the matching sample text set, and return to the step of inputting the sample texts in the initial sample text set into multiple multi-label sentiment classification models until the first loss function meets the predetermined conditions, and then determine the trained multi-label sentiment classification model from the multiple multi-label sentiment classification models.

[0128] The above steps 510 to 560 are described in general below.

[0129] In step 510, the initial sample text set is a set for storing sample texts. The initial sample text set includes multiple sample texts. Sample text refers to text used as a sample for training a model. For the sample text, each multi-label sentiment classification model in multiple multi-label sentiment classification models can be used to obtain the maximum probability sentiment classification corresponding to each multi-label sentiment classification model. Referring to the above, the multi-label sentiment classification model refers to a model for performing multiple sentiment classifications on text and outputting multiple labels of whether it belongs to the sentiment classification. For each sentiment classification in multiple sentiment classifications, the multi-label sentiment classification model can also output the probability of belonging to the sentiment classification. The maximum probability sentiment classification is the sentiment classification corresponding to the maximum probability among multiple probabilities belonging to a certain sentiment classification. For example, in combination with the above example, assume that the text is "The dog is cute" and assume that the sentiment classification includes joy, surprise, and expectation, a total of 3 types. The multi-label sentiment classification model performs sentiment classification on "The dog is cute" and outputs (joy, 0.8), (surprise, 0.6), (expectation, 0.3), a total of 3 probabilities of whether it belongs to the sentiment classification. Since 0.8>0.6>0.3, “happy” is determined as the emotion classification with the highest probability.

[0130] It should be understood that the present embodiment sets a plurality of multi-label sentiment classification models, and each multi-label sentiment classification model corresponds to a maximum probability sentiment classification. For example, if there are a total of 3 multi-label sentiment classification models, then for a sample text, there are a total of 3 maximum probability sentiment classifications. The maximum probability sentiment classifications corresponding to any two multi-label sentiment classification models may be the same or may be different.

[0131] After obtaining the maximum probability sentiment classification, execute step 520. The first dominant sentiment classification label refers to the first sentiment classification label that plays a dominant role among multiple first sentiment classification labels. For example, the first sentiment classification labels of the sample "The dog is cute" include (joy, 1), (surprise, 0), and (expectation, 0). Since only the label of "joy" is 1, the first dominant sentiment classification label is "joy". The first dominant sentiment classification label can be obtained by sentimentally annotating the sample text in advance. The sample text can be sentimentally annotated manually. The sample text can also be sentimentally annotated using a deep learning network. Whether it is manual annotation or model annotation, there may be annotation errors, that is, the sample has noise. Therefore, the embodiment of the present disclosure proposes to screen the sample text based on the comparison of the maximum probability sentiment classification output by the first dominant sentiment classification label and multiple models. In this way, the quality of the sample is improved, thereby improving the accuracy of model training.

[0132] It should be understood that the first dominant sentiment classification label may have a labeling error, and similarly, the maximum probability sentiment classification may also have a prediction error. Therefore, the embodiment of the present disclosure compares multiple maximum probability sentiment classifications with the first dominant sentiment classification label, and screens the sample text based on the comparison result.

[0133] In one embodiment, step 520 includes:

[0134] For the sample text, determine a first dominant sentiment classification label that is consistent with a maximum probability sentiment classification with a first ratio or above among multiple maximum probability sentiment classifications;

[0135] The consistent sample texts are selected from the initial sample text set as matching sample texts.

[0136] The first ratio is a ratio set in advance. If there is no consistency above the first ratio, it means that there is a labeling error in the first dominant sentiment classification label of the sample text, or it means that there is a prediction error in the maximum probability sentiment classification output by the multi-label sentiment classification model. Regardless of the error in either case, the sample text is not suitable for training the multi-label sentiment classification model and the sample text should be removed. Therefore, this embodiment uses the consistent sample text as the matching sample text only when it is determined that there is a consistency above the first ratio.

[0137] like Figure 6As shown, in one example, there are 3 multi-label sentiment classification models, denoted as MDL1, MDL2, and MDL3. The sample text is input into MDL1, MDL2, and MDL3 for sentiment classification. MDL1 outputs the maximum probability sentiment classification of the sample text as "surprise", MDL2 outputs the maximum probability sentiment classification of the sample text as "surprise", and MDL3 outputs the maximum probability sentiment classification of the sample text as "none". The first dominant sentiment classification label of the sample text is "surprise". After comparison, it is found that the maximum probability sentiment classification output by MDL1 and MDL2 is consistent with the first dominant sentiment classification label, and the maximum probability sentiment classification output by MDL3 is inconsistent with the first dominant sentiment classification label, that is, the consistency rate is 2 / 3. Assume that the first ratio is 0.5. Since 2 / 3 is greater than 0.5, the sample text is used as a matching sample text. The screened matching sample texts constitute the matching sample text set.

[0138] It is understandable that, for a certain sample text, if in multiple maximum probability sentiment classifications, the number of labels that are identical to the first dominant sentiment classification label is small, or the inconsistency rate is greater than the first ratio, then the sample text cannot be used as a matching sample text. On the contrary, for a certain sample text, if in multiple maximum probability sentiment classifications, the number of labels that are identical to the first dominant sentiment classification label is large, or the consistency rate is greater than the first ratio, then the sample text can be used as a matching sample text. The matching sample text set obtained after executing step 520 reduces the noise / labeling errors in the sample compared to the initial sample text set.

[0139] After obtaining the matching sample text set, execute step 530. Utilize each multi-label sentiment classification model to obtain the first positive discriminant probability corresponding to each sentiment classification. The first positive discriminant probability refers to the probability that the model discriminates that the sample text contains a certain sentiment classification. For example, combining the above example, the text is "The dog is cute", and the sentiment classification includes joy, surprise, and expectation, a total of 3 types. The multi-label sentiment classification model performs sentiment classification on "The dog is cute" and outputs (joy, 0.8), (surprise, 0.6), and (expectation, 0.3). For the sentiment classification of "joy", the first positive probability is 0.8. For the sentiment classification of "surprise", the first positive probability is 0.6. For the sentiment classification of "expectation", the first positive discriminant probability is 0.3.

[0140] After obtaining the first positive discriminant probability corresponding to each sentiment classification, execute step 540. The first sentiment classification label of each sentiment classification of the sample text can represent the sentiment classification that the sample text annotated in advance has or does not have, that is, the probability of the existence of a certain sentiment classification is 1 or 0. The first positive discriminant probability corresponding to each sentiment classification can represent the probability that the model predicts that the sample text has a certain sentiment classification, and the first positive discriminant probability is generally between 0 and 1. The first loss function can be calculated between the first sentiment classification label and the first positive discriminant probability. The first loss function can represent the gap between the first sentiment classification label and the first positive discriminant probability. If the gap is smaller, it means that the sentiment classification predicted by the model is more similar / identical to the sentiment classification annotated in advance.

[0141] It should be understood that the present embodiment sets a plurality of multi-label sentiment classification models, and each multi-label sentiment classification model corresponds to a first loss function. For example, if there are three multi-label sentiment classification models, then for a sample text, there are three first loss functions. The first loss functions corresponding to any two multi-label sentiment classification models may be the same or different.

[0142] In one embodiment, step 540 includes:

[0143] For each sentiment classification, based on the first sentiment classification label of the sentiment classification of the sample text and the first positive discriminant probability corresponding to the sentiment classification, calculate the first loss score function of the sentiment classification of the sample text;

[0144] Based on the first loss sub-functions of multiple sentiment classifications of the sample text, calculate the first loss sub-function of the sample text;

[0145] Based on the first loss sub-function of the plurality of sample texts, a first loss function is calculated.

[0146] The first loss score function represents the difference between the first sentiment classification label and the first positive discriminant probability. The smaller the first loss score function, the smaller the difference, which means that the first positive discriminant probability of a single sentiment classification predicted by the multi-label sentiment classification model is more accurate. The first loss sub-function represents the average or weighted average of multiple first loss score functions of a single sample text. If the first loss sub-function is smaller, it means that the multi-label sentiment classification model predicts the first positive discriminant probabilities of multiple sentiment classifications of the sample text more accurately. The first loss function represents the average or weighted average of the first loss sub-functions of multiple sample texts. If the first loss function is smaller, it means that the multi-label sentiment classification model predicts the first positive discriminant probabilities of multiple sentiment classifications of multiple sample texts more accurately.

[0147] In this embodiment, the calculation process of determining the first loss function is shown in Formula 1:

[0148]

[0149] In formula 1, L tag Represents the first loss function. bs indicates that there are bs sample texts in a batch. The initial sample text set has N sample texts in total, and one round of iteration requires N / bs batches. i represents the i-th sample text. j represents the j-th sentiment classification. M represents that there are M sentiment classifications in total. t[i] j Indicates the first sentiment classification label of the jth sentiment classification of the i-th sample text. o[i] j It represents the first positive discriminant probability of the jth sentiment classification of the i-th sample text. When the first sentiment classification label of a certain sentiment classification has a true value of 1, the left side of the plus sign of formula 1 takes effect, and when the true value is 0, the right side of formula 1 takes effect, so the information of a certain sample text under all labels can be learned.

[0150] The advantage of the above embodiment is that by first calculating the first loss sub-function of a single sentiment classification based on the first sentiment classification label and the first positive discriminant probability corresponding to the sentiment classification, then calculating the first loss sub-functions of multiple sentiment classifications in a single sample text, and finally calculating the first loss function of multiple samples, the calculation complexity can be reduced and the calculation efficiency can be improved while ensuring the accuracy of the first loss function.

[0151] After obtaining the first loss function, execute step 550. The purpose of training is to reduce the first loss function, and a reduction in the first loss function represents a more accurate prediction result. Therefore, the training process is to update the parameters in the multi-label sentiment classification model so that the updated model can reduce the first loss function finally generated. The gradient descent method can be used to train the multi-label sentiment classification model based on the first loss function, so that the parameters of the multi-label sentiment classification model can be updated in the direction of the gradient descent of the first loss function.

[0152] After step 550, execute step 560. In step 560, replace the initial sample text set with the matching sample text set, and return to the step of inputting the sample texts in the initial sample text set into multiple multi-label sentiment classification models until the first loss function satisfies the predetermined condition, and then determine the trained multi-label sentiment classification model from the multiple multi-label sentiment classification models.

[0153] It should be noted that in step 550, if the first loss function no longer decreases or is less than a predetermined threshold value in one round of iteration, the updating of the parameters in the multi-label sentiment classification model is stopped, and the current round of model learning is completed. However, in step 560, if it is necessary to stop iterative model learning, in addition to ensuring that the first loss function of this round no longer decreases or is less than a predetermined threshold value, the first loss function must also meet a predetermined condition. The predetermined condition is a condition set in advance to stop iterative model learning.

[0154] In one embodiment, the predetermined condition is that the first loss function of the current iteration is greater than or equal to the first loss function of the previous iteration, then step 560 includes replacing the initial sample text set with the matching sample text set, and returning to the step of inputting the sample texts in the initial sample text set into multiple multi-label sentiment classification models until the first loss function of the current iteration is greater than or equal to the first loss function of the previous iteration.

[0155] In another embodiment, if the first loss function of the current round iteration is greater than or equal to the weighted average of the first loss functions from the starting round iteration to the previous round iteration, step 560 includes replacing the initial sample text set with the matching sample text set, and returning to the step of inputting the sample texts in the initial sample text set into multiple multi-label sentiment classification models until the first loss function of the current round iteration is greater than or equal to the average of the first loss functions from the starting round iteration to the previous round iteration.

[0156] The above steps 510 to 560 constitute one round of iteration in iterative multi-model learning and data cleaning. Steps 510 to 560 may be repeated multiple times until the first loss function satisfies a predetermined condition, and a trained multi-label sentiment classification model is determined from multiple multi-label sentiment classification models.

[0157] The trained multi-label sentiment classification model is one of the multiple multi-label sentiment classification models. In one embodiment, one of the multiple multi-label sentiment classification models is arbitrarily selected as the trained multi-label sentiment classification model. In another embodiment, the multi-label sentiment classification models are sorted based on their accuracy, and the multi-label sentiment classification model with the highest accuracy is used as the trained multi-label sentiment classification model.

[0158] In one embodiment, referring to Figure 7 , determining the trained multi-label sentiment classification model from multiple multi-label sentiment classification models includes:

[0159] Step 710: Obtain a third test text set, where the third test text includes a plurality of third test texts, and the third test text has a second dominant sentiment classification label;

[0160] Step 720: input the third test text into each multi-label sentiment classification model respectively to obtain the maximum probability sentiment classification result of the multi-label sentiment classification model;

[0161] Step 730: Determine a second accuracy rate of the multi-label sentiment classification model based on a comparison between the second dominant sentiment classification label and the maximum probability sentiment classification result of the multi-label sentiment classification model;

[0162] Step 740: Determine a trained multi-label sentiment classification model based on the second accuracy of each multi-label sentiment classification model.

[0163] In step 710, the third test text set is a set for storing the third test text. The third test text is a text used as a test model. The third test text may have multiple sentiment classification labels. The second dominant sentiment classification label refers to the label that plays a dominant role among the multiple labels of the text. For example, the third text is "I am very happy". The sentiment classification labels of the text include "Surprise 3", "Like 2", etc. The second active sentiment classification label of the text is "Surprise 3".

[0164] In step 720, the maximum probability sentiment classification result is used to indicate whether the third test text belongs to the sentiment classification. For example, the maximum probability sentiment classification result is (love, 1), indicating that the third test text belongs to love. For another example, the maximum probability sentiment classification result is (period, 0), indicating that the third test text does not belong to period.

[0165] It should be understood that after the third test text is input into the multi-label sentiment classification model, the multi-label sentiment classification model performs sentiment classification on the third test text. The specific process is the same as the process of inputting the sample text into the multi-label sentiment classification model in step 510, and will not be repeated here.

[0166] In step 730, for each sentiment classification label, if the maximum probability sentiment classification result is accurate, the maximum probability sentiment classification result is the same as the second dominant sentiment classification label. If the maximum probability sentiment classification result is wrong, the maximum probability sentiment classification result is different from the second dominant sentiment classification label. The second accuracy of the multi-label sentiment classification model can be determined based on the number of identical texts and the number of different texts in the comparison results.

[0167] In step 740, the plurality of multi-label sentiment classification models may be sorted based on the second accuracy, and the multi-label sentiment classification model with the largest second accuracy may be used as the trained multi-label sentiment classification model. The multi-label sentiment classification model with the second accuracy greater than the accuracy threshold may be used as the trained multi-label sentiment classification model.

[0168] The advantage of the embodiment of steps 710-740 is that the second accuracy rate is determined by comparing the maximum probability sentiment classification model output by the model for the third test text set with the second dominant sentiment classification label, without having to process the sentiment classification results corresponding to all sentiment classifications. This reduces the computational complexity while ensuring the accuracy of calculating the second accuracy rate.

[0169] In one embodiment, step 730 includes:

[0170] determining the number of the third test texts in the third test text set as the seventh number;

[0171] The number of third test texts having the second dominant sentiment classification label in the third test text set and being consistent with the maximum probability sentiment classification result of the multi-label sentiment classification model is determined as an eighth number;

[0172] Based on the seventh number and the eighth number, a second accuracy rate is determined.

[0173] In this embodiment, the seventh number refers to the total number of third test texts included in the third test text set. The eighth number refers to the total number of third test texts in which the second dominant sentiment classification label appears consistent with the maximum probability sentiment classification result. The seventh number and the eighth number are used to determine the second accuracy rate. For example, the ratio of the eighth number to the seventh number is determined as the second accuracy rate.

[0174] like Figure 8 As shown, in one example, the third test text set includes 1000 third test texts, that is, the seventh number is 1000. There are 3 multi-label sentiment classification models, denoted as MDL1, MDL2, and MDL3. Each third test text is input into MDL1, MDL2, and MDL3 respectively. MDL1 outputs the maximum probability sentiment classification result of 900 third test texts as "Shock 1", that is, the eighth number is 900. The second accuracy of MDL1 is 900 / 1000=0.9. MDL2 outputs the maximum probability sentiment classification result of 600 third test texts as "Shock 1", that is, the eighth number is 600. The second accuracy of MDL2 is 600 / 1000=0.6. MDL3 outputs the maximum probability sentiment classification result of 700 third test texts as "Shock 1", that is, the eighth number is 700. The second accuracy of MDL3 is 700 / 1000=0.7. Since 0.9>0.7>0.6, MDL1 is used as the trained multi-label sentiment classification model.

[0175] Fig. 9A It is an overall structural diagram of a multi-label sentiment classification model training method according to an embodiment of the present disclosure.

[0176] exist Fig. 9AThe leftmost part is the initial sample text set in step 510. The initial sample text set is input into multiple multi-label sentiment classification models to perform iterative multi-model learning and data cleaning in steps 520-560. After completing the iterative multi-model learning and data cleaning, model screening is performed to obtain a model screening result. Model screening refers to selecting a multi-label sentiment classification model from multiple multi-label sentiment classification models as a trained multi-label sentiment classification model. In actual use, the sentiment classification of text using the trained multi-label sentiment classification model has a higher accuracy.

[0177] exist Fig. 9A In the present invention, the multi-label sentiment classification model training method also includes the processes of model prediction and label stratification, soft label generation, partial label correction, model-oriented soft label learning, and precise label learning. These processes will be described in conjunction with specific embodiments in the following text and are omitted here.

[0178] Fig. 9B is a schematic diagram of iterative multi-model learning and data cleaning according to an embodiment of the present disclosure. Fig. 9B In the example, there are three multi-label sentiment classification models, denoted as MDL1, MDL2, and MDL3. Before starting iterative multi-model learning and data cleaning, the sample text of the initial sample text set is input into MDL1, MDL2, and MDL3 to obtain the first positive discriminant probability corresponding to each sentiment classification output by MDL1, the first positive discriminant probability corresponding to each sentiment classification output by MDL2, and the first positive discriminant probability corresponding to each sentiment classification output by MDL3. The first loss function ( Fig. 9B The Loss corresponding to MDL1 shown in FIG. 1 is used to train MDL1 based on the first loss function. The first loss function is calculated based on the first positive discriminant probability corresponding to each sentiment classification output by MDL2 and the first dominant sentiment classification label of the sample text. Fig. 9B The corresponding Loss of MDL2 is shown in FIG. 1 ). MDL2 is trained based on the first loss function. The first loss function is calculated based on the first positive discriminant probability corresponding to each sentiment classification output by MDL3 and the first dominant sentiment classification label of the sample text ( Fig. 9B The corresponding Loss of MDL3 is shown in FIG. 1 ). MDL3 is trained based on the first loss function.

[0179] In each round of iterative multi-model learning and data cleaning, the sample text in the initial sample text set is input into MDL1, MDL2, and MDL3 to obtain the maximum probability sentiment classification corresponding to MDL1, the maximum probability sentiment classification corresponding to MDL2, and the maximum probability sentiment classification corresponding to MDL3. Based on the comparison of the first dominant sentiment classification label of the sample text, the maximum probability sentiment classification corresponding to MDL1, the maximum probability sentiment classification corresponding to MDL2, and the maximum probability sentiment classification corresponding to MDL3, the matching sample text is screened out, and the screened matching sample text constitutes the matching sample text set. MDL1, MDL2, and MDL3 are trained based on the matching sample text set (based on Fig. 9B The parameters of the Loss update model are shown). The initial sample text set is replaced by the matching sample text set, and the step of inputting the sample text in the initial sample text set into each multi-label sentiment classification model is returned until the first loss function satisfies the predetermined condition, and the trained multi-label sentiment classification model is determined from the multiple multi-label sentiment classification models.

[0180] Fig. 9C It is a schematic diagram of iterative multi-model learning and data cleaning according to another embodiment of the present disclosure. Fig. 9C and Fig. 9B The difference is that the model structure of the multi-label sentiment classification model is also shown.

[0181] like Fig. 9C As shown, the multi-label sentiment classification model includes an embedding layer, a basic learning model, a sentiment detail learning model, and a classification layer. When the multi-label sentiment classification model is trained based on the first loss function, the basic learning model, the sentiment detail learning model, and the classification layer are specifically trained based on the first loss function, and there is no need to train the embedding layer. Fig. 9C The embodiments shown will be described in detail in the following content, and only a brief description is given here. Fig. 9B The differences between the embodiments shown.

[0182] The advantage of the embodiment of steps 510-560 is that in each round of iteration, model training is performed after removing a portion of inaccurate samples, thereby improving the effect of model training. In addition, multiple rounds of iterations can be used to gradually test and remove some samples that are believed to be relatively poor in each round. Through multiple rounds of continuous cleaning and training of samples, the training accuracy is improved, and the classification accuracy of the trained multi-label sentiment classification model is improved.

[0183] Since steps 520-560 have been described in detail above, the above step 510 and the steps included after step 560 are described in detail below.

[0184] Detailed description of step 510

[0185] In one embodiment, referring to Fig.10 , step 510 includes:

[0186] Step 1010: for each multi-label sentiment classification model, input the sample text in the initial sample text set into the multi-label sentiment classification model to obtain a first positive discriminant probability corresponding to each sentiment classification;

[0187] Step 1020: Determine the sentiment classification with the largest first positive discriminant probability as the maximum probability sentiment classification corresponding to the multi-label sentiment classification model.

[0188] Sentiment classification can represent the category of sentiment. Fig.11A As shown in Figure 1, the emotion classification includes love, joy, expectation, faith, anger, surprise, resentment, worry, doubt and none. None means that the text does not have any emotion. For example, the text about the scene narration in the script is generally recognized as none. In one example, Fig. 11B As shown in the figure, the target text "The third prince realized that the emperor was going to announce him as the crown prince, and he was a little slow to react" was input into the multi-label sentiment classification model, and the first positive discriminant probability of the sentiment classification of surprise was 0.5, the first positive discriminant probability of the sentiment classification of no sentiment was 0.3, and the first positive discriminant probabilities of other sentiment classifications were all 0. Since 0.5>0.3, the maximum probability sentiment classification is surprise.

[0189] Sentiment classification can characterize the category of emotion and the intensity of emotion. Fig. 12A As shown, the emotion categories include love 1, love 2, love 3, happiness 1, happiness 2, happiness 3, expectation 1, expectation 2, expectation 3, trust 1, trust 2, trust 3, anger 1, anger 2, anger 3, surprise 1, surprise 2, surprise 3, resentment 1, resentment 2, resentment 3, worry 1, worry 2, worry 3, doubt 1, doubt 2, doubt 3 and none. Among them, 1, 2, 3 represent low, medium and high intensity of emotion respectively. In an example, as Fig. 12B As shown in the figure, the target text "The third prince realized that the emperor was going to announce him as the crown prince, and he was a little slow to react" was input into the multi-label sentiment classification model, and the first positive discriminant probability of the sentiment classification of "surprise 1" was 0.9, the first positive discriminant probability of the sentiment classification of "surprise 2" was 0.4, the first positive discriminant probability of the sentiment classification of "surprise 3" was 0.2, the first positive discriminant probability of the sentiment classification of "no" was 0.3, and the first positive discriminant probabilities of other sentiment classifications were all 0. Since 0.9>0.4>0.3>0.2, the highest probability sentiment classification is "surprise 1".

[0190] The advantage of the embodiment of steps 1010 - 1020 is that the maximum probability sentiment classification is determined by comparing the probabilities of multiple sentiment classifications, which has higher accuracy.

[0191] Reference Fig. 9CThe multi-label sentiment classification model includes an embedding layer, a basic learning model, a sentiment detail learning model, and a classification layer. In one embodiment, step 1010 includes:

[0192] Input the sample text in the initial sample text set into the embedding layer to obtain the sample text vector;

[0193] Input the sample text vector into the basic learning model to obtain the sample semantic feature vector;

[0194] Input the sample semantic feature vector into the emotional detail learning model to obtain the sample high-order emotional feature vector;

[0195] The sample high-order sentiment feature vector is input into the classification layer to obtain the maximum probability sentiment classification corresponding to the multi-label sentiment classification model.

[0196] In this embodiment, the embedding layer is a layer for converting text into a vector. The sample text vector refers to the vector corresponding to the sample text. The basic learning model is a model for analyzing / recognizing the semantic information contained in the vector. The sample semantic feature vector refers to a vector with emotional semantic information (such as information that the text belongs to a certain emotional category). The emotional detail learning model is a model for analyzing / recognizing the emotional detail information contained in the vector. The sample high-order emotional feature vector refers to a vector with high-order emotional semantic information (such as information that the text belongs to a certain emotional category and the emotional intensity corresponding to the emotional category). The basic learning model and the emotional detail learning model can be common deep neural networks, such as the Bert model. The classification layer is a layer for dividing vectors into different groups / categories. The classification layer can be composed of a fully connected layer.

[0197] The advantage of the above embodiment is that the basic learning model is first used to obtain a vector full of emotional semantic information, and then the emotional detail learning model is used to obtain a vector full of high-order emotional feature information from the vector full of emotional semantic information, thereby improving the accuracy of the maximum probability emotional classification classified by the classification layer.

[0198] Reference Fig.13 In one embodiment, step 1020 includes:

[0199] Step 1310: Obtain discrimination thresholds for multiple sentiment classifications;

[0200] Step 1320: select, among the multiple emotion classifications, the emotion classification whose first positive discrimination probability is greater than the discrimination threshold as a candidate emotion classification;

[0201] Step 1330: Determine the candidate sentiment classification with the largest first positive discriminant probability as the maximum probability sentiment classification corresponding to the multi-label sentiment classification model.

[0202] The discrimination threshold of sentiment classification refers to the threshold for determining whether the text contains the sentiment classification. The discrimination thresholds corresponding to different sentiment classifications are not necessarily the same. For a certain sentiment classification, the first positive discrimination probability of the sentiment classification output by the model is always relatively large, but the text does not actually contain the sentiment classification. By setting a relatively large discrimination threshold, even if the first positive discrimination probability is relatively large, the sentiment classification will not be used as a candidate sentiment classification. For example, assuming that the discrimination threshold of the sentiment category is 0.6, if the first positive discrimination probability of the sentiment classification is greater than 0.6, it means that the sentiment category of the sample text includes surprise. If the first positive discrimination probability of the sentiment classification is less than or equal to 0.6, it means that the sentiment category of the sample text does not include surprise.

[0203] In one example, if Fig.14 As shown, multiple emotion categories include "Shock 1", "Shock 2", "Shock 3", and "None", the discrimination thresholds are 0.5, 0.6, 0.7, and 0.8 respectively, and the first positive discrimination probabilities are 0.6, 0.4, 0.3, and 0.7 respectively. For "Shock 1", since 0.6 is greater than 0.5, "Shock 1" is selected as a candidate emotion classification. For "Shock 2", since 0.4<0.6, "Shock 2" cannot be selected as a candidate emotion classification. For "Shock 3", since 0.3<0.7, "Shock 3" cannot be selected as a candidate emotion classification. For "None", since 0.7<0.8, "None" cannot be selected as a candidate emotion classification. Since the candidate emotion classification with the highest first positive discrimination probability is "Shock 1", "Shock 1" is selected as the emotion classification with the highest probability.

[0204] It should be understood that the embodiments of the present disclosure set multiple multi-label sentiment learning models, so there are multiple maximum probability sentiment classifications. For example, if a total of 3 multi-label sentiment learning models are set, there are a total of 3 maximum probability sentiment classifications.

[0205] The advantage of the embodiment of steps 1310-1330 is that, based on the comparison between the discrimination threshold and the first positive discrimination probability, it is first determined whether the sentiment classification can be used as a candidate sentiment classification, thereby completing the first screening of the sentiment classification; the maximum probability sentiment classification is determined from the candidate sentiment classifications, thereby completing the second screening of the sentiment classification; after two screenings, the accuracy of determining the maximum probability sentiment classification is greatly improved.

[0206] In one embodiment, step 1310 includes:

[0207] Based on a predetermined interval step, a plurality of candidate discrimination thresholds are set;

[0208] For each sentiment classification, obtaining a second sentiment classification label of the sentiment classification of the first test text in the first test text set;

[0209] Inputting the first test text in the first test text set into the multi-label sentiment classification model to obtain a fifth sentiment classification result of sentiment classification of the first test text under each candidate discrimination threshold;

[0210] Determine a first accuracy rate corresponding to each candidate discrimination threshold based on a comparison between the second sentiment classification label and the fifth sentiment classification result corresponding to each candidate discrimination threshold;

[0211] Based on the first accuracy rates corresponding to the candidate discrimination thresholds, a discrimination threshold for sentiment classification is determined.

[0212] In this embodiment, the predetermined interval step refers to the length of the parameter increase or decrease. For example, when the determination threshold range is [0, 1], the predetermined interval step can be set to 0.3, and the candidate determination thresholds include 0.3, 0.6, and 0.9.

[0213] The first test text set refers to a set storing multiple first test texts. The first test text refers to the text used for testing. The second sentiment classification label is used to indicate whether the first test text belongs to the sentiment classification. For example, the first test text is "I like puppies", and the second sentiment classification label includes (happy, 1), (year, 0). Among them, (happy, 1) indicates that "I like puppies" belongs to "happy". (year, 0) indicates that "I like puppies" does not belong to "year".

[0214] The fifth sentiment classification result is used to indicate whether the first test text belongs to the sentiment classification. For the first test text, the multi-label sentiment classification model outputs a first positive discrimination probability. If the first positive discrimination probability is greater than the candidate discrimination threshold, it means that the first test text belongs to the sentiment classification corresponding to the fifth sentiment classification result. Therefore, under different discrimination thresholds, since the first positive discrimination probability may be greater than the candidate discrimination threshold or not greater than the candidate discrimination threshold, it affects the fifth sentiment classification result.

[0215] For each candidate discrimination threshold, the first accuracy rate corresponding to each candidate discrimination threshold can be determined based on the comparison between the second sentiment classification label and the fifth sentiment classification result. Specifically, for the first test text, if the second sentiment classification label is the same as the fifth sentiment classification result, the number of the same is increased by 1; based on the ratio of the same number to the total number of samples in the first test text set, the first accuracy rate is obtained.

[0216] After obtaining the first accuracy rates corresponding to the candidate discrimination thresholds, the candidate discrimination threshold with the largest first accuracy rate may be used as the discrimination threshold corresponding to the sentiment classification.

[0217] Reference Fig.15In one example, the first test text is "A and B looked at each other, B nodded, and walked into the house". The multi-label sentiment classification model is used to perform sentiment classification on "A and B looked at each other, B nodded, and walked into the house". When the candidate discrimination threshold is 0.3, the fifth sentiment classification results include (surprise 1, 1), (surprise 2, 1), (surprise 3, 1), and (none, 1), and the fifth sentiment classification results of the remaining sentiment classifications are all judged to be 0. When the candidate discrimination threshold is 0.6, the fifth sentiment classification results include (surprise 1, 1), and the remaining sentiment classifications are all judged to be 0. When the candidate discrimination threshold is 0.6, the fifth sentiment classification results of all sentiment classifications are judged to be 0. Assume that the second sentiment classification label is "surprise 1". When the candidate discrimination threshold is 0.3, since the fifth sentiment classification result includes (surprise 1, 1), which is the same as the second sentiment classification label, the first accuracy is 1. When the candidate discrimination threshold is 0.6, since the fifth sentiment classification result includes (surprise 1, 1), which is the same as the second sentiment classification label, the first accuracy rate is 1. When the candidate discrimination threshold is 0.9, since the fifth sentiment classification result includes (surprise 1, 0), which is different from the second sentiment classification label, the first accuracy rate is 0. Finally, the discrimination threshold for the sentiment classification of "surprise 1" can be any number between 0.3 and 0.6. In one embodiment, for "surprise 1", a predetermined interval step of 0.1 can be set between 0.3 and 0.6, and the candidate discrimination thresholds include 0.3, 0.4, 0.5 and 0.6. Based on the first accuracy rates corresponding to 0.3, 0.4, 0.5 and 0.6, respectively, the discrimination threshold of "surprise 1" is further determined.

[0218] Detailed description of the steps included after step 560

[0219] Based on the foregoing, it can be known that the multi-label sentiment classification model includes a basic learning model, a sentiment detail learning model and a classification layer. In step 550, the basic learning model, the sentiment detail learning model and the classification layer are jointly trained based on the first loss function. After step 550, the basic learning model can already better identify the semantic information of the vector (the semantic information enables the classification layer to obtain the sentiment classification of the text), but the sentiment detail learning model cannot yet identify the intensity of the sentiment well, which affects the intensity corresponding to the sentiment classification of the text obtained by the classification layer. Therefore, in one embodiment, the sentiment detail learning model and the classification layer need to be fine-tuned to make the multi-label sentiment classification model more accurate.

[0220] Reference Fig.16 In one embodiment, after step 560, the multi-label sentiment classification model training method further includes:

[0221] Step 1610: divide the first sentiment classification label of each sentiment classification of the matching sample text into a first category label, a second category label, and a non-sentiment label;

[0222] Step 1620: Obtain, from the matching sample text set, a first sample text with a second category label and a second sample text with a non-emotional label;

[0223] Step 1630: obtaining a first easily confused sample text from the first sample text, obtaining a second easily confused sample text from the second sample text, and determining other sample texts other than the first easily confused sample text and the second easily confused sample text in the matching sample text set as non-easy confused sample texts;

[0224] Step 1640: determine a second loss function based on the loss sub-functions of the first easily confused sample text, the second easily confused sample text, and the non-easy confused sample text, and adjust the sentiment detail learning model and the classification layer in the trained multi-label sentiment classification model based on the second loss function.

[0225] Steps 1610-1640 belong to Fig. 9A The process of model prediction and label stratification, soft label generation, and model-directed soft label learning is shown. Fig. 9A The partial label correction and accurate label learning shown will be explained in conjunction with specific embodiments below and will not be repeated here. Fig.17A As shown in Figure 1, the process of model prediction and label stratification can be divided into test set label prediction and label partitioning. The process of generating soft labels includes training sample label prediction and generating soft labels from easily confused labels and original labels.

[0226] In step 1610, the matching sample text set is used as a test set. For the test set label prediction process, the matching sample text set can be input into a multi-label sentiment classification model (the model is trained based on the first loss function) to obtain multiple first sentiment classification labels. For the label division process, the multiple first sentiment classification labels are divided into first-class labels, second-class labels, and non-sentiment labels. The degree of confusion between the first-class labels and the non-sentiment labels is less than the degree of confusion between the second-class labels and the non-sentiment labels. The degree of confusion refers to the degree of easy confusion between two labels. Multiple first sentiment classification labels include non-sentiment labels and various sentiment classification labels with various levels of sentiment intensity.

[0227] In step 1620, the first sample text is a matching sample text with a second type of label in the matching sample text set, and the second sample text is a matching sample text with a non-emotional label in the matching sample text set.

[0228] It should be understood that the sample text with the non-confusable label is generally the non-confusable sample text, and the sample text with the easily confused label is generally the easily confused sample text, but there may also be non-confusable sample text. Therefore, in step 1630, the real easily confused sample text is mainly extracted from the first sample text and the second sample text to achieve accurate division of the matching sample text set.

[0229] like Fig. 17B As shown, in one example, the matching sample text set includes 3 kinds of sample texts. The first kind of sample text carries a second type of label, and is therefore classified as the first sample text in the first sample set. The second kind of sample text carries a non-emotional label, and is therefore classified as the second sample text in the second sample text set. The third kind of sample text carries a first type of label, and is therefore classified as a non-confusable sample text. In the first sample text, a first confusable sample text and a first non-confusable sample text can be obtained based on the degree of confusability. In the second sample text, a second confusable sample text and a second non-confusable sample text can be obtained based on the degree of confusability. Among them, the first non-confusable sample text, the second non-confusable sample text, and the sample text with the first type of label are used as non-confusable sample texts in the non-confusable sample text set.

[0230] It should be noted that the means for obtaining the first easily confused sample text and the second easily confused sample text will be described below in conjunction with specific embodiments and will be omitted here.

[0231] In step 1640, although the first easily confused sample text, the second easily confused sample text, and the non-confusable sample text all come from the matching sample text set, the degree of confusion of the samples is different, so that the degree of error in the sentiment classification of the model discrimination samples is different. If the emotion detail learning model and the classification layer in the multi-label sentiment classification model are trained with a common loss function, the training effect is poor. The innovation of this embodiment is that, for model-oriented soft label learning, the matching sample text set is first carefully divided based on the degree of confusion, so that the second loss function can be determined based on the loss sub-functions of the samples with different degrees of confusion. In this way, the second loss function can fully represent the error of the model in predicting samples with different degrees of confusion, so that the emotion detail learning model and classification layer adjusted based on the second loss function have higher accuracy.

[0232] An advantage of the embodiment of steps 1610-1640 is that the adjusted emotion detail learning model and classification layer have higher accuracy.

[0233] In one embodiment, referring to Fig.18 , step 1610 includes:

[0234] Step 1810: extracting a non-emotion label from a plurality of first emotion classification labels;

[0235] Step 1820: for each emotion classification label of each emotion intensity, input the second test text with the emotion classification label in the second test text set into the multi-label emotion classification model to obtain a sixth emotion classification result;

[0236] Step 1830: Based on the comparison between the sixth emotion classification result and the emotion classification label, classify the emotion classification label into a first category label or a second category label.

[0237] In step 1810, if the first sentiment classification label is "none", "none" is classified as a no-sentiment label. For the sentiment classification of "none", the first sentiment classification label corresponding to "none" is classified as a no-sentiment label.

[0238] It should be understood that the first sentiment classification label indicates whether the text belongs to a sentiment classification. The value of the first sentiment classification label can be set to 1 (1 means belonging) or 0 (0 means not belonging).

[0239] In step 1820, the intensity of emotion can be divided into high, medium, and low. For example, Love 3, Love 2, and Love 1, which indicate that the emotion classification is love, has an intensity of high, medium, and low emotion. For a certain emotion classification, if the emotion intensity is high, the model can generally accurately predict the emotion classification. On the contrary, if the emotion intensity is low, the model may not be able to predict the accurate emotion classification, but predicts the emotion classification as "none". In other words, under different emotion intensity levels, the sixth emotion classification result predicted by the multi-label emotion classification model may be accurate or wrong.

[0240] In step 1830, for each sentiment classification label, if the sixth sentiment classification result is accurate, the sixth sentiment classification result is the same as the sentiment classification label, indicating that the sentiment classification label is not easily confused, so the sentiment classification label is classified as the first category label. If the sixth sentiment classification result is wrong, the sixth sentiment classification result is different from the sentiment classification label, indicating that the sentiment classification label is easily confused, so the sentiment classification label is classified as the second category label.

[0241] In one example, if Fig.19As shown, the emotion classification includes love 1, love 2, love 3, happiness 1, happiness 2, happiness 3, expectation 1, expectation 2, expectation 3, trust 1, trust 2, trust 3, anger 1, anger 2, anger 3, surprise 1, surprise 2, surprise 3, resentment 1, resentment 2, resentment 3, worry 1, worry 2, worry 3, doubt 1, doubt 2, doubt 3 and nothing. Among them, 1, 2, 3 represent low, medium and high emotion intensity respectively. First, "nothing" will be classified as no emotion label. The value of each emotion classification label is 1 or 0. For love 1, love 2, and love 3, love 1 is more likely to be confused with "nothing" due to its low emotion intensity, that is, the confusion degree is high. Therefore, love 1 will be classified as the second category label, while love 2 and love 3 are classified as the first category label. Similarly, the first emotion classification labels of other emotion classifications are classified as the first category label or the second category label.

[0242] The advantage of the embodiment of steps 1810-1830 is that the emotion-free label is first extracted from multiple first emotion classification labels, and then the model is used to extract the first category labels and the second category labels from the remaining first emotion classification labels, thereby improving the efficiency and accuracy of label division while achieving label division.

[0243] In one embodiment, step 1830 includes:

[0244] The number of second test texts with the sentiment classification label in the second test text set is determined as the fifth number;

[0245] The number of second test texts in the second test text set with the sentiment classification label and for which the sixth sentiment classification result is inconsistent with the sentiment classification label is determined as the sixth number;

[0246] determining a degree of confusability based on the fifth number and the sixth number;

[0247] If the degree of confusion is less than the first threshold, the sentiment classification label is classified as a first category label; if the degree of confusion is not less than the first threshold, the sentiment classification label is classified as a second category label.

[0248] In one example, assuming that the number of second test texts with sentiment classification labels (such as love 1) in the second test text set is 800, the fifth number is 800. If among the 800 second test texts, the number of second test texts whose sixth sentiment classification results (such as love 2, surprise 1, none, etc.) are inconsistent with the sentiment classification label (such as love 1) is 600, the sixth number is 600. The confusion degree is 600 / 800=0.75. If the first threshold is 0.6, since 0.75 is greater than 0.6, the sentiment classification label (such as love 1) is classified as the second category label. Assuming that the number of second test texts with sentiment classification labels (such as love 3) in the second test text set is 800, and the sixth number is 200. The confusion degree is 200 / 800=0.25, since 0.25 is less than 0.6, the sentiment classification label (love 3) is classified as the first category label.

[0249] In one embodiment, referring to Fig. 20 , step 1630 includes:

[0250] Step 210: input the first sample text into the trained multi-label sentiment classification model to obtain the maximum probability sentiment classification determined by the trained multi-label sentiment classification model and the first discrimination probability corresponding to the first sentiment classification label of the first sample text determined by the trained multi-label sentiment classification model;

[0251] Step 220: If the maximum probability sentiment classification determined by the trained multi-label sentiment classification model is inconsistent with the first sentiment classification label of the first sample text, and it is determined that the first discrimination probability and the first discrimination threshold corresponding to the first sentiment classification label of the first sample text satisfy a first relationship, the first sample text is determined as a first easily confusing sample text;

[0252] Step 2030: input the second sample text into the trained multi-label sentiment classification model to obtain a second discrimination probability corresponding to the first sentiment classification label of the second sample text discriminated by the trained multi-label sentiment classification model;

[0253] Step 2040: If it is determined that the second discrimination probability and the second discrimination threshold corresponding to the first sentiment classification label of the second sample text satisfy the second relationship, the second sample text is determined as a second easily confusing sample text.

[0254] In step 2010, the maximum probability sentiment classification refers to the sentiment classification with the largest discrimination probability among multiple sentiment classifications. The first discrimination probability refers to the discrimination probability of the sentiment classification with the same first sentiment label among multiple sentiment classifications. For example, for the first sample text "I go to the park on Saturday", the trained multi-label sentiment classification model outputs the probability of "happy" as 0.8, the probability of "expected" as 0.6, and the probability of "no" as 0.3, then the maximum probability sentiment classification is "happy". Assuming that the sentiment classification label corresponding to "I go to the park on Saturday" is "expected", the first discrimination probability is 0.6.

[0255] In step 220, since the trained multi-label sentiment model is screened from multiple multi-label sentiment classification models trained based on the first loss function, the trained multi-label sentiment classification model has a higher discrimination accuracy. The maximum probability sentiment classification output by the trained multi-label sentiment classification model should be accurate. If the maximum probability sentiment classification is inconsistent with the first sentiment classification label of the first sample text, it means that the model has a discrimination error, and it can also be said that the model cannot accurately identify the first sample text. Therefore, the first text in which the inconsistency occurs can be divided into a first easily confused sample text, so that the model based on the first easily confused sample text training model has the ability to accurately identify the sentiment classification of this type of text. In addition to considering whether there is an inconsistency, it is also considered whether the first discrimination probability and the first discrimination threshold corresponding to the first sentiment classification label of the first sample text satisfy the first relationship. The first relationship refers to the relationship that the first discrimination probability set in advance and the first judgment threshold corresponding to the first sentiment classification label should satisfy.

[0256] In one embodiment, determining that the first discrimination probability and the first discrimination threshold corresponding to the first sentiment classification label of the first sample text satisfy a first relationship includes:

[0257] Get the first confidence coefficient;

[0258] determining a first product of a first confidence coefficient and a first discrimination threshold;

[0259] If the first discrimination probability is less than the first product, it is determined that the first discrimination probability and the first discrimination threshold satisfy a first relationship.

[0260] The first confidence coefficient refers to the coefficient / constant used as a weight. For example, the first confidence coefficient is 0.5. The first product refers to the product of the first confidence coefficient and the first discrimination threshold. For example, the first discrimination threshold of "period" is 0.6, then the first product is 0.5*0.6=0.3. Assuming the first discrimination probability is 0.6, since 0.6 is greater than 0.3, it is determined that the first discrimination probability and the first discrimination threshold do not satisfy the first relationship. Assuming the first discrimination probability is 0.2, since 0.2 is less than 0.3, it is determined that the first discrimination probability and the first discrimination threshold satisfy the first relationship.

[0261] The advantage of this embodiment is that the first confidence coefficient is introduced, and instead of directly comparing the first discrimination threshold with the first discrimination probability, the first product obtained based on the first discrimination threshold and the first confidence coefficient is compared with the first discrimination probability, which can improve the flexibility of determination on the basis of determining whether the first relationship is satisfied; in addition, the first confidence coefficient can be flexibly adjusted, which improves the flexibility of determining the first product.

[0262] In one embodiment, after step 2020, the method further includes:

[0263] Divide the interval from 0 to the first discrimination threshold into a plurality of first sub-intervals;

[0264] For each first sub-interval, a predetermined proportion of the first easily confused sample texts is extracted from the first easily confused sample texts whose first discrimination probability falls within the first sub-interval, the first sentiment classification label is corrected, and the corrected first easily confused sample texts are used as non-easy to confuse sample texts.

[0265] The first sub-interval refers to a part of the interval from 0 to the first discrimination threshold. Assuming that the first discrimination threshold is 0.3 and the interval between adjacent sub-intervals is 0.1, the first sub-interval includes [0, 0.1), [0.1, 0.2), and [0.2, 0.3].

[0266] For each first easily confused sample text, the first discrimination probability is less than the first product, but the first discrimination probability is not necessarily less than the first discrimination threshold (for example, when the first confidence coefficient is less than 1, the first product is less than the first discrimination threshold). In this embodiment, the first easily confused sample text whose first discrimination probability is less than the first discrimination threshold is mainly targeted, and their first sentiment classification labels are corrected. The corrected first sentiment classification label is used as the first sentiment classification label of the first easily confused sample text, so that the corrected first easily confused sample text can be used as a non-easy to confuse sample text.

[0267] It should be understood that a predetermined proportion of the first easily confused sample texts can be directly extracted from the first easily confused sample texts whose first discrimination probability is less than the first discrimination threshold, and then correction can be performed. However, such extraction may affect the correction effect. For example, the texts extracted may all have a first discrimination probability that is relatively close to the first discrimination threshold (for example, the first discrimination probability is 0.59, the first discrimination threshold is 0.6, and the difference is 0.01), and it is not very meaningful to correct the first sentiment classification labels of these sample texts. Therefore, in this embodiment, a sub-interval method is introduced to divide the interval from 0 to the first discrimination threshold into multiple first sub-intervals, and then extract a corresponding number of texts from each first sub-interval, so that the extracted sample texts can be evenly distributed in the interval from 0 to the first discrimination threshold.

[0268] The advantage of this embodiment is that it improves the accuracy of data extraction and can achieve the purpose of effective correction.

[0269] In step 2030, the second discrimination probability refers to the discrimination probability of the sentiment classification with the same first sentiment classification label as the second sample text among the multiple sentiment classifications discriminated by the trained multi-label sentiment classification model. For example, for the second sample text "I am at home today", the trained multi-label sentiment classification model outputs the probability of "happy" as 0.3, the probability of "expected" as 0.9, and the probability of "no" as 0.6. If the first sentiment classification label corresponding to "I am at home today" is "no", then the second discrimination probability is 0.6.

[0270] In step 2040, if it is determined that the second discrimination probability and the second discrimination threshold corresponding to the first sentiment classification label of the second sample text satisfy the second relationship, the second sample text is determined as the second easily confused sample text. The second relationship refers to the relationship that the second discrimination probability set in advance and the second judgment threshold corresponding to the second sentiment classification label should satisfy.

[0271] In one embodiment, determining that the second discrimination probability and the second discrimination threshold corresponding to the first sentiment classification label of the second sample text satisfy the second relationship includes:

[0272] Get the second confidence coefficient;

[0273] determining a second product of a second confidence coefficient and a second discrimination threshold;

[0274] If the second discrimination probability is less than the second product, it is determined that the second discrimination probability and the second discrimination threshold satisfy a second relationship.

[0275] The second confidence coefficient refers to a coefficient / constant used as a weight. For example, the second confidence coefficient is 0.5. The second product refers to the product of the second confidence coefficient and the second discrimination threshold. For example, the second discrimination threshold of "love" is 0.8, then the second product is 0.5*0.8=0.4. Assuming the second discrimination probability is 0.5, since 0.5 is greater than 0.4, it is determined that the second discrimination probability and the second discrimination threshold do not satisfy the first relationship. Assuming the second discrimination probability is 0.3, since 0.3 is less than 0.4, it is determined that the second discrimination probability and the second discrimination threshold satisfy the second relationship.

[0276] The advantage of this embodiment is that a second confidence coefficient is introduced, and instead of directly comparing the second discrimination threshold with the second discrimination probability, a second product obtained based on the second discrimination threshold and the second confidence coefficient is compared with the second discrimination probability, which can improve the flexibility of determination on the basis of determining whether the second relationship is satisfied; in addition, the second confidence coefficient can be flexibly adjusted, which improves the flexibility of determining the second product.

[0277] like Fig. 9A As shown, the training process of the multi-label sentiment classification model includes a process of correcting some labels. In one embodiment, after step 2040, it also includes:

[0278] Dividing the interval from 0 to the second discrimination threshold into a plurality of second sub-intervals;

[0279] For each second sub-interval, a predetermined proportion of the second easily confused sample texts is extracted from the second easily confused sample texts whose second discrimination probability falls within the second sub-interval, the first sentiment classification label is corrected, and the corrected second easily confused sample texts are used as non-easy to confuse sample texts.

[0280] The second sub-interval refers to a part of the interval from 0 to the second discrimination threshold. Assuming that the second discrimination threshold is 0.3 and the interval between adjacent sub-intervals is 0.1, the first sub-interval includes [0, 0.1), [0.1, 0.2), and [0.2, 0.3]. For each second easily confused sample text, the second discrimination probability is less than the second product, but the second discrimination probability is not necessarily less than the second discrimination threshold (for example, when the second confidence coefficient is less than 1, the second product is less than the second discrimination threshold). In this embodiment, the second easily confused sample text whose second discrimination probability is less than the second discrimination threshold is mainly targeted, and their first sentiment classification labels are corrected. The corrected first sentiment classification label is used as the first sentiment classification label of the second easily confused sample text, so that the corrected second easily confused sample text can be used as a non-confusing sample text. In this embodiment, the sub-interval method is introduced to divide the interval from 0 to the second discrimination threshold into multiple second sub-intervals, and then extract a corresponding number of sample texts from each second sub-interval, so that the extracted sample texts can be evenly distributed in the interval from 0 to the second discrimination threshold.

[0281] The advantage of this embodiment is that the accuracy of data extraction is improved, and the purpose of effective correction can be achieved. Fig. 9A As shown, after generating soft labels and correcting some labels, accurate label learning can be performed, which further improves the training effect of the multi-label sentiment classification model and improves the classification accuracy of the multi-label sentiment classification model.

[0282] In one embodiment, referring to Fig.21 , step 1640 includes:

[0283] Step 2110: determining a first auxiliary label for each sentiment classification for the first easily confused sample text;

[0284] Step 2120: Calculate a second loss sub-function of the first easily confusing sample text based on the first auxiliary label;

[0285] Step 2130: for the second easily confusing sample text, determine a second auxiliary label for each sentiment classification;

[0286] Step 2140: Calculate a third loss sub-function of the second easily confusing sample text based on the second auxiliary label;

[0287] Step 2150, calculating the fourth loss sub-function of the non-confusable sample text;

[0288] Step 2160: Calculate the second loss function based on the second loss sub-function, the third loss sub-function, and the fourth loss sub-function.

[0289] In step 2110, the first auxiliary label refers to the auxiliary label of the first easily confused sample text, also known as a soft label. Since the multi-label sentiment classification model is prone to misidentify the sentiment classification result of the first easily confused sample text, this embodiment mainly introduces the first auxiliary label and uses the first auxiliary label to participate in the calculation of the loss function, thereby improving the model training effect.

[0290] In one embodiment, referring to Fig. 22 , step 2110 includes:

[0291] Step 2210: input the first easily confused sample text into the trained multi-label sentiment classification model to obtain a first discrimination probability corresponding to the first sentiment classification label discriminated by the trained multi-label sentiment classification model;

[0292] Step 2220: Determine a first auxiliary label of the sentiment classification corresponding to the first sentiment classification label based on the first sentiment classification label and the first discrimination probability corresponding to the first sentiment classification label.

[0293] In one example, the first easily confused sample text is "I am at home today". The first sentiment classification labels are (love, 0), (joy, 1), (concern, 0), (surprise, 0), and (doubt, 0), which can be expressed as a1 = [0, 1, 0, 0, 0]. Input "I am at home today" into the trained multi-label sentiment classification model, and the first discrimination probability corresponding to "love" is 0.4, the first discrimination probability corresponding to "joy" is 0, the first discrimination probability corresponding to "concern" is 0, the first discrimination probability corresponding to "surprise" is 0, and the first discrimination probability corresponding to "doubt" is 0.6, which can be expressed as a1 = [0.4, 0, 0, 0.6]. The first auxiliary label is the average of the first sentiment classification label and the first discrimination probability, that is, 0.5a1+0.5a2 = [0.2, 0.5, 0, 0, 0.3]. On the basis of calculating the average value, the average value can also be normalized with a modulus length of 1, so that the first auxiliary label falls within a specific numerical range, which is convenient for processing and comparing each first auxiliary label.

[0294] In one embodiment, step 2220 includes:

[0295] Obtaining a third weight of the first sentiment classification label and a fourth weight of the first discrimination probability;

[0296] Based on the third weight and the fourth weight, a second weighted sum of the first sentiment classification label and the first discrimination probability is determined as the first auxiliary label.

[0297] The third weight indicates the importance of the first sentiment classification label. The fourth weight indicates the importance of the first discrimination probability. The third weight and the fourth weight can be set according to actual needs. In the above example, the third weight and the fourth weight are both 0.5, indicating that the first sentiment classification label has the same importance as the first discrimination probability. In one example, assuming that the third weight is 0.6 and the fourth weight is 0.4, the first auxiliary label is 0.6a1+0.4a2=[0.24, 0.4, 0, 0, 0.36].

[0298] The advantage of this embodiment is that the third weight and the fourth weight are introduced, which improves the flexibility of determining the first auxiliary tag and has high applicability.

[0299] The advantage of the embodiment of steps 2210-2220 is that, in addition to the first emotion classification label, the first discrimination probability on each first emotion classification label (except the no emotion label) is retained, and the probabilities of all labels can be normalized, thereby improving the accuracy of determining the first auxiliary label.

[0300] In step 2120, based on the first auxiliary label, a second loss sub-function of the first easily confused sample text is calculated. The second loss sub-function indicates the prediction error of the multi-label sentiment classification model for the first easily confused sample text.

[0301] In one embodiment, referring to Fig.23 , step 2120 includes:

[0302] Step 2310: input the first easily confused sample text into the trained multi-label sentiment classification model to obtain the third positive discrimination probability of each sentiment classification discriminated by the trained multi-label sentiment classification model;

[0303] Step 2320: determining a first loss item for sentiment classification based on the third positive discriminant probability of sentiment classification and the first auxiliary label;

[0304] Step 2330: Accumulate the first loss items of multiple sentiment classifications of the first easily confused sample text to obtain a first accumulated loss item;

[0305] Step 2340: Accumulate the first cumulative loss items of the plurality of first easily confusing sample texts to obtain a second loss sub-function.

[0306] In step 2310, the third positive discriminant probability indicates the probability that the first easily confused sample text determined by the multi-label sentiment classification model belongs to the sentiment classification. For example, the first easily confused sample text is "I am at home today", and the multi-label sentiment classification model determines that the third positive probability of "happy" is 0.8, and the third positive probability of "surprised" is 0.1, etc.

[0307] In step 2320, the first loss term indicates the gap between the third positive discriminant probability and the first auxiliary label. The smaller the first loss term, the more accurate the multi-label sentiment classification model is. For example, in the first auxiliary label, the label of "happy" is 1, the label of "surprise" is 0, the first loss term of "love" is 0.2, and the first loss term of "surprise" is 0.1.

[0308] In one embodiment, step 2320 includes:

[0309] Determine a second difference between a third positive discriminant probability of the sentiment classification and the first auxiliary label;

[0310] The square of the second difference is determined as the first loss term.

[0311] In one example, based on the above example, the first loss term of “love” is 0.2*0.2=0.04, and the first loss term of “surprise” is 0.1*0.1=0.01.

[0312] The advantage of this embodiment is that the first loss term is determined based on the square of the second difference, which improves the accuracy of determination, thereby improving the accuracy of determining the second loss function.

[0313] In step 2330, the first accumulated loss term indicates the accumulation of multiple first loss terms of a first easily confused sample text. For example, based on the above example, the first accumulated loss term is 0.02+0.01=0.03.

[0314] In step 2340, the second loss sub-function indicates the accumulation of the first cumulative loss items of the plurality of first easily confused sample texts. For example, assuming there are three first easily confused sample texts, the first cumulative loss items are 0.03, 0.04, and 0.01, respectively. The second loss sub-function is 0.01+0.04+0.01=0.08.

[0315] In this embodiment, the calculation process of determining the second loss sub-function is shown in Formula 2:

[0316]

[0317] In formula 2, L reg2represents the second loss sub-function. n2 represents that there are n2 first easily confused sample texts among bs samples. i represents the i-th first easily confused sample text. j represents the j-th sentiment classification. M represents that there are M sentiment classifications in total. p[i] j Represents the third positive discriminant probability of the jth sentiment classification of the i-th first easily confused sample text. s[i] j Represents the first auxiliary label of the jth sentiment classification of the i-th sample text.

[0318] An advantage of the embodiment of steps 2310 - 2340 is that the accuracy of determining the second loss sub-function is improved.

[0319] In step 2130, the second auxiliary label refers to the auxiliary label of the second easily confused sample text. Since the multi-label sentiment classification model is easy to misidentify the sentiment classification result of the second easily confused sample text, this embodiment mainly introduces the second auxiliary label and uses the second auxiliary label to participate in the calculation of the loss function, thereby improving the model training effect.

[0320] In one embodiment, referring to Fig.24 , step 2130 includes:

[0321] Step 2410: input the second easily confused sample text into the trained multi-label sentiment classification model to obtain a second discrimination probability corresponding to the first sentiment classification label discriminated by the trained multi-label sentiment classification model;

[0322] Step 2420: Determine a second auxiliary label of the sentiment classification corresponding to the first sentiment classification label based on the first sentiment classification label and the second discrimination probability corresponding to the first sentiment classification label.

[0323] In one example, the second easily confused sample text is "I'm going out on Saturday". The first sentiment classification labels are (love, 0), (joy, 1), (concern, 0), (surprise, 0), and (doubt, 0), which can be expressed as a3 = [0, 1, 0, 0, 0]. Input "I'm going out on Saturday" into the trained multi-label sentiment classification model, and the second discrimination probability corresponding to "love" is 0.4, the second discrimination probability corresponding to "joy" is 0, the second discrimination probability corresponding to "concern" is 0, the second discrimination probability corresponding to "surprise" is 0, and the second discrimination probability corresponding to "doubt" is 0.6, which can be expressed as a4 = [0.4, 0, 0, 0.6]. The second auxiliary label is the average of the first sentiment classification label and the second discrimination probability, that is, 0.5a3 + 0.5a4 = [0.2, 0.5, 0, 0, 0.3]. On the basis of calculating the average value, the average value can also be normalized with a modulus of 1, so that the second auxiliary label falls within a specific numerical range, which is convenient for processing and comparing each second auxiliary label.

[0324] In one embodiment, step 2420 includes:

[0325] Obtaining a third weight of the first sentiment classification label and a fifth weight of the second discrimination probability;

[0326] Based on the third weight and the fifth weight, a third weighted sum of the first sentiment classification label and the second discrimination probability is determined as the second auxiliary label.

[0327] The third weight indicates the importance of the first sentiment classification label. The fifth weight indicates the importance of the second discrimination probability. The third weight and the fifth weight can be set according to actual needs. In the above example, the third weight and the fifth weight are both 0.5, indicating that the first sentiment classification label and the second discrimination probability have the same importance. In one example, assuming that the third weight is 0.6 and the fifth weight is 0.4, the second auxiliary label is 0.6a3+0.4a4=[0.24, 0.4, 0, 0, 0.36].

[0328] The advantage of this embodiment is that the third weight and the fifth weight are introduced, which improves the flexibility of determining the second auxiliary tag and has high applicability.

[0329] The advantage of the embodiment of steps 2410-2420 is that, in addition to the first sentiment classification label, the second discrimination probability retained on each first sentiment classification label (except the no-sentiment label) can also normalize the probabilities of all labels, thereby improving the accuracy of determining the second auxiliary label.

[0330] In step 2140, based on the second auxiliary label, a third loss sub-function of the second easily confused sample text is calculated. The third loss sub-function indicates the prediction error of the multi-label sentiment classification model for the second easily confused sample text.

[0331] In one embodiment, referring to Fig.25 , step 2140 includes:

[0332] Step 2510: input the second easily confused sample text into the trained multi-label sentiment classification model to obtain the fourth positive discrimination probability of each sentiment classification discriminated by the trained multi-label sentiment classification model;

[0333] Step 2520: Determine a second loss item for sentiment classification based on the fourth positive discriminant probability of sentiment classification and the second auxiliary label;

[0334] Step 2530: Accumulate the second loss items of multiple sentiment classifications of the second easily confused sample text to obtain a second accumulated loss item;

[0335] Step 2540: Accumulate the second cumulative loss items of the plurality of second easily confusing sample texts to obtain a third loss sub-function.

[0336] In step 2510, the fourth positive discriminant probability indicates the probability that the second easily confused sample text determined by the multi-label sentiment classification model belongs to the sentiment classification. For example, the second easily confused sample text is "I'm going out on Saturday", and the fourth positive probability of "happy" determined by the multi-label sentiment classification model is 0.8, and the fourth positive probability of "surprise" is 0.1, etc.

[0337] In step 2520, the second loss term indicates the gap between the fourth positive discriminant probability and the second auxiliary label. The smaller the second loss term, the more accurate the multi-label sentiment classification model is. For example, in the second auxiliary label, the label of "happy" is 1 and the label of "surprise" is 0, then the second loss term of "love" is 0.2, and the second loss term of "surprise" is 0.1.

[0338] In one embodiment, step 2520 includes:

[0339] Determine a third difference between a fourth positive discriminant probability of the sentiment classification and the second auxiliary label;

[0340] The square of the third difference is determined as the second loss term.

[0341] In one example, based on the above example, the second loss term of “love” is 0.2*0.2=0.04, and the second loss term of “surprise” is 0.1*0.1=0.01.

[0342] The advantage of this embodiment is that the second loss term is determined based on the square of the third difference, which improves the accuracy of determination, thereby improving the accuracy of determining the third loss function.

[0343] In step 2530, the second accumulated loss term indicates the accumulation of multiple second loss terms of a second easily confused sample text. For example, based on the above example, the second accumulated loss term is 0.02+0.01=0.03.

[0344] In step 2540, the third loss sub-function indicates the accumulation of the second cumulative loss items of the plurality of second easily confused sample texts. For example, assuming there are three second easily confused sample texts, the second cumulative loss items are 0.03, 0.04, and 0.01, respectively. The third loss sub-function is 0.03+0.04+0.01=0.08.

[0345] In this embodiment, the calculation process of determining the third loss sub-function is shown in Formula 3:

[0346]

[0347] In formula 3, L reg3represents the third loss sub-function. n3 represents that there are n3 second most easily confused sample texts among bs samples. i represents the i-th second most easily confused sample text. j represents the j-th sentiment classification. M represents that there are M sentiment classifications in total. P2[i] j S2[i] represents the fourth positive discriminant probability of the jth sentiment classification of the i-th second most confusing sample text. j Represents the second auxiliary label of the jth sentiment classification of the i-th second easily confused sample text.

[0348] An advantage of the embodiment of steps 2510 - 2540 is that the accuracy of determining the third loss sub-function is improved.

[0349] In step 2150, a fourth loss sub-function of the non-confusable sample text is calculated. The fourth loss sub-function indicates the prediction error of the multi-label sentiment classification model for the non-confusable sample text. The smaller the fourth loss sub-function, the more accurate the multi-label sentiment classification model.

[0350] In one embodiment, referring to Fig.26 , step 2150 includes:

[0351] Step 2610: input the non-confusable sample text into the trained multi-label sentiment classification model to obtain the fifth positive discrimination probability of each sentiment classification discriminated by the trained multi-label sentiment classification model;

[0352] Step 2610: Calculate the fourth loss sub-function of the non-confusable sample text based on the fifth forward discriminant probability.

[0353] In step 2610, the fifth positive discriminant probability indicates the probability that the non-confusion sample text determined by the multi-label sentiment classification model belongs to the sentiment classification. For example, the non-confusion sample text is "I am very happy today", and the multi-label sentiment classification model determines that the fifth positive probability of "happy" is 0.9, and the fourth positive probability of "surprise" is 0.6, etc.

[0354] In one embodiment, step 2620 includes:

[0355] For each sentiment classification, based on the first sentiment classification label of the sentiment classification of the non-confusable sample text and the fifth forward discriminant probability, calculate the third loss item of the sentiment classification of the non-confusable sample text;

[0356] The third loss items of multiple sentiment classifications of non-confusable sample texts are accumulated to obtain a third accumulated loss item;

[0357] The third cumulative loss items of multiple non-confusable sample texts are accumulated to obtain a fourth loss sub-function.

[0358] Specifically, the third loss term indicates the gap between the fifth positive discriminant probability and the first sentiment classification label. The smaller the third loss term, the more accurate the multi-label sentiment classification model. For example, based on the above example, in the first sentiment classification label, the label of "joy" is 1 and the label of "surprise" is 0, then the third loss term of "love" is 0.1, and the third loss term of "surprise" is 0.4. The third cumulative loss term indicates the accumulation of multiple third loss terms of a non-confusable sample text. The fourth loss sub-function indicates the accumulation of the third cumulative loss terms of multiple non-confusable sample texts. For example, assuming there are 3 non-confusable sample texts, the third cumulative loss terms are 0.01, 0.02, and 0.03 respectively. The fourth loss sub-function is 0.01+0.02+0.03=0.06.

[0359] In this embodiment, the calculation process of determining the fourth loss sub-function is shown in Formula 4:

[0360]

[0361] In formula 4, L tag1 represents the fourth loss subfunction. n1 represents that there are n1 non-confusable sample texts among bs sample texts. i represents the i-th sample text. j represents the j-th sentiment classification. M represents that there are M sentiment classifications in total. t[i] j Indicates the first sentiment classification label of the jth sentiment classification of the i-th sample text. o[i] j It represents the fifth positive discriminant probability of the jth sentiment category of the i-th sample text. When the true value of the first sentiment category label of a certain sentiment category is 1, the left side of the plus sign of formula 4 takes effect, and when the true value is 0, the right side of formula 4 takes effect, so the information of a certain sample text under all labels can be learned.

[0362] The advantage of this embodiment is that the accuracy of determining the fourth loss sub-function is improved.

[0363] An advantage of the embodiment of steps 2610 - 2620 is that the accuracy of determining the fourth loss sub-function is improved.

[0364] In step 2160, a second loss function is calculated based on the second loss sub-function, the third loss sub-function, and the fourth loss sub-function. The second loss function indicates the joint discrimination error of the multi-label sentiment classification model for the first easily confused sample text, the second easily confused sample text, and the non-confusable sample text.

[0365] In one embodiment, referring to Fig. 27 , step 2160 includes:

[0366] Step 2710: Obtain a sixth weight corresponding to the second loss sub-function, a seventh weight corresponding to the third loss sub-function, and an eighth weight corresponding to the fourth loss sub-function;

[0367] Step 2720: Based on the sixth weight, the seventh weight, and the eighth weight, calculate a fourth weighted sum of the second loss sub-function, the third loss sub-function, and the fourth loss sub-function as the second loss function.

[0368] In step 2710, the sixth weight indicates the importance of the second loss sub-function. The seventh weight indicates the importance of the third loss sub-function. The eighth weight indicates the importance of the fourth loss sub-function. The sixth weight, the seventh weight, and the eighth weight can be set according to actual needs.

[0369] In one embodiment, step 2710 includes:

[0370] 1.5 times the first proportion of the first easily confused sample text in the matching sample text set is determined as the sixth weight;

[0371] 0.5 times the second proportion of the second most confusing sample text in the matching sample text set is determined as the seventh weight;

[0372] The third proportion of the non-confusable sample text in the matching sample text set is determined as the eighth weight.

[0373] Specifically, in labels with different confusion levels, different loss calculation methods are used for samples of each label, and the weighted sum of the mean losses of all samples in the batch is finally calculated as the total loss. Among them, w1, w2, and w3 represent the weights of three types of data (non-confusable label sample text, first easily confused sample text, and second easily confused sample text). Assuming that there are bs samples in total, n1, n2, and n3 are the number of samples of the three types of data, then w1 is set to n1 / bs, w2 is 1.5*n2 / bs, and w3 is 0.5*n3 / bs.

[0374] In one example, the matching sample text set has a total of 1000 sample texts, specifically including 200 first easily confused sample texts, 300 second easily confused sample texts, and 500 non-easy to confuse sample texts. The first proportion of the first easily confused sample text is 0.2, then the sixth weight is 0.2*1.5=0.3. The second proportion of the second easily confused sample text is 0.3, then the sixth weight is 0.3*0.5=0.15. The third proportion of the non-easy to confuse sample text is 0.5, then the eighth weight is 0.5.

[0375] The advantage of this embodiment is that the determination of the weight is related to the proportion of the sample text in the matching sample text set, and has high accuracy and applicability.

[0376] In step 2720, the second loss function is obtained by summing the product of the sixth weight and the second loss sub-function, the product of the seventh weight and the third loss sub-function, and the product of the eighth weight and the fourth loss sub-function.

[0377] In one embodiment, the calculation process of the second loss function is shown in Formula 5:

[0378] L total =w1*L tag1 +w2*L reg2 +w3*L reg3 (Formula 5).

[0379] In Formula 5, L total refers to the second loss function, w1 refers to the sixth weight, L tag1 refers to the fourth loss sub-function, w2 refers to the seventh weight, L reg2 refers to the second loss sub-function, w3 refers to the eighth weight, L reg3 Refers to the third loss sub-function. In one example, assuming that the second loss sub-function is 0.2, the sixth weight is 0.2, the third loss sub-function is 0.3, the fifth weight is 0.4, the fourth loss sub-function is 0.4, and the eighth weight is 0.4, then the second loss function is 0.2*0.2+0.3*0.4+0.4*0.4=0..32.

[0380] The advantage of the embodiment of steps 2710-2720 is that the sixth weight, the seventh weight and the eighth weight are introduced, which improves the flexibility and accuracy of determining the second loss function and has high applicability.

[0381] Detailed description of the text multi-label sentiment classification method of the present disclosure

[0382] In one embodiment, referring to Fig.28 The text multi-label sentiment classification method of the embodiment of the present disclosure includes:

[0383] Step 2810: Obtain target text;

[0384] Step 2820: input the target text into the multi-label sentiment classification model, and obtain a first sentiment classification result for each sentiment classification.

[0385] In step 2810, the target text refers to the text to be identified and classified. The text conversion method in the related art can be used to directly extract the text from the PDF document to form the target text. The scanned document can also be converted into an image, and then the text is extracted using the optical character recognition (OCR) technology to form the target text. The speech recognition technology can also be used to convert the speech into text to form the target text.

[0386] In step 2820, the first sentiment classification result is used to indicate whether the target text belongs to the sentiment classification. For example, the target text "I am very happy" is input into the multi-label sentiment classification model, and the first sentiment classification result includes (happy, 1), (expected, 0), and (surprised, 1), which means that "I am very happy" belongs to "happy" and "surprised", but not to "expected". The multi-label sentiment classification model is trained by the multi-label sentiment classification model training method of the above embodiment. Since the training process has been detailed in the previous text, it will not be repeated here.

[0387] The advantage of the embodiment of steps 2810-2820 is that since the multi-label sentiment classification model is trained based on multi-model learning and data cleaning, it has higher accuracy, thereby improving the accuracy of multi-label sentiment classification of text.

[0388] Referring to the above, the multi-label sentiment classification model includes a basic learning model, a sentiment detail learning model and a classification layer. Fig.29 , step 2820 includes:

[0389] Step 2910, converting the target text into a target text vector;

[0390] Step 2920: Input the target text vector into the basic learning model to obtain a basic semantic feature vector;

[0391] Step 2930: input the basic semantic feature vector into the emotion detail learning model to obtain a high-order emotion feature vector;

[0392] Step 2940: input the high-order sentiment feature vector into the classification layer, and obtain the first sentiment classification result for each sentiment classification.

[0393] Fig. 30A It is an overall structural diagram of a text multi-label sentiment classification method according to an embodiment of the present invention.

[0394] Fig. 30A The leftmost side of is the target text in step 2910. In step 2910, the target text vector refers to the vector corresponding to the target text. The target text is a character-type feature, and the character-type feature can be input into the embedding layer to obtain the target text vector corresponding to the character-type feature. Since the character-type feature is a sequence of characters, each character is regarded as a discrete symbol, which is not recognizable by the machine. The role of the embedding layer is to convert discrete characters into vectors that can be recognized by the machine.

[0395] The character features are input into the embedding layer, and the embedding layer will output the corresponding unique target text vector for the character features. For example, after the character feature "student" is input into the embedding layer, the first feature vector is [1,0,1,0,...,1,1,0]; after the character feature "popular" is input into the embedding layer, the first feature vector is [1,1,0,1,...,1,0,0].

[0396] After obtaining the target text vector, execute step 2920. The basic learning model is a model for analyzing / recognizing the semantic information contained in the vector. The basic semantic feature vector refers to a vector with emotional semantic information (such as information that the text belongs to a certain emotional category). The basic learning model can be a common deep neural network, such as the Bert model.

[0397] After obtaining the basic semantic feature vector, execute step 2930. The emotional detail learning model is a model for analyzing / identifying the emotional detail information contained in the vector. The high-order emotional feature vector refers to a vector with high-order emotional semantic information (such as information that the text belongs to a certain emotional category and the emotional intensity corresponding to the emotional category). The emotional detail learning model can be a common deep neural network, such as the Bert model.

[0398] After obtaining the high-order emotion feature vector, step 2940 is executed. The classification layer is a layer for classifying vectors into different groups / categories. The high-order emotion feature vector is input into the classification layer to obtain the first emotion classification result corresponding to each emotion classification. In one example, Fig. 30B As shown, the high-order sentiment feature vector Q1 is input into the fully connected layer (the classification layer is the fully connected layer) to obtain the corresponding sentiment classification result corresponding to each sentiment classification.

[0399] It should be noted that a fully connected layer is a common layer in a neural network, also known as a dense layer. In a fully connected layer, each neuron is connected to all neurons in the previous layer, so each input has an effect on each neuron. Fully connected layers are usually used in the last few layers of a neural network to combine and integrate the features extracted by the previous layers and finally output the network's prediction results. Other types of layers can also be used to replace fully connected layers, such as convolutional layers and pooling layers.

[0400] The advantage of the embodiment of steps 2910-2940 is that the basic learning model, the sentiment detail learning model and the classification layer are used to jointly perform sentiment classification, thereby improving the accuracy of multi-label sentiment classification of text.

[0401] In one embodiment, referring to Fig.31 , step 2910 includes:

[0402] Step 3110: Segment the target text;

[0403] Step 3120: Assign position markers to the segmented words;

[0404] Step 3130: Input the segmented words and their corresponding position markers into the embedding layer to obtain the first word sub-vectors;

[0405] Step 3140: Connect the first word sub-vectors in the order of the words in the target text to obtain the target text vector.

[0406] In step 3110, the target text is a word sequence composed of multiple words. Word segmentation refers to separating the target text to obtain multiple discrete words.

[0407] In one embodiment, step 3110 includes:

[0408] Add a guiding statement to the target text;

[0409] Segment the guiding statement and the target text.

[0410] The guiding statement refers to the text that guides the model to perform sentiment classification on the input text. For example, if the target text is "I am at home today" and the guiding statement is "Please give the speaker's sentiment based on the following text", then the target text is updated to "Please give the speaker's sentiment based on the following text, I am at home today". By adding a guiding statement to the target text, it can guide the multi-label sentiment classification model to perform sentiment classification on the target text, reduce instability, and improve the accuracy of sentiment classification.

[0411] In one example, as Fig.32 shown, the target text is "The dog is very cute and it likes to play.", and after segmentation, it gets {dog, very, cute, SEP, it, likes, to, play, SEP}. Here, SEP indicates the separator. Fig.32 It also shows [CLS], which indicates the sentiment classification label of the input text. It should be understood that in actual use of the model, generally, the sentiment classification label of the target text cannot be known in advance, so [CLS] is empty.

[0412] In step 3120, the position marker indicates the position of the word in the target text. For example, the position marker of "[CLS]" is 0, the position marker of "dog" is 1, the position marker of "very" is 2, the position marker of "cute" is 3, the position marker of "SEP" is 5, the position marker of "it" is 6, the position marker of "likes" is 7, the position marker of "to" is 8, the position marker of "play" is 9, and the position marker of the second "SEP" is 10.

[0413] In step 3130, the first word sub-vector refers to the vector corresponding to the word and the position marker. The first word sub-vector is obtained by adding the word embedding corresponding to the word and the position embedding corresponding to the position marker. As Fig.32 shown, the word embedding corresponding to "[CLS]" is E [CLS] , the word embedding corresponding to "dog" is E [狗] , the word embedding corresponding to "very" is E [很] , the word embedding corresponding to "can" is E [可] , the word embedding corresponding to "love" is E [爱] , the word embedding corresponding to "SEP" is E [SEP] , the word embedding corresponding to "it" is E [它] , the word embedding corresponding to "like" is E [喜] , the word embedding corresponding to "happy" is E [欢] , the word embedding corresponding to "play" is E [玩]9 , the word embedding corresponding to "SEP" is E [SEP] . The position embedding corresponding to the position marker "0" is E0, the position embedding corresponding to the position marker "1" is E1, the position embedding corresponding to the position marker "2" is E2, the position embedding corresponding to the position marker "3" is E3, the position embedding corresponding to the position marker "4" is E4, the position embedding corresponding to the position marker "5" is E5, the position embedding corresponding to the position marker "6" is E6, the position embedding corresponding to the position marker "7" is E7, the position embedding corresponding to the position marker "8" is 8, the position embedding corresponding to the position marker "9" is E9, the position embedding corresponding to the position marker "10" is E 10 . The first word sub-vector corresponding to "[CLS]" is E [CLS] +E0. The first word sub-vector corresponding to "dog" is E [狗] +E1. The first word sub-vectors corresponding to the remaining words can be obtained in the same way and will not be elaborated here.

[0414] In step 3140, multiple first word sub-vectors are concatenated / concatenated in the order of the words in the target text to obtain the target text vector.

[0415] In one embodiment, step 3140 includes:

[0416] Determine the number of words into which the guiding language and the target text are divided;

[0417] If the number of words is less than the first number, determine the first difference between the first number and the number of words;

[0418] Generate the first difference number of second word sub-vectors;

[0419] The first word sub-vector is connected according to the order of words in the target text, and is connected with the first difference second word sub-vector to obtain a target text vector, wherein each of the first word sub-vector and the second word sub-vector is used as a word sub-vector.

[0420] In this embodiment, in order to have a standardized model input (so that it can respond to text inputs of different lengths), it is necessary to ensure that the target text vector contains a first number (for example, 77) of word sub-vectors. First, determine the number of words in the guide and the target text segmentation. Generally, the number of words is less than the first number (if the number of words is greater than the first number, the first number can be increased). If the number of words is less than the first number, it means that the number of the first word sub-vectors is less than the first number, so that the number of word sub-vectors contained in the target text vector is less than the first number. To this end, this embodiment determines the first difference between the first number and the number of words, and then generates the first difference second word sub-vectors. For example, if the number of words is 30, the first difference is 77-30=47, and 47 "0"s are used as 47 second word sub-vectors. Finally, based on the number of words, the first word sub-vectors and the first difference, the second word sub-vectors, the target text vector is obtained. The target text vector contains the first number of word sub-vectors.

[0421] The advantage of this embodiment is that it can ensure that the dimension of the target text vector is unified, so that the input of the model can be standardized, and the applicability of the model to texts of different lengths is improved.

[0422] The advantage of the embodiment of steps 3110-3140 is that the first word sub-vector is determined based on the vector corresponding to the word and the vector corresponding to the word position mark, thereby obtaining the target text vector, thereby improving the ability to carry information under the target text and improving the accuracy of the target text vector.

[0423] In one embodiment, the basic learning model includes a second number of first transformation sub-model layers cascaded up and down, and each first transformation sub-model layer includes a first number of first transformation sub-models. Fig.33 , step 2920 includes:

[0424] Step 3310: for each word sub-vector in the target text vector, input the word sub-vector into a first number of first transformation sub-models of the first transformation sub-model layer of the lowest layer;

[0425] Step 3320: except for the first transformation submodel layer of the uppermost layer, input the output subvector of each first transformation submodel in the first transformation submodel layer of each layer into the first number of first transformation submodels in the first transformation submodel layer of the upper layer;

[0426] Step 3330: cascade the basic semantic feature subvectors output by the first third number of first transformation submodels in the top first transformation submodel layer to obtain a basic semantic feature vector, where the third number is the number of words in the guide.

[0427] In this embodiment, if Fig.34 As shown, the basic learning model includes a second number ( Fig.34 3) first transformation sub-model layers cascaded up and down, each first transformation sub-model layer includes a first number of ( Fig.34 3) first transformation sub-models (such as Fig.34 Trm shown). The first transformation sub-model layer refers to a layer containing a first number of first transformation sub-models. The first transformation sub-model refers to a model that performs vector transformation on a word sub-vector. In one example, it is assumed that the basic learning model includes 3 first transformation sub-model layers. The output sub-vector of each first transformation sub-model in the first transformation sub-model layer of the 1st layer is input into the first number of first transformation sub-models in the first transformation sub-model layer of the 2nd layer; the output sub-vector of each first transformation sub-model in the first transformation sub-model layer of the 2nd layer is input into the first number of first transformation sub-models in the first transformation sub-model layer of the 3rd layer; the basic semantic feature sub-vectors output by the first third number of first transformation sub-models in the first transformation sub-model layer of the 3rd layer are cascaded to obtain a basic semantic feature vector.

[0428] like Fig.35A As shown in FIG. 1 , in one example, the basic learning model includes two first transformation sub-model layers. Each first transformation sub-model layer includes N Trms. The word sub-vectors in the target text vector are represented as E1, E2, ..., E N The N word sub-vectors are input into the N first transformation sub-models of the first transformation sub-model layer of the first layer. The output sub-vector of each Trm in the first transformation sub-model layer of the first layer is input into the N Trms in the first transformation sub-model layer of the second layer, and the N basic semantic feature sub-vectors are obtained, which are represented as T1, T2, ..., T N The basic semantic feature subvectors output by the first third number (the third number is less than M=N) of Trms in the first transformation submodel layer of the second layer are cascaded to obtain a basic semantic feature vector.

[0429] like Fig.35BAs shown, in one embodiment, Trm includes a multi-head attention model, a first residual connection layer, a first normalization layer, a feedforward network, a second residual connection layer, and a second normalization layer. The multi-head attention model refers to a model that has the ability to focus on and concentrate on specific learning when processing information. In this embodiment, the multi-head attention model is used to extract features from the target text vector, aiming to select / screen information that is conducive to emotional semantics between features from the word sub-vectors of the target text vector. Common multi-head attention models include self-attention models, convolutional attention models, and attention mechanisms. The first residual connection layer refers to a layer that adds the input vector and the processed input vector. The first normalization layer refers to a layer that normalizes the vector. In this embodiment, the first normalization layer is used to scale the vector elements of the vector to a range of 0 to 1. Feedforward network (Feed forward Neural Network) is a basic artificial neural network model. The feedforward network consists of a multi-layer structure composed of multiple neurons. Each layer of neurons is connected to the next layer, and information is transmitted from the input layer to the output layer through the hidden layer without forming a loop. In this embodiment, a feedforward network is used to add nonlinear information so that the output vector can better represent the complex nonlinear relationship between features. The second residual connection layer refers to a layer that adds the input vector and the processed input vector. The second normalization layer refers to a layer that normalizes the vector. In this embodiment, the second normalization layer is used to scale the vector elements of the vector to a range of 0 to 1.

[0430] exist Fig.35B In , the target text is input into the embedding layer as input to obtain multiple word embeddings. The word embedding is added to the position embedding to obtain the word sub-vector. The word sub-vector is input into Trm. After passing through the multi-head attention model, the first residual connection layer, the first normalization layer, the feedforward network, the second residual connection layer, and the second normalization layer, the output sub-vector output by Trm is obtained.

[0431] In one example, if Fig.35CAs shown, the multi-head attention model includes a first linear layer, a second linear layer, a third linear layer, h scale point product attention layers, a splicing layer and a fourth linear layer. The first linear layer performs a linear transformation on the input vector before entering the first weight matrix (V matrix). The second linear layer performs a linear transformation on the input vector before entering the second weight matrix (K matrix). The third linear layer performs a linear transformation on the input vector before entering the third weight matrix (Q matrix). The performance of the multi-head attention model can be further improved. Each scale point product attention layer performs feature attention on the input vector separately to obtain an attention score. The attention score can represent the feature information of each input vector itself. The attention score vector composed of multiple attention scores can represent the contextual semantic information between each vector on the basis of retaining the feature information of each input vector itself. The splicing layer is used to form an attention score vector with the h attention scores output by the h scale point product attention layers. The fourth linear layer applies a linear transformation to the attention score vector, further improving the performance of the multi-head attention model.

[0432] like Fig.36 As shown, the basic learning model can be used for question answering tasks, named entity recognition tasks, and sentence classification tasks. The text multi-label sentiment classification method of the embodiment of the present disclosure belongs to the question answering task. Fig.36 In the above example, the guide and target text are segmented into {Token1,......,TokenN,[SEP],Token1,......,TokenM}, where [CLS] indicates the sentiment classification label of the target text. The segmented words and the corresponding position tags are input into the embedding layer, and the target text vector is [E1,......,E N ,E [SEP] ,E1',......,E M '], where E [CLS] It may exist or be empty. Input the target text vector into the basic learning model, and obtain the basic semantic feature sub-vectors as T1, ..., T N , T [SEP] , T1', ..., T M '. Cascade the first N basic semantic feature sub-vectors to obtain the basic semantic feature vector [T1T2......T N ]. Star / endspan indicates the number of characters between the start character position and the end character position (for example, M).

[0433] The advantage of the embodiment of steps 3310-3330 is that the target text vector contains the vectors corresponding to the guide words and the target text, and the interaction and fusion between the guide words and the text can be fully realized by using the first transformation sub-model layer, so that each basic semantic feature sub-vector has rich contextual semantic information, so that the basic semantic feature vector determined based on the first third number of basic semantic feature sub-vectors has high accuracy. In addition, since the guide words generally do not change, the length of the basic semantic feature vector determined is basically unchanged, which improves the input specification of the vector input to the emotional detail learning model.

[0434] In one embodiment, the emotional detail learning model includes a fourth number of second transformation sub-model layers cascaded up and down, each second transformation sub-model layer includes a third number of second transformation sub-models, and the third number is the number of basic semantic feature sub-vectors in the basic semantic feature vector. Fig.37 , step 2930 includes:

[0435] Step 3710: for each basic semantic feature sub-vector in the basic semantic feature vector, input the basic semantic feature sub-vector into a third number of second transformation sub-models of the second transformation sub-model layer of the lowest layer;

[0436] Step 3720: except for the second transformation sub-model layer of the uppermost layer, input the output sub-vector of each second transformation sub-model in the second transformation sub-model layer of each layer into the third number of second transformation sub-models in the second transformation sub-model layer of the upper layer;

[0437] Step 3730: cascade the high-order emotion feature sub-vectors output by the third number of second transformation sub-models in the second transformation sub-model layer of the top layer to obtain a high-order emotion feature vector.

[0438] In this embodiment, if Fig.38 As shown, the emotion detail learning model includes the fourth number ( Fig.38 3) cascaded second transformation sub-model layers are shown, each of which includes a third number of ( Fig.38 3) second transformation sub-models (such as Fig.38 The second transformation sub-model layer refers to a layer including a third number of second transformation sub-models. The second transformation sub-model refers to a model for performing vector transformation on the basic semantic feature sub-vector.

[0439] like Fig.39 As shown in FIG. 1 , in one example, the emotion detail learning model includes two second transformation sub-model layers. Each second transformation sub-model layer includes N Trms. Each basic semantic feature sub-vector in the basic semantic feature vector is represented as T1, T2, ..., T NThe N basic semantic feature sub-vectors are input into the N Trms of the second transformation sub-model layer of the first layer. The output sub-vector of each Trm in the first transformation sub-model layer of the first layer is input into the N Trms in the second transformation sub-model layer of the second layer, and N high-order sentiment feature sub-vectors are obtained, which are represented as Q1, Q2, ..., Q N . Cascade N high-order sentiment feature sub-vectors to obtain a high-order sentiment feature vector [Q1, Q2, ..., Q N ].

[0440] The advantage of the embodiment of steps 3710-3730 is that the second transformation sub-model layer can fully realize the interaction and fusion between the basic semantic feature sub-vectors, so that each high-order emotional feature sub-vector has rich contextual semantic information, so that the high-order emotional feature vector determined based on the third number of high-order emotional feature sub-vectors has high accuracy. In addition, since the guide words generally do not change, the length of the determined high-order emotional feature vector remains basically unchanged, which improves the input specification of the vector input to the classification layer.

[0441] In one embodiment, referring to Fig.40 , step 2920 includes: step 4010, inputting the target text vector into the basic learning model to obtain a basic semantic feature vector and a second sentiment classification result corresponding to each sentiment classification;

[0442] After step 2940, the text multi-label sentiment classification method further includes: step 4020, integrating the first sentiment classification result and the second sentiment classification result to obtain a third sentiment classification result corresponding to each sentiment classification.

[0443] like Fig.41 As shown, the basic learning model outputs the basic semantic feature vector (composed of T1, T2, ..., T N In addition to the splicing), the second sentiment classification result corresponding to each sentiment classification is also output ( Fig.41 C) shown in FIG. 1 may integrate the corresponding sentiment classification result with the second sentiment classification result to obtain a third sentiment classification result.

[0444] In one embodiment, step 4020 includes:

[0445] For the first sentiment classification result, obtaining a first positive discriminant probability corresponding to the sentiment classification;

[0446] For the second sentiment classification result, obtaining a second positive discrimination probability corresponding to the sentiment classification discriminated by the basic learning model;

[0447] Obtaining a first weight corresponding to the first positive discriminant probability and a second weight corresponding to the second positive discriminant probability;

[0448] Determining a first weighted sum of a first positive discriminant probability and a second positive discriminant probability based on the first weight and the second weight;

[0449] Based on the first weighted sum, a third sentiment classification result corresponding to the sentiment classification is determined.

[0450] In one example, if Fig.42 As shown, for the first emotion classification result, the first positive discrimination probability corresponding to "Shock 1" is 0.8, and the first positive discrimination probabilities corresponding to the other emotion classifications are all 0. For the second emotion classification result, the second positive discrimination probability corresponding to "Nothing" is 0.9, and the second positive discrimination probabilities corresponding to the other emotion classifications are all 0. Assume that the first weight is 0.6 and the second weight is 0.4. Then the first weighted sum of "Shock 1" is 0.6*0.8+0.4*0=0.48, and the first weighted sum of "Nothing" is 0.6*0+0.4*0.9=0.36. Assume that the first discrimination threshold is 0.4. Since 0.48 is greater than 0.4, the third emotion classification result of "Shock 1" is 1. Since 0.36 is less than 0.4, the third emotion classification result of "Nothing" is 0. The first weighted sums of other emotion classifications are all 0, so the corresponding third emotion classification results are all 0.

[0451] It should be noted that the first weight and the second weight can be set according to actual needs and are not specifically limited in this embodiment.

[0452] The advantage of this embodiment is that the third sentiment classification result is determined by using the first weight and the second weight together, thereby improving the accuracy of the determination. Since the first weight and the second weight can be flexibly set, it also has high flexibility.

[0453] The advantage of the embodiment of steps 4010-4020 is that, in addition to the first sentiment classification result, the second sentiment classification result determined by the basic learning model can also be used to determine the third sentiment classification result of the target text, thereby improving the diversity and flexibility of sentiment classification.

[0454] Description of the apparatus and device of the present disclosure

[0455] It is to be understood that, although the steps in the above-mentioned flowcharts are sequentially displayed according to the characterization of arrows, these steps are not necessarily executed in sequence according to the order of arrow characterization. Unless there is a clear description in the present embodiment, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the above-mentioned flowcharts can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0456] Fig.43 A schematic diagram of the structure of a multi-label sentiment classification model training device 4300 provided in an embodiment of the present disclosure. The multi-label sentiment classification model training device 4300 includes:

[0457] The first input unit 4310 is used to input the sample text in the initial sample text set into multiple multi-label sentiment classification models to obtain the maximum probability sentiment classification corresponding to each multi-label sentiment classification model;

[0458] A cleaning unit 4320 is used to screen out matching sample texts from the initial sample text set based on a comparison between the first dominant sentiment classification label of the sample text and the maximum probability sentiment classification corresponding to each multi-label sentiment classification model, and the screened matching sample texts form a matching sample text set;

[0459] The second input unit 4330 is used to input the matching sample text into the multi-label sentiment classification model to obtain a first positive discriminant probability corresponding to each sentiment classification;

[0460] A calculation unit 4340 is used to calculate a first loss function corresponding to each multi-label sentiment classification model based on a first sentiment classification label of each sentiment classification and a first positive discriminant probability;

[0461] A first training unit 4350 is used to train the multi-label sentiment classification model based on a first loss function corresponding to the multi-label sentiment classification model for each multi-label sentiment classification model;

[0462] Return unit 4360 is used to replace the initial sample text set with the matching sample text set, and return to the step of inputting the sample texts in the initial sample text set into multiple multi-label sentiment classification models until the first loss function meets the predetermined conditions, and then determine the trained multi-label sentiment classification model from the multiple multi-label sentiment classification models.

[0463] Optionally, the multi-label sentiment classification model includes a sentiment detail learning model and a classification layer; after determining a trained multi-label sentiment classification model from a plurality of multi-label sentiment classification models, the multi-label sentiment classification model training device 4300 further includes a second training unit (not shown) for:

[0464] Divide a plurality of first sentiment classification labels of each sentiment classification of the matching sample text into first-category labels, second-category labels, and non-sentiment labels, wherein the degree of confusion between the first-category labels and the non-sentiment labels is less than the degree of confusion between the second-category labels and the non-sentiment labels;

[0465] In the matching sample text set, a first sample text with a second type of label and a second sample text with a non-emotional label are obtained;

[0466] Obtaining a first easily confused sample text from the first sample text, obtaining a second easily confused sample text from the second sample text, and determining other sample texts other than the first easily confused sample text and the second easily confused sample text in the matching sample text set as non-easy to confuse sample texts;

[0467] Based on the loss sub-functions of the first easily confused sample text, the second easily confused sample text and the non-easy confused sample text, a second loss function is determined, and the sentiment detail learning model and the classification layer in the trained multi-label sentiment classification model are adjusted based on the second loss function.

[0468] Fig.44 The structure diagram of a text multi-label sentiment classification device 4400 provided in an embodiment of the present disclosure. The text multi-label sentiment classification device 4400 includes:

[0469] An acquisition unit 4410 is used to acquire a target text;

[0470] The third input unit 4420 is used to input the target text into the multi-label sentiment classification model, and obtain a first sentiment classification result for each sentiment classification, where the first sentiment classification result is used to indicate whether the target text belongs to the sentiment classification;

[0471] The multi-label sentiment classification model is trained by the following process: inputting sample texts in the initial sample text set into multiple multi-label sentiment classification models to obtain the maximum probability sentiment classification corresponding to each multi-label sentiment classification model; based on the comparison of the first dominant sentiment classification label of the sample text and the maximum probability sentiment classification corresponding to each multi-label sentiment classification model, screening out matching sample texts from the initial sample text set, and the screened matching sample texts constitute a matching sample text set; inputting the matching sample texts into the multi-label sentiment classification model to obtain the first positive discrimination probability corresponding to each sentiment classification; based on the first sentiment classification label of each sentiment classification and the first positive discrimination probability, calculating the first loss function corresponding to each multi-label sentiment classification model; for each multi-label sentiment classification model, training the multi-label sentiment classification model based on the first loss function corresponding to the multi-label sentiment classification model; replacing the initial sample text set with the matching sample text set, and returning to the step of inputting the sample texts in the initial sample text set into multiple multi-label sentiment classification models, until the first loss function satisfies a predetermined condition, and then determining the trained multi-label sentiment classification model from the multiple multi-label sentiment classification models.

[0472] Optionally, inputting the target text vector into the basic learning model to obtain a basic semantic feature vector includes: inputting the target text vector into the basic learning model to obtain the basic semantic feature vector and a second sentiment classification result corresponding to each sentiment classification;

[0473] After inputting the target text into the multi-label sentiment classification model to obtain the first sentiment classification result corresponding to each sentiment classification, the text multi-label sentiment classification device 4400 also includes: an integration unit (not shown) for integrating the first sentiment classification result and the second sentiment classification result to obtain a third sentiment classification result corresponding to each sentiment classification.

[0474] Reference Fig.45 , Fig.45 The structural block diagram of the terminal part for implementing the multi-label sentiment classification model training method and the text multi-label sentiment classification method of the embodiment of the present disclosure includes: a radio frequency (RF) circuit 4510, a memory 4515, an input unit 4530, a display unit 4540, a sensor 4550, an audio circuit 4560, a wireless fidelity (WiFi) module 4570, a processor 4580, and a power supply 4590. Those skilled in the art can understand that Fig.45 The terminal structure shown does not constitute a limitation on the mobile phone or computer, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0475] RF circuit 4510 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information from the base station, it is sent to processor 4580 for processing; in addition, the designed uplink data is sent to the base station.

[0476] The memory 4515 may be used to store software programs and modules. The processor 4580 executes various functional applications and data processing of the content terminal by running the software programs and modules stored in the memory 4515 .

[0477] The input unit 4530 may be used to receive input digital or character information and generate key signal input related to the setting and function control of the content terminal. Specifically, the input unit 4530 may include a touch panel 4531 and other input devices 4532.

[0478] The display unit 4540 may be used to display input information or provided information and various menus of the content terminal. The display unit 4540 may include a display panel 4541.

[0479] The audio circuit 4560, the speaker 4561, and the microphone 4562 may provide an audio interface.

[0480] In this embodiment, the processor 4580 included in the terminal can execute the multi-label sentiment classification model training method and the text multi-label sentiment classification method of the previous embodiment.

[0481] The embodiments of the present disclosure can be applied to various scenarios, including but not limited to data processing, artificial intelligence, film and television production, etc.

[0482] Fig.46 A structural block diagram of part of a server for implementing the multi-label sentiment classification model training method and text multi-label sentiment classification method of the embodiments of the present disclosure. The server may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 4622 (for example, one or more processors) and memory 4632, and one or more storage media 4630 (for example, one or more mass storage devices) storing application programs 4642 or data 4644. Among them, the memory 4632 and the storage medium 4630 can be short-term storage or permanent storage. The program stored in the storage medium 4630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 4622 can be configured to communicate with the storage medium 4630 and execute a series of instruction operations in the storage medium 4630 on the server.

[0483] The server may also include one or more power supplies 4626, one or more wired or wireless network interfaces 4650, one or more input and output interfaces 4658, and / or, one or more operating systems 4641, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0484] The central processor 4622 in the server can be used to execute the multi-label sentiment classification model training method and the text multi-label sentiment classification method of the embodiments of the present disclosure.

[0485] The embodiments of the present disclosure also provide a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the multi-label sentiment classification model training method and the text multi-label sentiment classification method of each of the aforementioned embodiments.

[0486] The present disclosure also provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, so that the computer device implements the multi-label sentiment classification model training method and the text multi-label sentiment classification method.

[0487] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar contents, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0488] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated content, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the associated content before and after is in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0489] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0490] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0491] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0492] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0493] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server 130, or network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0494] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0495] The above is a specific description of the implementation methods of the present disclosure, but the present disclosure is not limited to the above implementation methods. Technical personnel familiar with the art can also make various equivalent modifications or substitutions without violating the spirit of the present disclosure. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A multi-label sentiment classification model training method, characterized in that: include: Inputting sample texts in the initial sample text set into multiple multi-label sentiment classification models to obtain the maximum probability sentiment classification corresponding to each of the multi-label sentiment classification models; Based on the comparison between the first dominant sentiment classification label of the sample text and the maximum probability sentiment classification corresponding to each of the multi-label sentiment classification models, matching sample texts are screened out from the initial sample text set, and the screened matching sample texts constitute a matching sample text set; Inputting the matching sample text into each of the multi-label sentiment classification models to obtain a first positive discriminant probability corresponding to each sentiment classification; Based on the first sentiment classification label of each of the sentiment classifications and the first positive discriminant probability, calculating a first loss function corresponding to each of the multi-label sentiment classification models; For each of the multi-label sentiment classification models, training the multi-label sentiment classification model based on the first loss function corresponding to the multi-label sentiment classification model; The initial sample text set is replaced by the matching sample text set, and the step of inputting the sample texts in the initial sample text set into the multiple multi-label sentiment classification models is returned to until the first loss function satisfies a predetermined condition, and then a trained multi-label sentiment classification model is determined from the multiple multi-label sentiment classification models.

2. The multi-label sentiment classification model training method according to claim 1, characterized in that: The step of determining a trained multi-label sentiment classification model from a plurality of the multi-label sentiment classification models comprises: Acquire a third test text set, wherein the third test text set includes a plurality of third test texts, and the third test texts have a second dominant sentiment classification label; Inputting the third test text into each of the multi-label sentiment classification models to obtain a maximum probability sentiment classification result of each of the multi-label sentiment classification models; Determine a second accuracy rate of each of the multi-label sentiment classification models based on a comparison between the second dominant sentiment classification label and the maximum probability sentiment classification result of each of the multi-label sentiment classification models; Based on the second accuracy of each of the multi-label sentiment classification models, the trained multi-label sentiment classification model is determined.

3. The multi-label sentiment classification model training method according to claim 1, characterized in that: The step of inputting the sample text in the initial sample text set into a plurality of the multi-label sentiment classification models to obtain the maximum probability sentiment classification corresponding to each of the multi-label sentiment classification models comprises: For each of the multi-label sentiment classification models, inputting the sample text in the initial sample text set into the multi-label sentiment classification model to obtain the first positive discriminant probability corresponding to each of the sentiment classifications; The sentiment classification with the largest first positive discriminant probability is determined as the maximum probability sentiment classification corresponding to the multi-label sentiment classification model.

4. The multi-label sentiment classification model training method according to claim 3, characterized in that: The determining the sentiment classification with the maximum first positive discrimination probability as the maximum probability sentiment classification corresponding to the multi-label sentiment classification model includes: Obtaining multiple discrimination thresholds of the sentiment classification; The sentiment classification whose first positive discrimination probability is greater than the discrimination threshold is used as a candidate sentiment classification; The candidate sentiment classification with the maximum first positive discriminant probability is determined as the maximum probability sentiment classification corresponding to the multi-label sentiment classification model.

5. The multi-label sentiment classification model training method according to claim 1, characterized in that: The multi-label sentiment classification model includes a sentiment detail learning model and a classification layer; After determining the trained multi-label sentiment classification model from the plurality of multi-label sentiment classification models, the multi-label sentiment classification model training method further comprises: Divide the first emotion classification label of each emotion classification of the matching sample text into a first category label, a second category label, and a non-emotion label, wherein the confusion degree between the first category label and the non-emotion label is less than the confusion degree between the second category label and the non-emotion label; From the matching sample text set, obtaining a first sample text with the second category label and a second sample text with the emotion-free label; Acquire a first easily confused sample text from the first sample text, acquire a second easily confused sample text from the second sample text, and determine other sample texts in the matching sample text set except the first easily confused sample text and the second easily confused sample text as non-easy to confuse sample texts; Based on the loss sub-functions of the first easily confused sample text, the second easily confused sample text and the non-easy confused sample text, a second loss function is determined, and based on the second loss function, the sentiment detail learning model and the classification layer in the trained multi-label sentiment classification model are adjusted.

6. The multi-label sentiment classification model training method according to claim 5, characterized in that: The plurality of first emotion classification labels include the no emotion label and various emotion classification labels of various emotion intensity levels; The step of dividing the first emotion classification label of each emotion classification of the matching sample text into a first category label, a second category label, and a non-emotion label comprises: Extracting the non-emotional label from the plurality of the first emotion classification labels; For each emotion classification label of each emotion intensity, input the second test text with the emotion classification label in the second test text set into the multi-label emotion classification model to obtain a sixth emotion classification result; Based on the comparison between the sixth emotion classification result and the emotion classification label, the emotion classification label is divided into the first category label or the second category label.

7. The multi-label sentiment classification model training method according to claim 6, characterized in that: The step of classifying the emotion classification label into the first category label or the second category label based on the comparison between the sixth emotion classification result and the emotion classification label comprises: Determine the number of second test texts with the sentiment classification label in the second test text set as the fifth number; Determine the number of second test texts in the second test text set with the sentiment classification label whose sixth sentiment classification result is inconsistent with the sentiment classification label as the sixth number; determining the confusability based on the fifth number and the sixth number; If the degree of confusion is less than a first threshold, the sentiment classification label is classified as the first category label; if the degree of confusion is not less than the first threshold, the sentiment classification label is classified as the second category label.

8. The multi-label sentiment classification model training method according to claim 5, characterized in that: The step of obtaining a first easily-confusing sample text from the first sample text and obtaining a second easily-confusing sample text from the second sample text includes: Inputting the first sample text into the trained multi-label sentiment classification model to obtain the maximum probability sentiment classification determined by the trained multi-label sentiment classification model and the first discrimination probability corresponding to the first sentiment classification label of the first sample text determined by the trained multi-label sentiment classification model; If the maximum probability sentiment classification determined by the trained multi-label sentiment classification model is inconsistent with the first sentiment classification label of the first sample text, and it is determined that the first discrimination probability and the first discrimination threshold corresponding to the first sentiment classification label of the first sample text satisfy a first relationship, then the first sample text is determined as the first easily confusing sample text; Inputting the second sample text into the trained multi-label sentiment classification model to obtain a second discrimination probability corresponding to the first sentiment classification label of the second sample text discriminated by the trained multi-label sentiment classification model; If it is determined that the second discrimination probability and the second discrimination threshold corresponding to the first sentiment classification label of the second sample text satisfy a second relationship, the second sample text is determined as the second easily confusing sample text.

9. The multi-label sentiment classification model training method according to claim 8, characterized in that: The step of determining that the first discrimination probability and the first discrimination threshold corresponding to the first sentiment classification label of the first sample text satisfy a first relationship includes: Get the first confidence coefficient; determining a first product of the first confidence coefficient and the first discrimination threshold; If the first discrimination probability is less than the first product, it is determined that the first discrimination probability and the first discrimination threshold satisfy the first relationship.

10. The multi-label sentiment classification model training method according to claim 8, characterized in that: The step of determining that the second discrimination probability and the second discrimination threshold corresponding to the first sentiment classification label of the second sample text satisfy a second relationship includes: Get the second confidence coefficient; determining a second product of the second confidence coefficient and the second discrimination threshold; If the second discrimination probability is less than the second product, it is determined that the second discrimination probability and the second discrimination threshold satisfy the second relationship.

11. The multi-label sentiment classification model training method according to claim 8, characterized in that: The determining of the second loss function based on the loss sub-functions of the first easily confused sample text, the second easily confused sample text, and the non-easy confused sample text comprises: For the first easily confusing sample text, determining a first auxiliary label for each of the sentiment categories; Based on the first auxiliary label, calculating a second loss sub-function of the first easily confused sample text; For the second easily confusing sample text, determining a second auxiliary label for each of the sentiment categories; Based on the second auxiliary label, calculating a third loss sub-function of the second easily confused sample text; Calculating a fourth loss sub-function of the non-confusable sample text; The second loss function is calculated based on the second loss sub-function, the third loss sub-function, and the fourth loss sub-function.

12. The multi-label sentiment classification model training method according to claim 11, characterized in that: The step of determining the first auxiliary label of each sentiment classification for the first easily confused sample text includes: Inputting the first easily confused sample text into the trained multi-label sentiment classification model to obtain the first discrimination probability corresponding to the first sentiment classification label of each sentiment classification discriminated by the trained multi-label sentiment classification model; The first auxiliary label of the sentiment classification is determined based on the first sentiment classification label and the first discrimination probability corresponding to the first sentiment classification label.

13. A text multi-label sentiment classification method, characterized in that: include: Get the target text; Inputting the target text into a multi-label sentiment classification model, obtaining a first sentiment classification result for each sentiment classification, wherein the first sentiment classification result is used to indicate whether the target text belongs to the sentiment classification; The multi-label sentiment classification model is trained by the following process: inputting sample texts in the initial sample text set into a plurality of the multi-label sentiment classification models to obtain the maximum probability sentiment classification corresponding to each of the multi-label sentiment classification models; based on the comparison of the first dominant sentiment classification label of the sample text and the maximum probability sentiment classification corresponding to each of the multi-label sentiment classification models, screening out matching sample texts from the initial sample text set, and the screened matching sample texts constitute a matching sample text set; inputting the matching sample texts into the multi-label sentiment classification model to obtain the first positive discriminant probability corresponding to each sentiment classification; Based on the first sentiment classification label of each of the sentiment classifications and the first forward discriminant probability, calculate the first loss function corresponding to each of the multi-label sentiment classification models; for each of the multi-label sentiment classification models, train the multi-label sentiment classification model based on the first loss function corresponding to the multi-label sentiment classification model; replace the initial sample text set with the matching sample text set, and return to the step of inputting the sample text in the initial sample text set into multiple multi-label sentiment classification models, until the first loss function satisfies a predetermined condition, and then determine the trained multi-label sentiment classification model from the multiple multi-label sentiment classification models.

14. The text multi-label sentiment classification method according to claim 13, characterized in that: The multi-label sentiment classification model includes a basic learning model, a sentiment detail learning model and a classification layer; The step of inputting the target text into a multi-label sentiment classification model and obtaining a first sentiment classification result for each sentiment classification includes: Converting the target text into a target text vector; Inputting the target text vector into the basic learning model to obtain a basic semantic feature vector; Inputting the basic semantic feature vector into the emotion detail learning model to obtain a high-order emotion feature vector; The high-order emotion feature vector is input into the classification layer, and the first emotion classification result is obtained for each emotion classification.

15. The text multi-label sentiment classification method according to claim 14, characterized in that: The emotion detail learning model includes a fourth number of second transformation sub-model layers cascaded up and down, each of the second transformation sub-model layers includes a third number of second transformation sub-models, and the third number is the number of basic semantic feature sub-vectors in the basic semantic feature vector; The step of inputting the basic semantic feature vector into the emotional detail learning model to obtain a high-order emotional feature vector includes: For each of the basic semantic feature sub-vectors in the basic semantic feature vector, inputting the basic semantic feature sub-vector into a third number of the second transformation sub-models of the second transformation sub-model layer at the lowest layer; Except for the second transformation sub-model layer at the uppermost layer, the output sub-vector of each second transformation sub-model in the second transformation sub-model layer of each layer is input into a third number of the second transformation sub-models in the second transformation sub-model layer at the uppermost layer; The high-order emotion feature sub-vectors output by the third number of the second transformation sub-models in the uppermost layer of the second transformation sub-model layer are cascaded to obtain the high-order emotion feature vector.

16. The text multi-label sentiment classification method according to claim 14, characterized in that: The step of inputting the target text vector into the basic learning model to obtain a basic semantic feature vector comprises: inputting the target text vector into the basic learning model to obtain a basic semantic feature vector and a second sentiment classification result corresponding to each sentiment classification; After the target text is input into a multi-label sentiment classification model and a first sentiment classification result is obtained for each sentiment classification, the text multi-label sentiment classification method further includes: integrating the first sentiment classification result and the second sentiment classification result of each of the sentiment classifications to obtain a third sentiment classification result.

17. A multi-label sentiment classification model training device, characterized in that: include: A first input unit is used to input the sample text in the initial sample text set into multiple multi-label sentiment classification models to obtain multiple maximum probability sentiment classifications corresponding to each of the multi-label sentiment classification models; A cleaning unit, configured to screen out matching sample texts from the initial sample text set based on a comparison between the first dominant sentiment classification label of the sample text and the maximum probability sentiment classification corresponding to each of the multi-label sentiment classification models, wherein the screened matching sample texts constitute a matching sample text set; A second input unit, used to input the matching sample text into each of the multi-label sentiment classification models to obtain a plurality of first positive discriminant probabilities corresponding to each of the sentiment classifications; A calculation unit, configured to calculate a first loss function corresponding to each of the multi-label sentiment classification models based on a first sentiment classification label of each of the sentiment classifications and the first forward discriminant probability; A first training unit is used to train each multi-label sentiment classification model based on the first loss function corresponding to the multi-label sentiment classification model; A returning unit is used to replace the initial sample text set with the matching sample text set, and return to the step of inputting the sample text in the initial sample text set into multiple multi-label sentiment classification models until the first loss function satisfies a predetermined condition, and then determine the trained multi-label sentiment classification model from the multiple multi-label sentiment classification models.

18. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, it implements the multi-label sentiment classification model training method according to any one of claims 1 to 12 or the text multi-label sentiment classification method according to any one of claims 13 to 16.

19. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multi-label sentiment classification model training method according to any one of claims 1 to 12 or the text multi-label sentiment classification method according to any one of claims 13 to 16 is implemented.

20. A computer program product, comprising a computer program, which is read and executed by a processor of a computer device, so that the computer device executes the multi-label sentiment classification model training method according to any one of claims 1 to 12 or the text multi-label sentiment classification method according to any one of claims 13 to 16.

Images