Method, device, equipment, storage medium and program product for determining sentence emotion

Abstract

This application provides a method, apparatus, device, storage medium, and program product for determining the sentiment of a statement. Embodiments of this application can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and in-vehicle systems, and relate to artificial intelligence technology. The method includes: identifying the sentiment category of dialogue statements in the text to be identified to obtain a dialogue sentiment category; identifying the sentiment category of related statements in the text corresponding to the dialogue statements to obtain an associated sentiment category, wherein the associated statements contain supplementary descriptions about the sentiment of the dialogue statements; when the dialogue sentiment category is neutral, determining the associated sentiment category as the target sentiment category of the dialogue statements in the text to be identified; when the dialogue sentiment category is not neutral, determining the dialogue sentiment category as the target sentiment category of the dialogue statements in the text to be identified. This application can improve the accuracy of statement sentiment determination.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a method, apparatus, device, storage medium, and program product for determining the sentiment of a statement. Background Technology

[0002] To make the synthesized audio more vivid and realistic, it needs to be able to express different emotions, such as happiness and surprise. Related technologies first perform emotion recognition on the dialogue in the text, and then label the analyzed emotion categories based on a speech synthesis markup language, so as to synthesize emotionally charged audio based on the dialogue content of the text.

[0003] In related technologies, dialogue statements in text, or the context of dialogue statements, may not necessarily contain emotional information related to the dialogue statements, making it impossible to accurately identify the emotion category of the dialogue statements. Therefore, the emotion determined from dialogue statements in text may not be accurate, resulting in a low accuracy rate in determining the emotion of a statement. Summary of the Invention

[0004] This application provides a method, apparatus, device, computer-readable storage medium, and program product for determining the sentiment of sentences, which can improve the accuracy of sentence sentiment determination.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a method for determining the sentiment of a statement, including:

[0007] The emotion category of the dialogue statements in the text to be identified is identified to obtain the dialogue emotion category, wherein the dialogue emotion category is the emotion category expressed by the dialogue statement.

[0008] The emotion category of the associated sentences in the text to be identified corresponding to the dialogue sentence is identified to obtain the associated emotion category. The associated emotion category is the emotion category expressed by the associated sentence. The associated sentence contains supplementary descriptions about the emotion of the dialogue sentence.

[0009] When the dialogue emotion category is a neutral category, the associated emotion category is determined as the target emotion category of the dialogue statement in the text to be identified; wherein, the neutral category represents an emotion category with an intensity lower than a threshold.

[0010] When the dialogue emotion category is not the neutral category, the dialogue emotion category is determined as the target emotion category of the dialogue statement in the text to be identified.

[0011] This application provides a sentence sentiment determination device, including:

[0012] The dialogue recognition module is used to identify the emotion category of dialogue statements in the text to be recognized, and obtain the dialogue emotion category, wherein the dialogue emotion category is the emotion category expressed by the dialogue statement.

[0013] The association recognition module is used to identify the emotion category of the associated statements in the text to be recognized that correspond to the dialogue statement, and to obtain the associated emotion category. The associated emotion category is the emotion category expressed by the associated statement, and the associated statement contains supplementary descriptions about the emotion of the dialogue statement.

[0014] The category determination module is used to determine the associated emotion category as the target emotion category of the dialogue statement in the text to be identified when the dialogue emotion category is a neutral category; wherein, the neutral category represents an emotion category with an intensity lower than a threshold; when the dialogue emotion category is not the neutral category, the dialogue emotion category is determined as the target emotion category of the dialogue statement in the text to be identified.

[0015] In some embodiments of this application, the association recognition module is further configured to detect multiple emotion keywords of the associated statement from the associated statement corresponding to the dialogue statement in the text to be recognized; and based on the dialogue statement, to identify the associated emotion category of the associated statement corresponding to the dialogue statement from the multiple emotion keywords.

[0016] In some embodiments of this application, the association recognition module is further configured to calculate the text distance for the dialogue statement and each of the emotion keywords respectively, to obtain the text distance corresponding to each of the emotion keywords; determine the emotion keyword corresponding to the smallest text distance among the multiple emotion keywords as the target keyword; and perform category recognition on the target keyword to obtain the associated emotion category of the associated statement corresponding to the dialogue statement in the text to be recognized.

[0017] In some embodiments of this application, the association recognition module is further configured to perform emotion classification on the text to be recognized to obtain the text emotion category of the text to be recognized; when the text emotion category is not the neutral category, multiple emotion keywords of the associated statement are detected from the associated statement corresponding to the dialogue statement in the text to be recognized.

[0018] In some embodiments of this application, the statement emotion determination device further includes: an audio synthesis module; the audio synthesis module is used to generate speech mark information of the dialogue statement based on the target emotion category; the speech mark information is used to control the speech features during audio synthesis, the speech features including at least: timbre, speech rate, volume, and pitch; based on the speech mark information, audio synthesis is performed on the dialogue statement to obtain synthesized audio corresponding to the dialogue statement; wherein, the synthesized audio is audio with the emotion corresponding to the target emotion category.

[0019] In some embodiments of this application, the text to be identified includes: a plurality of the dialogue statements; the statement emotion determination device further includes: a statement determination module; the statement determination module is used to extract associated dialogue from the text to be identified for each of the dialogue statements in the text to be identified; when the associated dialogue contains emotion information, the associated dialogue is determined as the associated statement of each of the dialogue statements.

[0020] In some embodiments of this application, the dialogue recognition module is further configured to identify the emotion category of the dialogue statements in the text to be recognized by using a dialogue emotion model, and obtain the dialogue emotion category.

[0021] In some embodiments of this application, the association recognition module is further configured to identify the emotion category of the associated statements in the text to be recognized that correspond to the dialogue statements through an association emotion model, thereby obtaining the associated emotion category.

[0022] In some embodiments of this application, the statement emotion determination device further includes: a model training module; the model training module is used to predict the emotion category of unsupervised training data using the initial recognition model of the i-th iteration, to obtain the predicted category of the unsupervised training data and the confidence level of the predicted category; wherein, the initial recognition model of the first iteration is obtained by fine-tuning the pre-trained model obtained using supervised training data; 1≤i≤N, i is a positive integer; based on the confidence level, enhanced data is selected from the unsupervised training data, and the predicted category corresponding to the enhanced data is determined as the temporary label of the enhanced data; enhanced training data is obtained by fusing the temporary label, the enhanced data, and the supervised training data, and the pre-trained model is fine-tuned using the enhanced training data to obtain the updated recognition model of the i-th iteration; the updated recognition model of the i-th iteration is fine-tuned using the supervised training data to obtain the initial recognition model of the i+1-th iteration; when i is iterated to N, the dialogue emotion model is selected from the updated recognition models obtained in N iterations.

[0023] In some embodiments of this application, the model training module is further configured to: use the pre-trained model to identify the emotion category of the text data in the augmented training data, and obtain the category identification result; calculate the first loss value of the augmented training data based on the difference between the category identification result and the label information in the augmented training data; use the pre-trained model to predict the predicted probability distribution of the unsupervised training data on multiple emotion categories; calculate the second loss value of the unsupervised training data based on the predicted probability distribution of the unsupervised training data; determine the fusion result of the first loss value and the second loss value as the target loss value, and use the target loss value to fine-tune the model parameters of the pre-trained model until the fine-tuning termination condition is met, thereby obtaining the updated identification model for the i-th iteration.

[0024] In some embodiments of this application, the model training module is further configured to: replace sentiment words in the acquired original text data using a mask to obtain pre-training text data; use the model to be trained to predict words in the mask in the pre-training text data to obtain predicted words; determine discrimination information for the predicted words and determine a pre-training loss value based on the discrimination information; the discrimination information indicates whether the discrimination word and the sentiment word are the same; and adjust the model parameters of the model to be trained using the pre-training loss value until the pre-training stopping condition is reached to obtain the pre-training model.

[0025] This application provides an electronic device, including:

[0026] Memory, used to store executable instructions;

[0027] The processor, when executing executable instructions stored in the memory, implements the statement sentiment determination method provided in the embodiments of this application.

[0028] This application provides a computer-readable storage medium storing executable instructions for implementing the statement sentiment determination method provided in this application when executed by a processor.

[0029] This application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the statement sentiment determination method provided in this application.

[0030] The embodiments of this application have the following beneficial effects: In addition to identifying the emotion category of dialogue statements, the electronic device also identifies the emotion category of related statements in the text to be identified that contain supplementary descriptions of the emotion of the dialogue statements, and determines whether the dialogue statements contain emotion based on whether the dialogue emotion category is neutral. Based on whether the dialogue statements contain emotion, the final target emotion category is selected for the dialogue statements from the dialogue emotion category and the related emotion category. In this way, the emotion information of the related statements can be used to supplement the emotion information of the dialogue information, thereby enabling the emotion category of the dialogue information to be accurately determined and improving the accuracy of the emotion determination of the statements. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the architecture of the statement sentiment determination system provided in the embodiments of this application;

[0032] Figure 2 This is provided by the embodiments of this application. Figure 1 A schematic diagram of the server structure in the diagram;

[0033] Figure 3 This is a flowchart illustrating a method for determining the sentiment of statements provided in an embodiment of this application;

[0034] Figure 4 This is another flowchart illustrating the sentence sentiment determination method provided in the embodiments of this application;

[0035] Figure 5 This is another flowchart illustrating the sentence sentiment determination method provided in the embodiments of this application;

[0036] Figure 6 This is another flowchart illustrating the sentence sentiment determination method provided in the embodiments of this application;

[0037] Figure 7 This is a schematic diagram of the processing flow for synthesizing audiobooks provided in the embodiments of this application;

[0038] Figure 8 This is a flowchart illustrating the emotion analysis of narration content provided in an embodiment of this application;

[0039] Figure 9 This is a schematic diagram of the model pre-training process provided in the embodiments of this application;

[0040] Figure 10 This is a schematic diagram illustrating the process of iteratively enhancing and fine-tuning a pre-trained model using unsupervised data, as provided in an embodiment of this application.

[0041] Figure 11 This is a schematic diagram illustrating the fine-tuning process that incorporates the entropy regularization algorithm, as provided in the embodiments of this application.

[0042] Figure 12 This is a schematic diagram illustrating the principle of entropy regularization provided in the embodiments of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0045] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0047] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0048] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0049] 1) Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0050] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0051] 2) Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0052] 3) Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0053] 4) Text to Speech (TTS) refers to the process of converting a text file into natural speech output.

[0054] 5) Pre-training is a strategy in deep learning that refers to the process of training a model in advance. This involves training a network model using large-scale data; the resulting model is generally called a pre-trained model. Pre-trained models can be further optimized using data specific to a particular task to obtain a model truly suited for that task.

[0055] 6) Fine-tuning: The process of training a pre-trained model for a specific task by adjusting its parameters. In other words, it is the process of continuing to optimize the pre-trained model using data corresponding to the specific task.

[0056] 7) Entropy Regularization (ER) is an algorithm used to solve multi-class classification problems where the probability distribution of each class is uniform.

[0057] 8) Confidence score, used to represent the reliability of the results predicted by the model. The higher the confidence score, the more reliable the results predicted by the model; the lower the confidence score, the less reliable the results predicted by the model.

[0058] 9) Speech Synthesis Markup Language (SSML) is an XML (Extensible Markup Language) markup language used in speech synthesis. It unifies a set of common, annotated tags for speech synthesis. SSML can provide tags for timbre, speech rate, pitch, volume, file type, sampling rate, etc.

[0059] With the research and advancement of artificial intelligence (AI) technology, it is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with further technological development, AI will be applied in even more areas and play an increasingly important role.

[0060] Text-to-speech is a major branch of artificial intelligence technology, involving both natural language processing and audio synthesis technologies. It aims to output text to users in the form of speech and is widely used in scenarios such as audiobooks and intelligent customer service.

[0061] To make the synthesized audio more vivid and realistic, it needs to be able to express different emotions, such as happiness and surprise. Related technologies first perform emotion recognition on the dialogue in the text, and then label the analyzed emotion categories based on a speech synthesis markup language, so as to synthesize emotionally charged audio based on the text dialogue content.

[0062] In related technologies, features are constructed directly based on dialogue statements in the text or the context of the dialogue statements. Then, a trained emotion recognition model is used to perform emotion recognition on the constructed features, thereby obtaining the emotion category of the dialogue statements.

[0063] However, dialogue statements in text, or the context of dialogue statements, may not necessarily contain emotional information related to the dialogue statements, making it impossible to accurately identify the emotion category of the dialogue statements. Therefore, the emotions determined by related technologies for dialogue statements in text may not be accurate, resulting in low accuracy in determining the emotion of statements.

[0064] In addition, the performance of the emotion recognition model also affects the accuracy of sentence emotion determination. Most related technologies construct sentence-level and word-level features from the training text, and then use these features to train a deep learning model to obtain the emotion recognition model.

[0065] However, this approach requires feature construction before model training begins, which is time-consuming and labor-intensive, and the constructed features require additional storage space. At the same time, when the model is deep, overfitting is likely to occur during training, thus limiting the performance of the emotion recognition model.

[0066] This application provides a method, apparatus, device, computer-readable storage medium, and program product for determining the sentiment of statements, which can improve the accuracy of statement sentiment determination. The following describes exemplary applications of the electronic devices provided in this application for determining statement sentiment. These electronic devices can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or as servers. The following describes exemplary applications when the electronic device is implemented as a server.

[0067] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the sentence sentiment determination system provided in this application embodiment. To support a sentence sentiment determination application, in the sentence sentiment determination system 100, a terminal 400 (terminals 400-1 and 400-2 are shown as examples) connects to a server 200 via a network 300. The network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. The sentence sentiment determination system 100 also includes a database 500 to provide data support to the server 200. The database 500 can be integrated into the server 200 or operate independently of the server 200. Figure 1 This illustrates the scenario where database 500 is independent of server 200.

[0068] Terminal 400-1 is used to respond to the author's input operation on the graphical interface 410-1, generate the text to be recognized, and send the text to be recognized to the server 200 via network 300.

[0069] Server 200 is used to identify the emotion category of dialogue statements in the text to be identified, thereby obtaining the dialogue emotion category, where the dialogue emotion category is the emotion category expressed by the dialogue statement; to identify the emotion category of related statements in the text to be identified that correspond to the dialogue statements, thereby obtaining the related emotion category, where the related emotion category is the emotion category expressed by the related statement, and the related statement contains supplementary descriptions of the emotion of the dialogue statement; when the dialogue emotion category is a neutral category, the related emotion category is determined as the target emotion category of the dialogue statement in the text to be identified; when the dialogue emotion category is not a neutral category, the dialogue emotion category is determined as the target emotion category of the dialogue statement in the text to be identified, thus completing the emotion determination for the dialogue statement, where a neutral category represents an emotion category with an intensity below a threshold.

[0070] Server 200 is also used to synthesize speech based on the target emotion category of the dialogue statements in the text to be recognized, to obtain synthesized speech with the target emotion category, and then sends the synthesized speech to terminal 400-2 via network 300. Terminal 400-2 is used to respond to the user's operation on the graphical interface 410-2 and play the synthesized speech.

[0071] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 400 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart home appliance, in-vehicle terminal, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the invention.

[0072] See Figure 2 , Figure 2 This is provided by the embodiments of this application. Figure 1 A schematic diagram of the structure of a server (an embodiment of an electronic device) in the diagram. Figure 2 The server 200 shown includes at least one processor 210, memory 250, at least one network interface 220, and a user interface 230. The various components in server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 240.

[0073] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0074] User interface 230 includes one or more output devices 231 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0075] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices physically located away from the processor 210.

[0076] Memory 250 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 250 described in this application embodiment is intended to include any suitable type of memory.

[0077] In some embodiments, memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0078] Operating system 251 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0079] The network communication module 252 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220, such as Bluetooth, Wi-Fi, and Universal Serial Bus (USB).

[0080] Presentation module 253 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 231 (e.g., a display screen, a speaker, etc.) associated with user interface 230;

[0081] The input processing module 254 is used to detect and translate one or more user inputs or interactions from one or more input devices 232.

[0082] In some embodiments, the statement sentiment determination device provided in this application can be implemented in software. Figure 2 A sentence sentiment determination device 255 stored in memory 250 is shown. This device can be software in the form of programs and plugins, and includes the following software modules: a dialogue recognition module 2551, an association recognition module 2552, a category determination module 2553, an audio synthesis module 2554, a sentence determination module 2555, and a model training module 2556. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0083] In other embodiments, the statement sentiment determination device provided in this application can be implemented in hardware. As an example, the statement sentiment determination device provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the statement sentiment determination method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0084] In some embodiments, a terminal or server (both possible implementations of an electronic device) can implement the statement sentiment determination method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as an audiobook generation APP; it can also be a mini-program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin.

[0085] For example, the computer program in this application embodiment may be an audiobook production platform mounted on a terminal or server. This audiobook production platform can input novels created by authors, generate audio from the novels, and imbue the audio corresponding to dialogues spoken by characters in the novel with emotion, while other sentences can be output normally, thereby making the audiobook more vivid through emotionally charged audio.

[0086] The embodiments of this application can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and vehicle-mounted systems. Below, we will describe the sentence sentiment determination method provided by the embodiments of this application in conjunction with exemplary applications and implementations of the electronic devices provided in the embodiments of this application.

[0087] See Figure 3 , Figure 3 This is a flowchart illustrating a sentence sentiment determination method provided in an embodiment of this application, which will be combined with... Figure 3 The steps shown are explained.

[0088] S101. Identify the emotion category of the dialogue statements in the text to be identified, and obtain the dialogue emotion category.

[0089] This application embodiment is implemented in a scenario where the emotion category of dialogue statements in text is determined. For example, the emotion category of dialogue in a novel is determined to synthesize audio with emotion for the character's dialogue. Another example is determining the emotion category of lines in a script to guide the actors' performance. In this application embodiment, the electronic device first identifies the dialogue statements from the text to be recognized, and then begins to identify the emotion category of the dialogue statements to clarify the emotion category expressed by the dialogue statements, recording this emotion category as the dialogue emotion category. That is, the dialogue emotion category is the emotion category expressed by the dialogue statements.

[0090] In some embodiments, the electronic device can input dialogue statements into a trained dialogue emotion model to determine the emotion category, and use the emotion category inferred by the dialogue emotion model as the dialogue emotion category.

[0091] In other embodiments, the electronic device can perform similarity matching between dialogue statements and preset statements corresponding to different emotion categories, and determine the emotion category corresponding to the preset statement that is sufficiently similar to the dialogue statement (i.e., the maximum similarity or the similarity is greater than a threshold) as the dialogue emotion category.

[0092] It is understood that, in this embodiment of the application, the text to be identified can be any type of text such as novels, biographies, scripts, and textbooks, and this embodiment of the application does not impose any limitations on it. Dialogue statements are statements spoken by characters in the text to be identified. In addition to dialogue statements, the text to be identified may also contain statements describing the plot, environment, etc., which do not need to be spoken by characters in the text to be identified.

[0093] It should be noted that electronic devices can identify dialogue statements from text using dialogue statement recognition rules. For example, statements enclosed in double quotation marks are identified as dialogue statements. Electronic devices can also classify each statement in the text to be recognized, obtaining statement types such as dialogue category, narration type, etc., and then identify statements classified as dialogue types as dialogue statements.

[0094] It is understood that the emotional categories in the dialogue embodiments of this application can be emotional categories with strong intensity, such as happiness, disgust, anger, fear, liking, surprise, gratitude, etc., i.e., emotional categories with intensity above the threshold, or neutral categories with weak intensity, such as no emotion, calm, confusion, etc., i.e., emotional categories with intensity below the threshold.

[0095] S102. Identify the emotion category of the related sentences in the text to be identified that correspond to the dialogue sentences, and obtain the related emotion category.

[0096] For a given dialogue statement, the electronic device extracts related statements from the text to be recognized; that is, it extracts the associated statements of the dialogue statement. Next, the electronic device identifies the emotion category expressed by the associated statements and designates the identified emotion category as the associated emotion category. Here, the associated emotion category is the emotion category expressed by the associated statements.

[0097] It should be noted that the associated statements contain supplementary descriptions of the emotions expressed in the dialogue. In other words, in this embodiment, the associated statements are selected statements that can provide supplementary explanations of the specific emotions expressed in the dialogue, and will definitely include the emotions corresponding to the dialogue statements.

[0098] In some embodiments, the electronic device may select from the narration corresponding to the dialogue statement. In this case, the electronic device can select the statement containing the emotion from the narration corresponding to the dialogue statement as the associated statement. For example, when the text to be recognized is: XX happily said: "You are really my good son", then the electronic device will select the narration statement "XX happily said" as the associated statement.

[0099] In other embodiments, the electronic device can also filter relevant statements from other dialogue statements besides the current one. In this case, the electronic device can first determine the character speaking the current dialogue statement, and then determine dialogue statements from other dialogue statements that describe the character's emotions. For example, if the current dialogue statement is "You are my good son," and the character speaking it is the father, then if other dialogue statements include "Are you happy for me too, Dad?", this dialogue statement can be identified as a relevant statement.

[0100] It is understood that the associated emotion categories in the embodiments of this application can be strong emotion categories such as happiness, disgust, anger, fear, liking, surprise, gratitude, etc., i.e., emotion categories with intensity higher than the threshold, or neutral emotion categories such as no emotion, calm, confusion, etc., i.e., emotion categories with intensity lower than the threshold.

[0101] S103. When the dialogue emotion category is neutral, the associated emotion category is determined as the target emotion category of the dialogue statement in the text to be identified.

[0102] When the emotional category of a conversation is neutral, the electronic device determines that the conversation contains almost no strong emotions. Therefore, it directly uses the associated emotional category identified from related sentences as the target emotional category for the conversation, thus supplementing the emotional content of the conversation with the emotional category of the related sentences. The neutral category represents emotional categories with intensity below a threshold.

[0103] S104. When the dialogue emotion category is not a neutral category, the dialogue emotion category is determined as the target emotion category of the dialogue statement in the text to be identified.

[0104] When the emotion category of a dialogue is not neutral, it indicates that the dialogue contains a relatively strong emotion. In this case, the electronic device no longer needs to supplement the emotion category of the dialogue with the emotion category of the associated statement; that is, it directly determines the emotion category of the dialogue as the target emotion category.

[0105] In other words, in this embodiment, after identifying the emotional category of a dialogue statement, the electronic device determines whether the emotional category is neutral, i.e., whether the dialogue statement contains a relatively strong emotion. Then, based on whether the emotional category is neutral, it determines whether to assign the emotional category of the dialogue statement as the target emotional category or to assign the related emotional category as the target emotional category. That is, the electronic device decides whether to use the emotional category of the dialogue statement or the related emotional category of the related statement as the final emotional category of the dialogue statement, depending on whether the emotion expressed by the dialogue statement is strong or obvious.

[0106] Understandably, compared to related technologies where dialogue statements in text, or the context of dialogue statements, may not necessarily contain emotional information related to the dialogue statements, thus making it difficult to accurately identify the emotional category of dialogue statements, in this embodiment, the electronic device, in addition to identifying the emotional category of the dialogue statements, also identifies the emotional category of related statements in the text to be identified that provide supplementary descriptions of the emotions of the dialogue statements. Furthermore, it determines whether the dialogue statement contains emotions based on whether the dialogue emotional category is neutral. Based on whether the dialogue statement contains emotions, it selects the final target emotional category for the dialogue statement from the dialogue emotional category and related emotional categories. In this way, the emotional information of related statements can be used to supplement the emotional information of the dialogue information, thereby enabling accurate determination of the emotional category of the dialogue information and improving the accuracy of statement emotion determination.

[0107] based on Figure 3 See Figure 4 , Figure 4 This is another flowchart illustrating the sentence emotion determination method provided in this application. In some embodiments of this application, the specific implementation process of identifying the emotion category of associated sentences corresponding to dialogue sentences in the text to be identified, i.e., S102, may include: S1021-S1022, as follows:

[0108] S1021. Detect multiple emotion keywords from the associated sentences corresponding to the dialogue sentences in the text to be identified.

[0109] The electronic device detects emotional keywords in the associated statements to obtain all emotional keywords contained in the associated statements. It should be noted that the emotional keywords in the embodiments of this application can be not only emotional words such as "happy" or "joyful", but also descriptions of actions that can express emotions, such as "clapping and laughing" or "clapping and praising".

[0110] S1022. Based on the dialogue statements, identify the associated emotion categories of the related statements corresponding to the dialogue statements from multiple emotion keywords.

[0111] After recognizing multiple emotion keywords, electronic devices combine them with dialogue statements to filter out target keywords for category identification. Then, they perform category identification on the target keywords to obtain the associated emotion category.

[0112] It is understood that electronic devices can select target keywords from multiple emotion keywords by utilizing the text distance between dialogue statements and each emotion keyword; electronic devices can also select target keywords from multiple emotion keywords by utilizing the semantic correlation between dialogue statements and each emotion keyword, which is not limited in the embodiments of this application.

[0113] based on Figure 4 See Figure 5 , Figure 5 This is another flowchart illustrating the sentence emotion determination method provided in the embodiments of this application. In some embodiments of this application, the specific implementation process of identifying the associated emotion category of the related sentence corresponding to the dialogue sentence from multiple emotion keywords based on the dialogue sentence, i.e., S1022, may include: S1022a-S1022c, as follows:

[0114] S1022a. Calculate the text distance for each dialogue statement and each emotion keyword to obtain the text distance corresponding to each emotion keyword.

[0115] It is understood that the text distance between each emotion keyword and the dialogue statement can refer to the number of characters or the number of statements between each emotion keyword and the dialogue statement. This application embodiment does not limit this.

[0116] S1022b: Determine the emotional keyword that has the smallest text distance among multiple emotional keywords as the target keyword.

[0117] After obtaining the text distance between each emotion keyword and the dialogue statement, the electronic device sorts the multiple text distances in ascending order to determine the smallest text distance. Then, it selects the emotion keyword corresponding to the smallest text distance as the target keyword.

[0118] Of course, in other embodiments, the electronic device may also compare multiple text distances with a distance threshold and filter out the sentiment keywords corresponding to text distances less than the distance threshold as target keywords.

[0119] S1022c. Perform category identification on the target keywords to obtain the associated sentiment category of the related sentences in the text to be identified that correspond to the dialogue sentences.

[0120] Electronic devices categorize selected target keywords to determine the emotional category to which the target keywords belong, and then identify that emotional category as the associated emotional category. For example, if an electronic device categorizes the action description "clapping and laughing" as expressing an emotion, and finds that its emotional category is "happiness," then it will identify "happiness" as the associated emotional category.

[0121] In this embodiment of the application, the electronic device can use the emotion keyword closest to the dialogue text to determine the associated emotion category, so that the associated emotion category is reliable enough.

[0122] In some embodiments of this application, before detecting multiple sentiment keywords of the associated sentences from the associated sentences corresponding to the dialogue sentences in the text to be identified, i.e. before S1021, the method may further include: S1023, as follows:

[0123] S1023. Perform emotion classification on the text to be identified to obtain the text emotion category of the text to be identified.

[0124] Electronic devices perform emotion classification on the text to be identified, determining the text's emotion category based on whether the entire text contains emotional information. In other words, the text's emotion category reflects whether the text to be identified contains emotion.

[0125] At this point, the detection of multiple emotion keywords from the associated statements corresponding to the dialogue statements in the text to be identified, i.e., the specific implementation process of S1021, may include: S1021a:

[0126] S1021a When the text emotion category is not neutral, multiple emotion keywords are detected from the associated sentences corresponding to the dialogue sentences in the text to be identified.

[0127] If the text emotion category is not neutral, it indicates that the text to be identified contains emotion. In this case, the electronic device may be able to detect emotion keywords from the associated sentences. When the text emotion category is neutral, it indicates that the text to be identified does not contain emotion. In this case, the electronic device cannot detect valid emotion keywords from the associated sentences. Therefore, in this embodiment, the electronic device can use whether the text emotion category is neutral to determine whether to start detecting emotion keywords in the associated sentences, thus avoiding starting the detection of emotion keywords in the associated sentences when the text to be identified does not contain emotion, saving computing resources.

[0128] based on Figure 3See Figure 6 , Figure 6 This is another flowchart illustrating the sentence emotion determination method provided in the embodiments of this application. In some embodiments of this application, after determining the associated emotion category as the target emotion category of the dialogue sentence in the text to be identified, or after determining the dialogue emotion category as the target emotion category of the dialogue sentence in the text to be identified, that is, after S103 or S104, the method may further include: S105-S106, as follows:

[0129] S105. Generate speech tag information for dialogue sentences based on the target emotion category.

[0130] In some embodiments, the electronic device may maintain a data table that records the feature tags of audio corresponding to different emotion categories, and then locate the feature tag corresponding to the target emotion category from the data table, and determine the feature tag as the speech tag information of the dialogue statement.

[0131] In other embodiments, the electronic device can directly acquire the feature tags of audio corresponding to the same historical emotion category as the target emotion category, and determine the acquired feature tags as the speech tag information of the dialogue statement.

[0132] It should be noted that speech tagging information is used to control the speech features during audio synthesis. Speech features include at least: timbre, speech rate, volume, and pitch. That is, speech tagging information can control at least the timbre, speech rate, pitch, and volume of dialogue sentences during speech synthesis, thereby making the synthesized audio from dialogue sentences more vivid and realistic.

[0133] It is understood that the speech tag information can be SSML tag information or CSSML tag information, and this application embodiment does not limit it.

[0134] S106. Based on speech tag information, perform audio synthesis on dialogue statements to obtain synthesized audio corresponding to the dialogue statements.

[0135] Electronic devices control the audio synthesis process of dialogue sentences based on the audio features recorded by speech tag information, resulting in synthesized audio that is more emotionally charged and vivid. In other words, the synthesized audio carries the emotion corresponding to the target emotion category.

[0136] In this embodiment, the electronic device synthesizes audio for dialogue statements in the text to be identified based on the target emotion category, thereby making the synthesized audio corresponding to the dialogue statements more vivid and realistic with emotional color.

[0137] In some embodiments of this application, the text to be identified includes multiple dialogue statements. In this case, before identifying the emotion category of the associated statements corresponding to the dialogue statements in the text to be identified, i.e., before S102, the method may further include: S107-S108, as follows:

[0138] S107. For each dialogue statement in the text to be identified, extract the associated dialogue from the text to be identified.

[0139] When the text to be recognized contains multiple dialogue statements, the electronic device can filter out the related dialogues that are associated with each dialogue statement from the remaining dialogue statements in the text to be recognized after removing that dialogue statement.

[0140] In some embodiments, the electronic device may identify related dialogues as those whose text distance from each of the remaining dialogue statements does not exceed a distance threshold. For example, the electronic device may identify dialogues whose text distance from each of the remaining dialogue statements does not exceed two sentences as related dialogues.

[0141] In other embodiments, the electronic device can determine the speaker of each dialogue statement and then identify other dialogue statements containing that speaker as associated dialogues. For example, when the speaker of a dialogue statement is Zhang San, the electronic device can identify other dialogue statements containing Zhang San, such as "Zhang San, don't be so agitated," as associated statements.

[0142] S108. When the associated dialogue contains emotional information, the associated dialogue is identified as the associated statement for each dialogue statement.

[0143] Next, the electronic device uses semantic analysis, part-of-speech tagging, and other methods to determine whether the associated dialogue contains emotional information. If the associated dialogue contains emotional information, the associated dialogue for each dialogue statement is directly identified as the associated statement for that dialogue statement. In this way, the process of identifying associated statements is completed.

[0144] In this embodiment, the electronic device can determine the associated sentences containing emotional information for each dialogue sentence from other dialogue sentences, so as to supplement the emotional information of the dialogue sentences with the emotional information contained in the associated sentences.

[0145] In some embodiments of this application, the specific implementation process of identifying the emotion category of dialogue statements in the text to be identified, i.e., obtaining the dialogue emotion category, may include: S1011, as follows:

[0146] S1011. Using a dialogue emotion model, the emotion category of the dialogue statements in the text to be identified is determined, and the dialogue emotion category is obtained.

[0147] Similarly, the emotional category of the associated sentences corresponding to the dialogue sentences in the text to be identified is determined to obtain the associated emotional category. The specific implementation process of S102 can include: S1024, as follows:

[0148] S1024. Using the associated sentiment model, the sentiment category of the associated sentences in the text to be identified, which correspond to the dialogue sentences, is identified to obtain the associated sentiment category.

[0149] It is understood that both dialogue emotion models and related emotion models can be models trained directly using supervised training data, or models that are pre-trained using unsupervised data and then fine-tuned using supervised data. This application does not limit the specific models.

[0150] The following section uses a dialogue emotion model as an example to illustrate the model training process in the embodiments of this application.

[0151] In some embodiments of this application, before identifying the emotion category of dialogue statements in the text to be identified using a dialogue emotion model, i.e., before S1011, the method may further include: S1012-S1016, as follows:

[0152] S1012. Using the initial recognition model of the i-th iteration, predict the emotion category of the unsupervised training data to obtain the predicted category of the unsupervised training data and the confidence level of the predicted category.

[0153] Where 1 ≤ i ≤ N, and i is a positive integer. It can be understood that the initial recognition model for the first iteration is obtained by fine-tuning the pre-trained model using supervised training data. That is, the electronic device uses supervised data to fine-tune the pre-trained model, and after fine-tuning, the initial recognition model for the first iteration is obtained, serving as the starting point for model iteration updates. The initial recognition model for each subsequent iteration is obtained based on the updated recognition model from the previous iteration.

[0154] It is understandable that supervised training data refers to labeled text data, while unsupervised training data is unlabeled text data.

[0155] S1013. Based on confidence, augmented data is selected from unsupervised training data, and the predicted category corresponding to the augmented data is determined as the temporary label of the augmented data.

[0156] The electronic device filters out data from the unsupervised training data whose predicted class confidence is higher than the confidence threshold, and uses it as augmented data. It also filters out the predicted classes from the unsupervised training data that correspond to the predicted classes of the augmented data, and uses them as temporary labels for the augmented data.

[0157] S1014. Using temporary labels, augmented data, and supervised training data, we fuse them to obtain augmented training data, and use the augmented training data to fine-tune the pre-trained model to obtain the updated recognition model for the i-th iteration.

[0158] Electronic devices can use temporary labels and augmented data to create new training data, which is then fused with the existing supervised training data to obtain a larger-scale training dataset, known as augmented training data. The electronic device then uses this augmented training data to fine-tune the pre-trained model, resulting in the updated recognition model for the i-th iteration.

[0159] S1015. Using supervised training data, fine-tune the updated recognition model of the i-th iteration to obtain the initial recognition model of the i+1-th iteration.

[0160] Next, the electronic device will use supervised training data to fine-tune the updated recognition model of the i-th iteration, and use the model obtained after fine-tuning as the initial recognition model of the (i+1)-th iteration to start the (i+1)-th iteration update.

[0161] It is understandable that since the updated recognition model in the i-th iteration is based on augmented training data, that is, the temporary labels and augmented data obtained by the initial recognition model in the i-th iteration participate in the training process of the updated recognition model in the i-th iteration. This is equivalent to transferring the knowledge learned by the initial recognition model in the i-th iteration to the updated recognition model in the i-th iteration. Thus, the initial recognition model in the i-th iteration can be regarded as the teacher model, and the updated recognition model in the i-th iteration can be regarded as the student model. In this way, the updated recognition model in the i-th iteration can converge faster and better during fine-tuning, resulting in better performance of the updated recognition model in the i-th iteration.

[0162] S1016. When i is iterated to N, the dialogue emotion model is selected from the updated recognition model obtained from N iterations.

[0163] When the electronic device iterates from i to N, it uses test data to determine the performance of the updated recognition model obtained in all iterations. The best-performing updated recognition model is then designated as the dialogue emotion model. In this way, the electronic device obtains a high-performance dialogue emotion model, enabling more accurate dialogue emotion categories to be derived subsequently. Furthermore, no features need to be pre-built during training, saving storage space required for features.

[0164] In some embodiments of this application, the pre-trained model is fine-tuned using enhanced training data to obtain the updated recognition model for the i-th iteration, i.e., the specific implementation process of S1014 may include: S1014a-S1014e, as follows:

[0165] S1014a. Using a pre-trained model, the emotion category of the text data in the augmented training data is identified to obtain the category identification result.

[0166] S1014b: Based on the difference between the category recognition result and the label information in the augmented training data, the first loss value of the augmented training data is calculated.

[0167] It is understandable that the first loss value is the loss value generated during supervised training.

[0168] S1014c: Using a pre-trained model, predict the probability distribution of unsupervised training data across multiple emotion categories.

[0169] Electronic devices also utilize pre-trained models to predict the probability of each emotion category from the unsupervised training data, thereby obtaining the predicted probability distribution of the unsupervised training data for each emotion category. This is based on the predicted probability distribution.

[0170] S1014d: Based on the predicted probability distribution of the unsupervised training data, the second loss value of the unsupervised training data is calculated.

[0171] Next, the electronic device calculates the entropy value of the predicted probability distribution and determines the calculated entropy value as the second loss value. For example, equation (1) illustrates the process of calculating the entropy value of the predicted probability distribution:

[0172]

[0173] Among them, y u It predicts the probability distribution. E(y) represents the m-th predicted probability in the predicted probability distribution, where M is the total number of emotion categories. u ) is the entropy value.

[0174] Understandably, the introduction of entropy as the second loss value is mainly to avoid the situation where the probability distribution of different emotion categories is too uniform in the i-th iteration of the updated recognition model during emotion recognition, that is, to avoid low confidence and inaccurate emotion recognition.

[0175] S1014e, The fusion result of the first loss value and the second loss value is determined as the target loss value, and the model parameters of the pre-trained model are fine-tuned using the target loss value until the fine-tuning termination condition is met, thus obtaining the updated recognition model of the i-th iteration.

[0176] The electronic device can obtain the target loss value by weighting the first loss value and the second loss value to obtain a fusion result of the first loss value and the second loss value. Alternatively, the electronic device can obtain the target loss value by multiplying the first loss value and the second loss value. Then, based on the target loss value and the learning rate, the electronic device calculates update components for the model parameters of the pre-trained model, and updates the model parameters according to these update components. This process is repeated until a fine-tuning termination condition is met, such as the number of iterations during fine-tuning reaching a preset number, or the model accuracy during fine-tuning reaching a preset accuracy, etc. Upon completion of fine-tuning, the updated recognition model of the i-th iteration is obtained.

[0177] At this point, the electronic device has completed the fine-tuning process of the updated recognition model in the i-th iteration. It is evident that during this process, the electronic device can improve the accuracy of the updated recognition model in the i-th iteration in emotion category recognition by introducing a second loss value, namely, an entropy value, thereby enhancing the accuracy of the dialogue emotion model.

[0178] In some embodiments of this application, the initial recognition model of the i-th iteration is used to predict the emotion category of the unsupervised training data, thereby obtaining the predicted category of the unsupervised training data and the confidence level of the predicted category. That is, before S1012, the method may further include: S1017-10110, as follows:

[0179] S1017. Use a mask to replace the emotion words in the obtained original text data to obtain pre-trained text data.

[0180] The electronic device first acquires raw text data, which is unlabeled text data. Then, the electronic device can identify emotion words from the raw text data through methods such as part-of-speech tagging and semantic recognition, and generate masks for the emotion words. The emotion words are then replaced using the masks to obtain pre-trained text data.

[0181] It is understood that the mask can be an obstruction or other characters, and this application embodiment does not limit it.

[0182] S1018. Use the model to be trained to predict words from the mask in the pre-trained text data.

[0183] It is understandable that the model to be trained can be a network model obtained by randomly initializing parameters, or a network model that has already been pre-trained once, such as a network model pre-trained using English characters. In this case, the original text data can be Chinese characters.

[0184] S1019. Determine the discrimination information for the predicted word, and determine the pre-training loss value based on the discrimination information.

[0185] It should be noted that the discriminant information represents whether the discriminant word and the sentiment word are the same. That is, the electronic device needs to identify whether the predicted word in the text to be trained is the same as the sentiment word in the original text data. If they are the same, the discriminant information is set to 1; if they are different, the discriminant information is set to 0. Next, the electronic device will generate a pre-training loss value based on the discriminant information. For example, the average of the discriminant information of different sentiment words can be used as the pre-training loss value.

[0186] S10110. Use the pre-training loss value to adjust the model parameters of the model to be trained until the pre-training termination condition is met, and then obtain the pre-trained model.

[0187] Finally, the electronic device calculates update components based on the pre-training loss value, or the pre-training loss value and the learning rate, and uses these update components to update the model parameters of the model to be trained. This process is repeated until the pre-training stopping condition is met, at which point the pre-trained model is obtained.

[0188] It is understood that the pre-training stopping condition can be that the number of iterations reaches 10,000, or that the accuracy of the discrimination information of emotion words reaches 100%. This application embodiment does not limit this.

[0189] At this point, the electronic device has completed the pre-training process and obtained a pre-trained model that can be used in subsequent fine-tuning processes.

[0190] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0191] This application embodiment is implemented in the scenario of synthesizing audiobooks from novels (text to be identified) uploaded by creators, in order to synthesize emotionally charged audio (synthesized audio) for the dialogue content (dialogue sentences) in the audiobook, thereby increasing the listener's interest and sense of immersion.

[0192] The emotional categories of dialogue in a novel can be categorized as follows: anger, disgust, sadness, fear, happiness, liking, surprise, gratitude, and neutral (neutral category). Neutral refers to a category that does not belong to the first eight emotional categories or is devoid of emotion.

[0193] Figure 7 This is a schematic diagram of the processing flow for synthesizing audiobooks provided in an embodiment of this application. See also... Figure 7 The process may include:

[0194] S201. The server (electronic device) retrieves the narration (related statements) and dialogue content from the novel.

[0195] S202. The server uses the narration algorithm model (associative sentiment model) to determine the sentiment category (associative sentiment category) of the narration content.

[0196] S203. The server uses a dialogue algorithm model (dialogue emotion model) to determine the emotion category (dialogue emotion category) of the dialogue content.

[0197] The server first obtains the emotion categories of the dialogue content and the narration content, and then selects the emotion category of the dialogue content according to the priority strategy and rules.

[0198] S204. The server determines whether the emotion category of the dialogue content is neutral. If yes, proceed to S205; otherwise, proceed to S206.

[0199] S205. The server determines the emotion category of the narration as the emotion tag for the dialogue content.

[0200] S206. The server outputs the emotion tag (target emotion category) of the dialogue content. The emotion tag of the dialogue content is used for audio synthesis.

[0201] The emotion category of the dialogue content takes precedence over the emotion category of the narration content. That is, if the emotion category of the dialogue content is not neutral and the confidence level is greater than 0.8, the emotion of the dialogue content is directly output. If the emotion category of the dialogue content is neutral and the emotion category of the narration content is not neutral, the emotion category of the narration content is directly used as the emotion label of the dialogue content.

[0202] After the server outputs the emotion tag (target emotion category) for each dialogue content, it converts the emotion tag of each dialogue content into SSML markup language. Then, it uses SSML markup language to synthesize the audio corresponding to the emotion of the dialogue content. In other words, during the synthesis process, the server automatically recognizes SSML markup statements and synthesizes the audio corresponding to each emotion tag, and finally obtains audio with emotion.

[0203] Because much of the narration in the novel carries the emotion of the dialogue, for example, XX happily says, "You are my good son." Since the emotional level of the dialogue is relatively weak, while the narration has a strong emotion and contains the emotional keyword "happy," the emotional category of the narration can be used as the final emotional label for the dialogue.

[0204] The narration algorithm model is mainly based on a model pre-trained with Chinese text (pre-trained model) and fine-tuned for the downstream task, namely narration sentiment analysis. See also Figure 8 , Figure 8 This is a flowchart illustrating the sentiment analysis of narration content provided in an embodiment of this application. The process may include:

[0205] S301, The server takes the narration and dialogue as input.

[0206] S302. The server classifies the input to obtain the classification category (text sentiment category).

[0207] S303. The server determines whether the category is neutral. If yes, proceed to S307; otherwise, proceed to S304.

[0208] S304. The server detects sentiment keywords in the input.

[0209] In other words, if the server determines that the input category is neutral, it will not perform sentiment keyword detection on the narration content and will directly use neutrality as the sentiment category of the narration content. If the input category is not neutral, the server will continue to perform sentiment keyword detection on the narration content.

[0210] S305. The server determines whether there is only one sentiment keyword. If not, proceed to S306; otherwise, proceed to S307.

[0211] S306. The server retrieves the emotion keyword (target keyword) that is closest to the dialogue content as the emotion category.

[0212] S307, The emotional category of the narration output by the server.

[0213] Below, we will use the training of a dialogue algorithm model as an example to illustrate the training process of the narration algorithm model and the dialogue algorithm model. That is, the narration algorithm model can be obtained through a training method similar to that used for the dialogue algorithm model.

[0214] The server first uses unsupervised novel dialogue data (unsupervised training data) as pre-training corpus, and continues pre-training on the basic pre-trained model, ELETRA. During pre-training, the server generates masks for all sentiment words in the novel dialogue data, ensuring that no more than 20% of the words are covered by the masks. During pre-training, the generator predicts sentiment words, and the discriminator determines whether the predicted words are the same as the original sentiment words. Based on the discrimination results, the model parameters are adjusted, and the generator is used as the pre-trained model upon completion of pre-training.

[0215] For example, Figure 9 This is a schematic diagram of the model pre-training process provided in this application embodiment. The server generates a mask 9-2 for text 9-1 (original text data), i.e., XX happily said "Okay," and uses mask 9-2 to cover the word "happy," obtaining the model input 9-3 (pre-training text). The server uses generator 9-4 to predict the portion of input 9-3 covered by mask 9-2, obtaining the output 9-5 of generator 9-4, i.e., XX excitedly said "Okay." Next, discriminator 9-6 is used to determine whether each character in the output 9-5 of generator 9-4 has been replaced; if replaced, it is marked as 0, and if not, it is marked as 1. Then, the parameters of generator 9-4 are adjusted. This method achieves better model training results than text obtained by randomly covering the text with a mask.

[0216] After obtaining the pre-trained model, the server will use unsupervised data (unsupervised training data) to iteratively enhance the model and fine-tune it. Figure 10 This is a schematic diagram illustrating the process of iteratively enhancing and fine-tuning a pre-trained model using unsupervised data, as provided in an embodiment of this application. This process may include:

[0217] S401. The server uses supervised data D (supervised training data) to generate a teacher model.

[0218] That is, the server uses supervised data D to fine-tune the pre-trained model and uses the resulting emotion classification model as the teacher model (the initial recognition model for the first iteration).

[0219] S402. The server uses the teacher model and unsupervised data U to generate pseudo data D'.

[0220] At this point, the server uses the teacher model to infer the unsupervised data U, obtains its corresponding emotion category and confidence level, and uses data with a confidence level greater than 0.9 and its emotion category to obtain pseudo data D' (reinforced training data).

[0221] S403, The server uses D+D' and the pre-trained model to generate a learning model.

[0222] The server merges D+D' into new supervised data (enhanced training data) to fine-tune the pre-trained model, resulting in the student model (the updated recognition model in the i-th iteration).

[0223] S404. The server uses supervised data to fine-tune the student model to obtain a new teacher model (the initial recognition model in the (i+1)th iteration).

[0224] The server repeats the S402-S404 operations until N iterations are completed, and uses the optimal learning model as the dialogue algorithm model.

[0225] In this way, through N iterations of unsupervised data enhancement, the generalization ability of the model can be improved, thereby making the final dialogue algorithm model have a significant improvement in the effect of emotion classification.

[0226] The server can also incorporate entropy regularization algorithms during pre-training and fine-tuning processes.

[0227] For multi-classification tasks, the large number of emotion categories can lead to misclassification or low classification confidence in the model. Since the inference process aims for the model to predict emotion categories with high confidence, it is necessary to introduce an entropy regularization algorithm to ensure that the trained model avoids low-confidence emotion categories during inference.

[0228] Entropy regularization algorithm mainly addresses the issue of uniformly distributed emotions during inference and prediction. The smaller the entropy value, the higher the category confidence, and the more accurate the classification of the emotion category.

[0229] During fine-tuning, the input to the pre-trained model is supervised data, while the input to the entropy regularization algorithm is unsupervised data with the same batch size as the supervised data. The entropy value of the unsupervised data is calculated during training, and it continuously decreases until it approaches zero.

[0230] Figure 11 This is a schematic diagram illustrating the fine-tuning process that incorporates an entropy regularization algorithm, as provided in an embodiment of this application. This process may include:

[0231] S501, The server acquires supervised data.

[0232] S502. The server uses supervised data to fine-tune the pre-trained model to obtain the supervised training loss value (first loss value).

[0233] S503, The server acquires unsupervised data.

[0234] S504. The server uses the model parameters obtained from supervised training to predict unsupervised data and obtain the probability distribution (predicted probability distribution).

[0235] S505. The server uses the probability distribution to calculate the entropy value (second loss value).

[0236] S506: The server merges the loss value and entropy value to obtain the final loss (the target loss value).

[0237] S507, server backpropagation final loss.

[0238] Figure 12 This is a schematic diagram illustrating the principle of entropy regularization provided in an embodiment of this application. When the model parameters obtained from supervised training are θ... * At that time, unsupervised data x u After the model parameter is θ * Obtain the probability distribution y u At this moment, I hope y u One category has a larger value, while the values ​​of the other categories are smaller. Therefore, the entropy value can be calculated using equation (1) and incorporated into the fine-tuning process. Assuming there are 5 emotion categories, before introducing entropy regularization, y u The distribution is uniform, meaning the probability of each of the five emotion categories is 0.2. After introducing entropy regularization, y u Instead of a uniform distribution, the probability of the second emotion category is set to 1, and the rest to 0, or the probability of the fifth emotion category is set to 1, and the rest to 0.

[0239] Therefore, introducing entropy regularization during fine-tuning can effectively avoid the situation where the probability distribution of multiple emotion categories is uniform in forward inference, ensuring that there is always an emotion category with a larger probability value among multiple emotion categories, thereby accurately predicting the emotion of the dialogue content.

[0240] It is understood that in the embodiments of this application, user information is involved, such as data related to the text to be identified or the novels created. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0241] The following description continues to illustrate the exemplary structure of the sentence sentiment determination device 255 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software module stored in the voice emotion determination device 255 in the memory 250 may include:

[0242] The dialogue recognition module 2551 is used to identify the emotion category of dialogue statements in the text to be recognized, and obtain the dialogue emotion category, wherein the dialogue emotion category is the emotion category expressed by the dialogue statement.

[0243] The association recognition module 2552 is used to identify the emotion category of the associated sentences in the text to be recognized that correspond to the dialogue sentences, and obtain the associated emotion category. The associated emotion category is the emotion category expressed by the associated sentences, and the associated sentences contain supplementary descriptions about the emotion of the dialogue sentences.

[0244] The category determination module 2553 is used to determine the associated emotion category as the target emotion category of the dialogue statement in the text to be identified when the dialogue emotion category is a neutral category; wherein, the neutral category represents an emotion category with an intensity lower than a threshold; when the dialogue emotion category is not the neutral category, the dialogue emotion category is determined as the target emotion category of the dialogue statement in the text to be identified.

[0245] In some embodiments of this application, the association recognition module 2552 is further configured to detect multiple emotion keywords of the associated statement from the associated statement corresponding to the dialogue statement in the text to be recognized; and based on the dialogue statement, to identify the associated emotion category of the associated statement corresponding to the dialogue statement from the multiple emotion keywords.

[0246] In some embodiments of this application, the association recognition module 2552 is further configured to calculate the text distance for the dialogue statement and each of the emotion keywords respectively, to obtain the text distance corresponding to each of the emotion keywords; determine the emotion keyword corresponding to the smallest text distance among the multiple emotion keywords as the target keyword; perform category recognition on the target keyword to obtain the associated emotion category of the associated statement corresponding to the dialogue statement in the text to be recognized.

[0247] In some embodiments of this application, the association recognition module 2552 is further configured to perform emotion classification on the text to be recognized to obtain the text emotion category of the text to be recognized; when the text emotion category is not the neutral category, multiple emotion keywords of the associated statement are detected from the associated statement corresponding to the dialogue statement in the text to be recognized.

[0248] In some embodiments of this application, the statement emotion determination device 255 further includes: an audio synthesis module 2554; the audio synthesis module 2554 is used to generate speech tag information of the dialogue statement based on the target emotion category; the speech tag information is used to control the speech features during audio synthesis, and the speech features include at least: timbre, speech rate, volume, and pitch; based on the speech tag information, audio synthesis is performed on the dialogue statement to obtain the synthesized audio corresponding to the dialogue statement; wherein, the synthesized audio is audio with the emotion corresponding to the target emotion category.

[0249] In some embodiments of this application, the text to be identified includes: a plurality of the dialogue statements; the statement emotion determination device 255 further includes: a statement determination module 2555; the statement determination module 2555 is used to extract associated dialogue from the text to be identified for each of the dialogue statements in the text to be identified; when the associated dialogue contains emotion information, the associated dialogue is determined as the associated statement of each of the dialogue statements.

[0250] In some embodiments of this application, the dialogue recognition module 2551 is further configured to identify the emotion category of the dialogue statements in the text to be recognized by using a dialogue emotion model, and obtain the dialogue emotion category.

[0251] In some embodiments of this application, the association recognition module 2552 is further configured to identify the emotion category of the associated statements in the text to be recognized that correspond to the dialogue statements through an association emotion model, thereby obtaining the associated emotion category.

[0252] In some embodiments of this application, the sentiment determination device 255 further includes: a model training module 2556; the model training module 2556 is used to predict the sentiment category of unsupervised training data using the initial recognition model of the i-th iteration, to obtain the predicted category of the unsupervised training data and the confidence level of the predicted category; wherein, the initial recognition model of the first iteration is obtained by fine-tuning the pre-trained model obtained using supervised training data; 1≤i≤N, i is a positive integer; based on the confidence level, enhanced sentiment determination models are selected from the unsupervised training data. The data is processed, and the predicted category corresponding to the augmented data is determined as the temporary label of the augmented data. The temporary label, the augmented data, and the supervised training data are fused to obtain augmented training data, and the pre-trained model is fine-tuned using the augmented training data to obtain the updated recognition model for the i-th iteration. The updated recognition model for the i-th iteration is fine-tuned using the supervised training data to obtain the initial recognition model for the i+1-th iteration. When i is iterated to N, the dialogue emotion model is selected from the updated recognition models obtained in N iterations.

[0253] In some embodiments of this application, the model training module 2556 is further configured to: use a pre-trained model to identify the emotion category of the text data in the augmented training data, and obtain a category identification result; calculate a first loss value of the augmented training data based on the difference between the category identification result and the label information in the augmented training data; use the pre-trained model to predict the predicted probability distribution of the unsupervised training data on multiple emotion categories; calculate a second loss value of the unsupervised training data based on the predicted probability distribution of the unsupervised training data; determine the fusion result of the first loss value and the second loss value as the target loss value, and use the target loss value to fine-tune the model parameters of the pre-trained model until the fine-tuning termination condition is met, thereby obtaining the updated identification model for the i-th iteration.

[0254] In some embodiments of this application, the model training module 2556 is further configured to replace sentiment words in the acquired original text data using a mask to obtain pre-training text data; use the model to be trained to predict words in the mask in the pre-training text data to obtain predicted words; determine discrimination information for the predicted words, and determine a pre-training loss value based on the discrimination information; the discrimination information indicates whether the discrimination word and the sentiment word are the same; and adjust the model parameters of the model to be trained using the pre-training loss value until the pre-training stopping condition is reached to obtain the pre-training model.

[0255] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the statement sentiment determination method described above in this application.

[0256] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the statement sentiment determination method provided in this application. For example, ... Figure 3 The method for determining the sentiment of a statement is shown.

[0257] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0258] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0259] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0260] As an example, executable instructions can be deployed to execute on a single computing device (an implementation of an electronic device), or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0261] In summary, through the embodiments of this application, in addition to identifying the emotion category of dialogue statements, the electronic device also identifies the emotion category of related statements in the text to be identified that contain supplementary descriptions of the emotion of the dialogue statements. Furthermore, it determines whether the dialogue statement contains emotion based on whether the dialogue emotion category is neutral. Based on whether the dialogue statement contains emotion, it selects the final target emotion category for the dialogue statement from the dialogue emotion category and the related emotion categories. In this way, the emotion information of the related statements can be used to supplement the emotion information of the dialogue information, thereby enabling accurate determination of the emotion category for the dialogue information and improving the accuracy of statement emotion determination.

[0262] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for determining the sentiment of a statement, characterized in that, The method includes: Using the initial recognition model of the i-th iteration, the emotion category is predicted on the unsupervised training data to obtain the predicted category of the unsupervised training data and the confidence level of the predicted category; wherein, the initial recognition model of the first iteration is obtained by fine-tuning the pre-trained model obtained using supervised training data; 1≤i≤N, where i is a positive integer; Based on the confidence level, augmented data is obtained by filtering from the unsupervised training data, and the predicted category corresponding to the augmented data is determined as the temporary label of the augmented data; Using the temporary labels, the augmented data, and the supervised training data, augmented training data is fused together to obtain augmented training data, and the pre-trained model is fine-tuned using the augmented training data to obtain the updated recognition model for the i-th iteration. Using the supervised training data, the updated recognition model of the i-th iteration is fine-tuned to obtain the initial recognition model of the i+1-th iteration; When i is iterated to N, the dialogue emotion model is selected from the updated recognition model obtained from N rounds of iteration; The dialogue emotion model identifies the emotion category of dialogue statements in the text to be identified, and obtains the dialogue emotion category, wherein the dialogue emotion category is the emotion category expressed by the dialogue statement. By using an associated sentiment model, the sentiment category of associated sentences in the text to be identified that correspond to the dialogue sentence is identified, and the associated sentiment category is obtained. The associated sentiment category is the sentiment category expressed by the associated sentence, and the associated sentence contains supplementary descriptions about the sentiment of the dialogue sentence. When the dialogue emotion category is a neutral category, the associated emotion category is determined as the target emotion category of the dialogue statement in the text to be identified; wherein, the neutral category represents an emotion category with an intensity lower than a threshold. When the dialogue emotion category is not the neutral category, the dialogue emotion category is determined as the target emotion category of the dialogue statement in the text to be identified.

2. The method according to claim 1, characterized in that, The step of identifying the emotion category of the associated statements in the text to be identified that correspond to the dialogue statements, and obtaining the associated emotion category, includes: From the associated statements corresponding to the dialogue statements in the text to be identified, multiple emotion keywords of the associated statements are detected; Based on the dialogue statement, the associated emotion category of the associated statement corresponding to the dialogue statement is identified from a plurality of emotion keywords.

3. The method according to claim 2, characterized in that, The step of identifying the associated emotion category of the associated statement corresponding to the dialogue statement from a plurality of emotion keywords, based on the dialogue statement, includes: The text distance is calculated for each of the dialogue statements and each of the emotion keywords to obtain the text distance corresponding to each emotion keyword; The emotional keyword that has the smallest text distance among the multiple emotional keywords is identified as the target keyword; The target keyword is categorized to obtain the associated emotion category of the associated statement in the text to be identified that corresponds to the dialogue statement.

4. The method according to claim 2, characterized in that, Before detecting multiple emotion keywords of the associated statements from the associated statements corresponding to the dialogue statements in the text to be identified, the method further includes: The text to be identified is classified into sentiment categories to obtain the sentiment category of the text to be identified. The step of detecting multiple emotion keywords from the associated statements corresponding to the dialogue statements in the text to be identified includes: When the text emotion category is not the neutral category, multiple emotion keywords of the associated sentences are detected from the associated sentences corresponding to the dialogue sentences in the text to be identified.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The speech tag information of the dialogue statement is generated based on the target emotion category; the speech tag information is used to control the speech features during audio synthesis, and the speech features include at least: timbre, speech rate, volume and pitch; Based on the speech tag information, audio synthesis is performed on the dialogue statement to obtain the synthesized audio corresponding to the dialogue statement; wherein, the synthesized audio is audio with the emotion corresponding to the target emotion category.

6. The method according to any one of claims 1 to 4, characterized in that, The text to be identified includes: multiple dialogue statements; before identifying the emotion category of the associated statements in the text to be identified that correspond to the dialogue statements, and obtaining the associated emotion category, the method further includes: For each dialogue statement in the text to be identified, the associated dialogue is extracted from the text to be identified; When the associated dialogue contains emotional information, the associated dialogue is identified as the associated statement of each of the dialogue statements.

7. The method according to claim 1, characterized in that, The step of fine-tuning the pre-trained model using the enhanced training data to obtain the updated recognition model for the i-th iteration includes: Using the pre-trained model, the text data in the augmented training data is used to identify the emotion category, and the category identification result is obtained; Based on the difference between the category recognition result and the label information in the augmented training data, the first loss value of the augmented training data is calculated; Using the pre-trained model, the predicted probability distribution of the unsupervised training data across multiple emotion categories is obtained; Based on the predicted probability distribution of the unsupervised training data, the second loss value of the unsupervised training data is calculated. The fusion result of the first loss value and the second loss value is determined as the target loss value, and the model parameters of the pre-trained model are fine-tuned using the target loss value until the fine-tuning termination condition is met, thus obtaining the updated recognition model for the i-th iteration.

8. The method according to claim 1, characterized in that, Before using the initial recognition model from the i-th iteration to predict the emotion category of the unsupervised training data, and obtaining the predicted category and the confidence level of the predicted category, the method further includes: The pre-training text data is obtained by replacing sentiment words in the acquired raw text data using a mask. The model to be trained is used to predict words from the mask in the pre-trained text data; For the predicted word, discrimination information is determined, and a pre-training loss value is determined based on the discrimination information; the discrimination information indicates whether the discrimination word and the emotion word are the same. The model parameters of the model to be trained are adjusted using the pre-training loss value until the pre-training stopping condition is met, thus obtaining the pre-trained model.

9. A device for determining the sentiment of a statement, characterized in that, The device includes: The dialogue recognition module is used to identify the emotion category of dialogue statements in the text to be recognized through a dialogue emotion model, and obtain the dialogue emotion category, wherein the dialogue emotion category is the emotion category expressed by the dialogue statement. The association recognition module is used to identify the emotion category of the associated sentences in the text to be identified that correspond to the dialogue sentences through an association emotion model, and obtain the associated emotion category. The associated emotion category is the emotion category expressed by the associated sentences, and the associated sentences contain supplementary descriptions about the emotion of the dialogue sentences. The category determination module is used to determine the associated emotion category as the target emotion category of the dialogue statement in the text to be identified when the dialogue emotion category is a neutral category; wherein, the neutral category represents an emotion category with an intensity lower than a threshold; when the dialogue emotion category is not the neutral category, the dialogue emotion category is determined as the target emotion category of the dialogue statement in the text to be identified. The model training module is used to predict the emotion category of unsupervised training data using the initial recognition model of the i-th iteration, to obtain the predicted category of the unsupervised training data and the confidence level of the predicted category. The initial recognition model of the first iteration is obtained by fine-tuning a pre-trained model using supervised training data; 1 ≤ i ≤ N, where i is a positive integer. Based on the confidence level, augmented data is selected from the unsupervised training data, and the predicted category corresponding to the augmented data is determined as a temporary label for the augmented data. Augmented training data is obtained by fusing the temporary label, the augmented data, and the supervised training data, and the pre-trained model is fine-tuned using the augmented training data to obtain the updated recognition model of the i-th iteration. The updated recognition model of the i-th iteration is fine-tuned using the supervised training data to obtain the initial recognition model of the (i+1)-th iteration. When i iterates to N, a dialogue emotion model is selected from the updated recognition models obtained in N iterations.

10. The apparatus according to claim 9, characterized in that, The association recognition module is further configured to detect multiple emotion keywords of the associated statements from the associated statements corresponding to the dialogue statements in the text to be recognized; Based on the dialogue statement, the associated emotion category of the associated statement corresponding to the dialogue statement is identified from a plurality of emotion keywords.

11. The apparatus according to claim 9, characterized in that, The device further includes: An audio synthesis module is used to generate speech tag information for the dialogue statement based on the target emotion category; the speech tag information is used to control the speech features during audio synthesis, and the speech features include at least: timbre, speech rate, volume, and pitch; based on the speech tag information, audio synthesis is performed on the dialogue statement to obtain the synthesized audio corresponding to the dialogue statement; wherein, the synthesized audio is audio with the emotion corresponding to the target emotion category.

12. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the statement sentiment determination method according to any one of claims 1 to 8.

13. A computer-readable storage medium storing executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the statement sentiment determination method according to any one of claims 1 to 8.

14. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the statement sentiment determination method according to any one of claims 1 to 8.

Images