Voice emotion classification method, system, storage medium and terminal

Abstract

The application provides a speech emotion classification method, system, storage medium and terminal, and the method comprises the following steps: constructing a speech emotion category set; acquiring a corresponding synthetic image for each emotion category; acquiring speech to be classified in terms of emotion, and acquiring text corresponding to the speech; extracting semantic features based on the text, extracting visual features based on the synthetic image, extracting acoustic features based on the speech, and extracting emotion features based on the speech emotion category set; acquiring acoustic semantic features and visual emotion features based on the semantic features, the visual features, the acoustic features, the semantic visual features and the emotion features; and acquiring an emotion classification result of the speech based on the acoustic semantic features, the visual emotion features and the converged emotion features. The speech emotion classification method, system, storage medium and terminal of the application realize arbitrary emotion classification of speech based on the interactive fusion of multi-modal information.

Description

Technical Field

[0001] This invention belongs to the technical field of speech processing and relates to a speech emotion classification method, system, storage medium, and terminal. Background Technology

[0002] With the rapid popularization of artificial intelligence, human-computer interaction, and intelligent voice technology, the demand for voice emotion recognition and classification is becoming increasingly urgent in fields such as intelligent customer service, in-vehicle voice, smart wearables, remote education, and mental health monitoring. Voice, as one of the most direct carriers of human emotions, not only contains semantic information but also deep emotional characteristics such as joy, anger, sorrow, fear, calmness, and irritability. Accurate automatic classification of voice emotions is a core technology for improving the intelligence and human-centered experience of human-computer interaction.

[0003] Existing speech emotion classification schemes mostly rely on manually designed acoustic features, modeling based solely on shallow parameters such as speech rate, volume, fundamental frequency, and formants. Their feature representation capabilities are limited, and they are easily affected by environmental noise, channel interference, accent differences, speech rate variations, and subtle emotional fluctuations, resulting in low generalization and recognition accuracy. Furthermore, traditional machine learning models often employ shallow classification architectures, making it difficult to uncover temporal correlations and hidden emotional features in speech, leading to poor classification performance for speech samples in complex, mixed-emotion scenarios and low signal-to-noise ratio environments. Summary of the Invention

[0004] The purpose of this invention is to provide a voice emotion classification method, system, storage medium, and terminal that comprehensively captures implicit emotions in voice based on multimodal information, adapts to diverse emotional scenarios, and thus achieves accurate and reliable classification of various emotions in voice.

[0005] In a first aspect, the present invention provides a speech emotion classification method, the method comprising the following steps: constructing a speech emotion category set; obtaining a corresponding synthetic image for each emotion category in the speech emotion category set; obtaining the speech to be emotion classified and obtaining the text corresponding to the speech; extracting semantic features based on the text, extracting visual features based on the synthetic image, extracting acoustic features based on the speech, and extracting emotion features based on the speech emotion category set; obtaining semantic-visual features based on the semantic features and the visual features; obtaining convergent acoustic features, convergent semantic features, convergent visual features, and convergent emotion features based on the acoustic features, the semantic-visual features, and the emotion features; obtaining acoustic-semantic features based on the convergent acoustic features and the convergent semantic features; obtaining visual emotion features based on the convergent visual features and the convergent emotion features; and obtaining the emotion classification result of the speech based on the acoustic-semantic features, the visual emotion features, and the convergent emotion features.

[0006] In one implementation of the first aspect, obtaining the corresponding synthesized image for each emotion category in the set of speech emotion categories includes the following steps:

[0007] Generate a synthetic image prompt, which indicates the generation of a portrait of a person with an emotional expression;

[0008] The synthesized image prompts are input into the stable diffusion model to obtain the corresponding synthesized images.

[0009] In one implementation of the first aspect, the text corresponding to the speech is extracted based on an automatic speech recognition model; the semantic features corresponding to the text are extracted based on a CLIP text encoder; the visual features corresponding to the synthesized image are extracted based on a VIT model; the acoustic features corresponding to the speech are extracted based on a wav2vec model; and the emotional features corresponding to the speech emotion category set are extracted based on a CLIP text encoder.

[0010] In one implementation of the first aspect, obtaining semantic visual features based on the semantic features and the visual features includes the following steps:

[0011] The semantic features and the visual features are merged into a first input feature matrix;

[0012] The first input feature matrix is ​​input into the first transformer encoder to obtain the semantic visual features.

[0013] In one implementation of the first aspect, obtaining convergent acoustic features, convergent semantic features, convergent visual features, and convergent emotional features based on the acoustic features, the semantic visual features, and the emotional features includes the following steps:

[0014] The acoustic features, the semantic visual features, and the emotional features are combined into a second input feature matrix;

[0015] The second input feature matrix is ​​input into the second transformer encoder to obtain acoustic semantic visual emotion features;

[0016] The acoustic features, acoustic semantic visual emotion features, emotion features, acoustic feature characters, semantic feature characters, visual feature characters, and emotion feature characters are input into the first large language model to obtain the output matrix;

[0017] The vectors in the output matrix that have the same index positions as the acoustic feature characters, semantic feature characters, visual feature characters, and emotional feature characters are respectively used as the converged acoustic feature, converged semantic feature, converged visual feature, and converged emotional feature.

[0018] In one implementation of the first aspect, obtaining acoustic semantic features based on the convergent acoustic features and the convergent semantic features; obtaining visual emotional features based on the convergent visual features and the convergent emotional features includes the following steps:

[0019] A cross-attention operation is performed on the converged acoustic features and the converged semantic features to obtain the acoustic semantic features;

[0020] A cross-attention operation is performed on the convergent visual features and the convergent emotional features to obtain visual emotional features.

[0021] In one implementation of the first aspect, obtaining the emotion classification result of the speech based on the acoustic semantic features, the visual emotion features, and the convergent emotion features includes the following steps:

[0022] The acoustic semantic features and the visual emotional features are input into the second language model to obtain fused features;

[0023] The fused features are then subjected to global average pooling to obtain the retrieval feature vector;

[0024] The aggregated emotion features and the retrieved feature vector are multiplied by a matrix to obtain the probability of each voice emotion category in the voice emotion category set;

[0025] The voice emotion category with the highest probability is selected as the emotion classification result.

[0026] Secondly, the present invention provides a voice emotion classification system, the system comprising a construction module, a first acquisition module, a second acquisition module, an extraction module, a third acquisition module, a fourth acquisition module, a fifth acquisition module, and a classification module;

[0027] The construction module is used to construct a set of voice emotion categories;

[0028] The first acquisition module is used to acquire a corresponding synthesized image for each emotion category in the set of speech emotion categories;

[0029] The second acquisition module is used to acquire the speech to be classified for emotion and to acquire the text corresponding to the speech;

[0030] The extraction module is used to extract semantic features based on the text, visual features based on the synthesized image, acoustic features based on the speech, and emotional features based on the speech emotion category set.

[0031] The third acquisition module is used to acquire semantic visual features based on the semantic features and the visual features;

[0032] The fourth acquisition module is used to acquire converged acoustic features, converged semantic features, converged visual features, and converged emotional features based on the acoustic features, the semantic visual features, and the emotional features.

[0033] The fifth acquisition module is used to acquire acoustic semantic features based on the converged acoustic features and the converged semantic features; and to acquire visual emotional features based on the converged visual features and the converged emotional features;

[0034] The classification module is used to obtain the emotion classification result of the speech based on the acoustic semantic features, the visual emotion features, and the convergent emotion features.

[0035] Thirdly, the present invention provides a terminal, the terminal comprising: a processor and a memory;

[0036] The memory is used to store computer programs;

[0037] The processor is used to execute the computer program stored in the memory, so that the terminal performs the above-described voice emotion classification method.

[0038] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a terminal, implements the above-described voice emotion classification method.

[0039] As described above, the speech emotion classification method, system, storage medium, and terminal of the present invention have the following beneficial effects:

[0040] (1) By integrating multi-dimensional information such as speech, text, and images, and using information interaction and fusion algorithms, high-precision speech emotion classification was achieved;

[0041] (2) It can achieve arbitrary fine-grained emotion classification, which can not only cover the basic emotion categories of joy, sadness, anger, calmness and surprise, but also accurately identify complex emotions, weak emotions and borderline emotion types such as anxiety, depression, irritability, depression, teasing and helplessness. It supports custom emotion tags to expand the classification dimensions.

[0042] (3) Significantly improve the accuracy and fine-grained discrimination of emotion classification, get rid of the limitation of fixed emotion categories, realize the intelligent classification of any voice emotion in all dimensions and expandable, and meet the needs of refined emotion recognition applications such as human-computer interaction, psychological judgment, and intelligent customer service in multiple scenarios. Attached Figure Description

[0043] Figure 1 The diagram shows an application scenario of the terminal of the present invention in one embodiment.

[0044] Figure 2The flowchart shown is an embodiment of the speech emotion classification method of the present invention;

[0045] Figure 3 The diagram shows a framework schematic of the speech emotion classification method of the present invention in one embodiment;

[0046] Figure 4 The diagram shown is a structural schematic of the speech emotion classification system of the present invention in one embodiment.

[0047] Figure 5 The diagram shown is a structural schematic of the terminal of the present invention in one embodiment. Detailed Implementation

[0048] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0049] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0050] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed by this invention.

[0051] The following embodiments of the present invention provide a numerical calculation verification method, which can be applied to, for example... Figure 1The electronic devices shown are as follows. The electronic devices described in this invention may include mobile phones 11 with wireless charging capabilities, tablet computers 12, laptop computers 13, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. This invention does not impose any restrictions on the specific type of electronic device.

[0052] For example, the electronic device may be a station (STAION, ST) in a WLAN with wireless charging capability, a cellular phone, cordless phone, Session Initiation Protocol (SIP) phone, Wireless Local Loop (WLL) station, Personal Digital Assistant (PDA) device, handheld device with wireless charging capability, computing device or other processing device, computer, laptop computer, handheld communication device, handheld computing device, and / or other devices for communication over a wireless system, as well as next-generation communication systems, such as mobile terminals in 5G networks, mobile terminals in future evolved Public Land Mobile Networks (PLMNs), or mobile terminals in future evolved Non-terrestrial Networks (NTNs).

[0053] For example, the electronic device can communicate with networks and other devices wirelessly. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0054] The technical solutions of the present invention will now be described in detail with reference to the accompanying drawings.

[0055] like Figure 2 and Figure 3 As shown, in one embodiment, the speech emotion classification method of the present invention includes steps S1-S8.

[0056] Step S1: Construct a set of voice emotion categories.

[0057] Specifically, obtain the possible voice emotion categories, such as happy, sad, angry, calm, surprised, anxious, depressed, irritable, down, teasing, helpless, etc., totaling M categories, and construct the set of the voice emotion categories.

[0058] Step S2: Obtain the corresponding synthesized image for each emotion category in the set of voice emotion categories.

[0059] Specifically, for each emotion category, a corresponding synthetic image cue is first generated. This synthetic image cue indicates the generation of a portrait with an emotional expression. For example, generating a portrait with a joyful emotional expression. The synthetic image cue is then input into a stable diffusion model to obtain the corresponding synthetic image.

[0060] Step S3: Obtain the speech to be classified for emotion and obtain the text corresponding to the speech.

[0061] Specifically, the text corresponding to the speech is extracted based on an Automatic Speech Recognition (ASR) model. Preferably, the speech is first preprocessed, including audio noise reduction, noise filtering, amplitude normalization, and removal of invalid silence segments to optimize audio signal quality and reduce the interference of environmental noise and human breathing noise on recognition accuracy. Next, the preprocessed clean speech audio is fed into the ASR model. The model analyzes the speech time-domain and frequency-domain signals through acoustic coding, accurately capturing speech details such as syllable pronunciation, phoneme features, and connected speech pauses, and completes word matching and grammar correction in conjunction with the language decoding layer. Finally, the ASR model, based on its built-in language knowledge base, corrects spoken pronunciation deviations and homophone confusion, accurately transcribes the text content carried by the speech, and extracts standardized and highly accurate corresponding text.

[0062] Step S4: Extract semantic features based on the text, extract visual features based on the synthesized image, extract acoustic features based on the speech, and extract emotional features based on the speech emotion category set.

[0063] Specifically, the semantic features are numerical representations obtained from text transformation, used to characterize the text content, main idea, logic, word meaning, and contextual meaning, reflecting the core information and semantic connections expressed in the text. The visual features are feature data extracted from the synthesized image, representing visual attributes such as image texture, contour, color, spatial structure, and content elements, reflecting intuitive information about the image. The acoustic features are features extracted from speech audio, describing acoustic physical attributes such as timbre, pitch, energy, spectrum, speech rate, and prosody, reflecting the inherent characteristics of the audio itself. The emotional features are features obtained based on a set of speech emotion categories, used to characterize the emotional tendency, emotion type, and emotional intensity conveyed by the speech, such as joy, anger, and sadness. In this invention, the semantic features corresponding to the text are extracted based on the CLIP (Contrastive Language–Image Pre-training) text encoder, with a shape of (1, 512). The visual features corresponding to the synthesized image are extracted based on the VIT (Vision Transformer) model, with a shape of (M, 512). The acoustic features corresponding to the speech are extracted based on the wav2vec model, with a shape of (1, 512). The emotional features corresponding to the speech emotion category set are extracted based on the CLIP text encoder, and the shape is (M, 512).

[0064] Step S5: Obtain semantic visual features based on the semantic features and the visual features.

[0065] Specifically, the semantic features and the visual features are merged into a first input feature matrix with a shape of (1+M, 512). The first input feature matrix is ​​then input into a first transformer encoder to fuse the semantic features and the visual features, thereby obtaining the semantic-visual features.

[0066] Step S6: Based on the acoustic features, the semantic visual features, and the emotional features, obtain converged acoustic features, converged semantic features, converged visual features, and converged emotional features.

[0067] Specifically, obtaining convergent acoustic features, convergent semantic features, convergent visual features, and convergent emotional features based on the acoustic features, the semantic visual features, and the emotional features includes the following steps:

[0068] 61) The acoustic features, the semantic visual features, and the emotional features are combined into a second input feature matrix.

[0069] The acoustic features, semantic visual features, and emotional features are combined into a second input feature matrix with a shape of (1+1+M+M, 512).

[0070] 62) Input the second input feature matrix into the second transformer encoder to obtain acoustic semantic visual emotion features.

[0071] 63) Input the acoustic features, the acoustic semantic visual emotion features, the emotion features, the acoustic feature characters, the semantic feature characters, the visual feature characters, and the emotion feature characters into the first large language model to obtain the output matrix.

[0072] Among them, four special characters are defined: acoustic feature characters. Semantic feature characters <t>Visual characteristic characters <v>and emotional character <c>These are used to aggregate voice information, text information, image information, and emotional information, respectively.

[0073] The acoustic features, acoustic-semantic-visual-emotional features, emotional features, acoustic feature characters, semantic feature characters, visual feature characters, and emotional feature characters are input into the first Large Language Model (LLM) in a preset order. A Large Language Model is an artificial intelligence model designed to understand and generate human language. They are trained on massive amounts of text data and can perform a wide range of tasks, including text summarization and translation, sentiment analysis, and more. LLMs are characterized by their massive scale, containing billions of parameters, which helps them learn complex patterns in language data. These models are typically based on deep learning architectures, such as transformers, which contributes to their impressive performance on various NLP tasks. The first Large Language Model focuses on information from different modalities, pooling this information into vectors corresponding to four specific characters to obtain the output matrix.

[0074] 64) The vectors in the output matrix that have the same index positions as the acoustic feature characters, semantic feature characters, visual feature characters and emotional feature characters are respectively used as the converged acoustic feature, converged semantic feature, converged visual feature and converged emotional feature.

[0075] Step S7: Obtain acoustic semantic features based on the converged acoustic features and the converged semantic features; obtain visual emotional features based on the converged visual features and the converged emotional features.

[0076] Specifically, a cross-attention operation is performed on the converged acoustic features and the converged semantic features to obtain acoustic semantic features. The converged acoustic features serve as the query, and the converged semantic features serve as the key and value. A cross-attention operation is then performed on the converged visual features and the converged sentiment features to obtain visual sentiment features. The converged visual features serve as the query, and the converged sentiment features serve as the key and value.

[0077] Step S8: Obtain the emotion classification result of the speech based on the acoustic semantic features, the visual emotion features, and the convergent emotion features.

[0078] Specifically, obtaining the emotion classification result of the speech based on the acoustic semantic features, the visual emotion features, and the convergent emotion features includes the following steps:

[0079] 81) Input the acoustic semantic features and the visual emotional features into the second language model to obtain the fused features.

[0080] The second language model fuses the acoustic semantic features and the visual emotional features to obtain the fused features.

[0081] 82) Perform global average pooling on the fused features to obtain the retrieval feature vector.

[0082] In this process, global average pooling is performed on the fused features to aggregate the information from the four modalities onto the retrieval feature vector, which has a shape of (512).

[0083] 83) Perform matrix multiplication on the aggregated emotion features and the retrieved feature vector to obtain the probability of each voice emotion category in the voice emotion category set.

[0084] The shape of the aggregated emotion feature is (M, 512), and the remaining retrieved feature vectors are multiplied by matrix to obtain a probability vector of speech emotion category with M elements.

[0085] 84) Select the voice emotion category with the highest probability as the emotion classification result.

[0086] The speech emotion classification method of the present invention will be further illustrated below through specific embodiments.

[0087] In this embodiment, fine-grained emotion classification is performed on the input speech, thereby integrating four types of features: text, image, acoustic, and emotion, to achieve high-precision emotion recognition.

[0088] Step 1: Construct a set of voice emotion categories.

[0089] Among them, five standard emotion sets are constructed: {anger, happiness, sadness, calmness, fear}.

[0090] Step 2: Obtain the corresponding synthetic image for each emotion category.

[0091] Specifically, an emotional facial image synthesis model is used to generate corresponding visual samples for each emotion. For example, anger corresponds to an angry facial image; happiness corresponds to a smiling facial image; sadness corresponds to a crying facial image; calmness corresponds to a neutral facial image; and fear corresponds to a terrified facial image.

[0092] Step 3: Obtain the speech to be classified and its corresponding text.

[0093] The speech to be classified is a user complaint audio file (audio.wav). The corresponding text is identified using an ASR model: "What's wrong with you? The service is terrible!"

[0094] Step 4: Extract semantic features based on the text, extract visual features based on the synthesized image, extract acoustic features based on the speech, and extract emotional features based on the speech emotion category set.

[0095] Semantic features: 768-dimensional vectors are extracted from the text based on the BERT model to represent the meaning of the content and the sentiment.

[0096] Visual features: CNN image features were extracted from 5 synthetic emotion images to represent facial expressions and facial structure.

[0097] Acoustic features: The wav2vec model extracts speech acoustic vectors to represent tone, speech rate, pitch, and energy.

[0098] Sentiment Features: CLIP text encoder generates 5-dimensional one-hot / probabilistic features from the set of sentiment categories to represent category attributes.

[0099] Step 5: Fuse to obtain semantic visual features.

[0100] The semantic features and the visual features are spliced ​​and fused to obtain semantic visual features (which combine text meaning and emotional expression information).

[0101] Step 6: Based on the acoustic features, the semantic visual features, and the emotional features, obtain converged acoustic features, converged semantic features, converged visual features, and converged emotional features.

[0102] Wherein, the converged acoustic features represent global acoustic information of speech; the converged semantic features represent global semantic information of text; the converged visual features represent global information of visual expression; and the converged emotional features represent global information of emotional category.

[0103] Step 7: Obtain acoustic semantic features based on the converged acoustic features and the converged semantic features; obtain visual emotional features based on the converged visual features and the converged emotional features.

[0104] Specifically, the converged acoustic features and converged semantic features yield acoustic semantic features, which integrate speech tone and textual emotion. The converged visual features and converged emotional features yield visual emotional features, which integrate facial expression images and emotion category information.

[0105] Step 8: Obtain the emotion classification result of the speech based on the acoustic semantic features, the visual emotion features, and the convergent emotion features.

[0106] The output probabilities for each emotion category are as follows: anger 0.91, happiness 0.02, sadness 0.03, calmness 0.02, and fear 0.02. Therefore, the emotion classification result is anger.

[0107] The scope of protection of the speech emotion classification method described in this embodiment is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this invention is included within the scope of protection of this invention.

[0108] This invention also provides a voice emotion classification system, which can implement the voice emotion classification method described in this invention. However, the implementation device of the voice emotion classification system described in this invention includes, but is not limited to, the structure of the voice emotion classification system listed in this embodiment. All structural modifications and substitutions of the prior art made in accordance with the principles of this invention are included within the protection scope of this invention.

[0109] like< / c> < / v> < / t> Figure 4 As shown, in one embodiment, the voice emotion classification system of the present invention includes a construction module 41, a first acquisition module 42, a second acquisition module 43, an extraction module 44, a third acquisition module 45, a fourth acquisition module 46, a fifth acquisition module 47, and a classification module 48.

[0110] The construction module 41 is used to construct a set of voice emotion categories.

[0111] The first acquisition module 42 is connected to the construction module 41 and is used to acquire the corresponding synthetic image for each emotion category in the voice emotion category set.

[0112] The second acquisition module 43 is used to acquire the speech to be classified for emotion and to acquire the text corresponding to the speech.

[0113] The extraction module 44 is connected to the construction module 41, the first acquisition module 42 and the second acquisition module 43, and is used to extract semantic features based on the text, extract visual features based on the synthesized image, extract acoustic features based on the speech, and extract emotional features based on the speech emotion category set.

[0114] The third acquisition module 45 is connected to the extraction module 44 and is used to acquire semantic visual features based on the semantic features and the visual features.

[0115] The fourth acquisition module 46 is connected to the third acquisition module 45 and is used to acquire converged acoustic features, converged semantic features, converged visual features and converged emotional features based on the acoustic features, the semantic visual features and the emotional features.

[0116] The fifth acquisition module 47 is connected to the fourth acquisition module 46 and is used to acquire acoustic semantic features based on the convergent acoustic features and the convergent semantic features; and to acquire visual emotional features based on the convergent visual features and the convergent emotional features.

[0117] The classification module 48 is connected to the fifth acquisition module 47 and is used to acquire the emotion classification result of the speech based on the acoustic semantic features, the visual emotion features and the convergent emotion features.

[0118] The structure and principle of the construction module 41, the first acquisition module 42, the second acquisition module 43, the extraction module 44, the third acquisition module 45, the fourth acquisition module 46, the fifth acquisition module 47 and the classification module 48 correspond one-to-one with the steps in the above-mentioned speech emotion classification method, so they will not be described again here.

[0119] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0120] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of the present invention, depending on actual needs. For example, the functional modules / units in the various embodiments of the present invention may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0121] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0122] This invention also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. This available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0123] This invention also provides a terminal. The terminal includes a processor and a memory.

[0124] The memory is used to store computer programs.

[0125] The memory includes various media capable of storing program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card, or optical disk.

[0126] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the terminal performs the above-described voice emotion classification method.

[0127] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0128] like Figure 5As shown, the terminal of the present invention is presented in the form of a general-purpose computing device. The components of the terminal may include, but are not limited to: one or more processors or processing units 51, a memory 52, and a bus 53 connecting different system components (including the memory 52 and the processing unit 51).

[0129] Bus 53 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0130] Terminals typically include various computer system-readable media. These media can be any available media that can be accessed by the terminal, including volatile and non-volatile media, and removable and non-removable media.

[0131] Memory 52 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 521 and / or cache memory 522. The terminal may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 523 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 5 Not shown; usually referred to as a "hard drive"). Although Figure 5 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 53 via one or more data media interfaces. Memory 52 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0132] A program / utility 524 having a set (at least one) of program modules 5241 may be stored, for example, in memory 52. ​​Such program modules 5241 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 5241 typically perform the functions and / or methods described in the embodiments of the present invention.

[0133] The terminal can also communicate with one or more external devices (e.g., keyboard, pointing device, display, etc.), one or more devices that enable user interaction with the terminal, and / or any device that enables the terminal to communicate with one or more other computing devices (e.g., network interface card, modem, etc.). This communication can be performed through input / output (I / O) interface 54. Furthermore, the terminal can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 55. Figure 5 As shown, network adapter 55 communicates with other modules of the terminal via bus 53. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the terminal, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0134] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A voice emotion classification method, characterized in that, The method includes the following steps: Construct a set of voice emotion categories; For each emotion category in the set of speech emotion categories, obtain the corresponding synthesized image; Obtain the speech to be classified for emotion, and obtain the text corresponding to the speech; Semantic features are extracted based on the text, visual features are extracted based on the synthesized image, acoustic features are extracted based on the speech, and emotional features are extracted based on the speech emotion category set. Semantic visual features are obtained based on the semantic features and the visual features; Based on the acoustic features, the semantic-visual features, and the emotional features, convergent acoustic features, convergent semantic features, convergent visual features, and convergent emotional features are obtained; Acoustic semantic features are obtained based on the convergent acoustic features and the convergent semantic features; visual emotional features are obtained based on the convergent visual features and the convergent emotional features; The emotion classification result of the speech is obtained based on the acoustic semantic features, the visual emotion features, and the convergent emotion features.

2. The speech emotion classification method according to claim 1, characterized in that, Obtaining the corresponding synthetic image for each emotion category in the set of speech emotion categories includes the following steps: Generate a synthetic image prompt, which indicates the generation of a portrait of a person with an emotional expression; The synthesized image prompts are input into the stable diffusion model to obtain the corresponding synthesized images.

3. The speech emotion classification method according to claim 1, characterized in that, The text corresponding to the speech is extracted based on an automatic speech recognition model; the semantic features corresponding to the text are extracted based on a CLIP text encoder; the visual features corresponding to the synthesized image are extracted based on a VIT model; and the acoustic features corresponding to the speech are extracted based on a wav2vec model. The CLIP text encoder is used to extract the emotional features corresponding to the set of speech emotion categories.

4. The speech emotion classification method according to claim 1, characterized in that, Obtaining semantic visual features based on the semantic features and the visual features includes the following steps: The semantic features and the visual features are merged into a first input feature matrix; The first input feature matrix is ​​input into the first transformer encoder to obtain the semantic visual features.

5. The speech emotion classification method according to claim 1, characterized in that, Obtaining convergent acoustic features, convergent semantic features, convergent visual features, and convergent emotional features based on the acoustic features, the semantic visual features, and the emotional features includes the following steps: The acoustic features, the semantic visual features, and the emotional features are combined into a second input feature matrix; The second input feature matrix is ​​input into the second transformer encoder to obtain acoustic semantic visual emotion features; The acoustic features, acoustic semantic visual emotion features, emotion features, acoustic feature characters, semantic feature characters, visual feature characters, and emotion feature characters are input into the first large language model to obtain the output matrix; The vectors in the output matrix that have the same index positions as the acoustic feature characters, semantic feature characters, visual feature characters, and emotional feature characters are respectively used as the converged acoustic feature, converged semantic feature, converged visual feature, and converged emotional feature.

6. The speech emotion classification method according to claim 1, characterized in that, Acquiring acoustic semantic features based on the convergent acoustic features and the convergent semantic features; obtaining visual emotional features based on the convergent visual features and the convergent emotional features includes the following steps: A cross-attention operation is performed on the converged acoustic features and the converged semantic features to obtain the acoustic semantic features; A cross-attention operation is performed on the convergent visual features and the convergent emotional features to obtain visual emotional features.

7. The voice emotion classification method according to claim 1, characterized in that, Obtaining the emotion classification result of the speech based on the acoustic semantic features, the visual emotion features, and the convergent emotion features includes the following steps: The acoustic semantic features and the visual emotional features are input into the second language model to obtain fused features; The fused features are then subjected to global average pooling to obtain the retrieval feature vector; The aggregated emotion features and the retrieved feature vector are multiplied by a matrix to obtain the probability of each voice emotion category in the voice emotion category set; The voice emotion category with the highest probability is selected as the emotion classification result.

8. A voice emotion classification system, characterized in that, The system includes a construction module, a first acquisition module, a second acquisition module, an extraction module, a third acquisition module, a fourth acquisition module, a fifth acquisition module, and a classification module; The construction module is used to construct a set of voice emotion categories; The first acquisition module is used to acquire a corresponding synthesized image for each emotion category in the set of speech emotion categories; The second acquisition module is used to acquire the speech to be classified for emotion and to acquire the text corresponding to the speech; The extraction module is used to extract semantic features based on the text, visual features based on the synthesized image, acoustic features based on the speech, and emotional features based on the speech emotion category set. The third acquisition module is used to acquire semantic visual features based on the semantic features and the visual features; The fourth acquisition module is used to acquire converged acoustic features, converged semantic features, converged visual features, and converged emotional features based on the acoustic features, the semantic visual features, and the emotional features. The fifth acquisition module is used to acquire acoustic semantic features based on the converged acoustic features and the converged semantic features; and to acquire visual emotional features based on the converged visual features and the converged emotional features; The classification module is used to obtain the emotion classification result of the speech based on the acoustic semantic features, the visual emotion features, and the convergent emotion features.

9. A terminal, characterized in that, The terminal includes: a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory to cause the terminal to perform the voice emotion classification method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the terminal, it implements the voice emotion classification method as described in any one of claims 1 to 7.

Images