Artificial intelligence-based emotion classification method and device, computer device and medium

Abstract

The application relates to the technical field of artificial intelligence, in particular to an emotion classification method and device based on artificial intelligence, a computer device and a medium. The method carries out preprocessing on original audio, inputs a self-attention layer after splicing the preprocessing result, obtains an attention vector, multiplies the attention vector with the splicing result, inputs a convolution layer to obtain audio features, respectively inputs the audio features into a speaker classifier and a gender classifier to obtain a reference speaker and a reference gender, splices the audio features, the reference speaker and the reference gender, inputs the splicing result into an emotion classifier to perform classification, and obtains a classification result. The method can provide more abundant audio information, improves the generalization ability of emotion classification, and further improves the accuracy of emotion classification, so that a user can obtain good user experience when applying a financial service platform, and the probability of complaints about the financial service platform is reduced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an emotion classification method, apparatus, computer device, and medium based on artificial intelligence. Background Technology

[0002] Currently, with the rapid development of artificial intelligence technology, emotion classification technology for speech has been widely applied in financial service platforms. These platforms can be insurance systems, banking systems, transaction systems, order systems, etc. They can support functions such as shopping, social networking, interactive games, and resource transfer, and can also have functions such as applying for loans, credit cards, or purchasing insurance and wealth management products.

[0003] Emotion classification technology can be applied to intelligent interaction scenarios in financial service platforms. This allows the intelligent chatbot customer service to adjust its language based on the user's real-time emotions when users communicate with the platform via telephone or other means. Existing voice emotion classification typically uses convolutional neural network models to extract features from the speech spectrum before performing emotion classification. The emotion classification results can then be used to assist the intelligent interaction process, improving the intelligence of the interaction scenario. This would allow users to have a better user experience when using the financial service platform and reduce the probability of complaints against the platform.

[0004] However, relying solely on speech spectrum to provide feature information leads to low accuracy in sentiment classification. Existing methods typically use speech content to provide additional semantic feature information. However, in practice, speech spectrum is often difficult to align accurately with speech content, which further reduces the accuracy of sentiment classification. Therefore, improving the accuracy of sentiment classification has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an artificial intelligence-based emotion classification method, apparatus, computer device, and medium to solve the problem of low accuracy in emotion classification.

[0006] In a first aspect, embodiments of the present invention provide an artificial intelligence-based emotion classification method, the emotion classification method comprising:

[0007] The acquired raw audio is preprocessed, and the preprocessed short-time Fourier transform spectrum, Mel spectrum, and Mel frequency cepstral coefficients are concatenated to obtain the concatenated result.

[0008] The splicing result is input into a trained self-attention layer for feature mapping. The feature mapping result is determined as an attention vector. The attention vector is multiplied by the splicing result and input into a trained convolutional layer for feature extraction to obtain audio features.

[0009] The audio features are input into a trained speaker classifier and a trained gender classifier for feature classification to obtain a reference speaker and a reference gender.

[0010] The audio features, the reference speaker, and the reference gender are concatenated and then input into a trained emotion classifier for classification, resulting in a classification result that represents the emotion category of the original audio.

[0011] Secondly, embodiments of the present invention provide an artificial intelligence-based emotion classification device, the emotion classification device comprising:

[0012] The preprocessing module is used to preprocess the acquired raw audio, and concatenate the preprocessed short-time Fourier transform spectrum, Mel spectrum and Mel frequency cepstral coefficients to obtain the concatenated result;

[0013] The feature extraction module is used to input the splicing result into a trained self-attention layer for feature mapping, determine the feature mapping result as an attention vector, multiply the attention vector with the splicing result, and input it into a trained convolutional layer for feature extraction to obtain audio features;

[0014] The feature classification module is used to input the audio features into a trained speaker classifier and a trained gender classifier respectively for feature classification to obtain a reference speaker and a reference gender.

[0015] The emotion classification module is used to concatenate the audio features, the reference speaker, and the reference gender, and then input the result into a trained emotion classifier for classification to obtain a classification result representing the emotion category of the original audio.

[0016] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the emotion classification method as described in the first aspect.

[0017] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the sentiment classification method as described in the first aspect.

[0018] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows:

[0019] The acquired raw audio is preprocessed, and the preprocessed short-time Fourier transform spectrum, Mel spectrum, and Mel frequency cepstral coefficients are concatenated to obtain the concatenated result. This concatenated result is then input into a trained self-attention layer for feature mapping, and the feature mapping result is determined as an attention vector. The attention vector is multiplied with the concatenated result and input into a trained convolutional layer for feature extraction to obtain audio features. These audio features are then input into a trained speaker classifier and a trained gender classifier for feature classification to obtain reference speaker and reference gender. The audio features, reference speaker, and reference gender are concatenated and input into a trained sentiment classifier for classification to obtain the classification result representing the sentiment category of the raw audio. Using the concatenation of the multimodal representation of the raw audio as input provides richer audio information, thereby improving the accuracy of sentiment classification. Furthermore, the additional speaker and gender classification, and the concatenation of the obtained reference speaker and reference gender as reference information with the audio features, improves the generalization ability when processing sentiment classification of the raw audio, thus improving the accuracy of sentiment classification. This allows users to have a better user experience when using the financial service platform and reduces the probability of complaints against the financial service platform. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of an application environment for an artificial intelligence-based emotion classification method provided in Embodiment 1 of the present invention;

[0022] Figure 2 This is a flowchart illustrating an artificial intelligence-based emotion classification method provided in Embodiment 1 of the present invention.

[0023] Figure 3 This is a schematic diagram of the structure of an artificial intelligence-based emotion classification device provided in Embodiment 2 of the present invention;

[0024] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0026] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0027] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0029] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0031] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0032] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0033] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0034] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0035] The first embodiment of this invention provides an artificial intelligence-based emotion classification method, which can be applied to, for example... Figure 1 In this application environment, the client communicates with the server. The client includes, but is not limited to, handheld computers, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, and personal digital assistants (PDAs). The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0036] The client and server can be deployed within a financial service platform, which may include insurance, banking, transaction, and order systems. This platform can support functions such as shopping, social networking, interactive games, and resource transfer. It can also provide functions for applying for loans, credit cards, or purchasing insurance and wealth management products. Sentiment classification technology can be applied to intelligent interaction scenarios within the financial service platform to enhance their intelligence. This allows the platform to adjust its communication style based on the user's real-time emotions when communicating with customer service representatives via telephone or other means, thereby improving the user experience and reducing the likelihood of complaints against the platform.

[0037] See Figure 2 This is a flowchart illustrating an artificial intelligence-based sentiment classification method provided in Embodiment 1 of the present invention. The aforementioned sentiment classification method can be applied to... Figure 1 The client, a computer device connected to the server, obtains raw audio from the server. This raw audio can be audio that requires emotion classification. The client's computer device has a pre-trained emotion classification model deployed within it. This model may include a trained self-attention layer, a trained convolutional layer, a trained speaker classifier, a trained gender classifier, and a trained emotion classifier. This trained emotion classification model can be used to classify the emotion of the input raw audio. Figure 2 As shown, this sentiment classification method may include the following steps:

[0038] Step S201: Preprocess the acquired raw audio, and concatenate the preprocessed short-time Fourier transform spectrum, Mel spectrum, and Mel frequency cepstral coefficients to obtain the concatenation result.

[0039] Here, the original audio can refer to audio that needs to undergo emotion classification processing. In this embodiment, applied to a financial service platform, the original audio can refer to audio of customers consulting about financial services, audio of customers purchasing financial service products, etc. The original audio can be represented as an audio signal. Preprocessing can be used to remove noise from the original audio to improve the signal strength of the original audio. The short-time Fourier transform spectrum can refer to the spectrum information obtained by the short-time Fourier transform of the audio signal corresponding to the original audio. The Mel spectrum can be used to characterize key audio information that conforms to human hearing. The Mel frequency cepstral coefficients can refer to audio features represented by nonlinearity.

[0040] Specifically, this embodiment can be applied to application scenarios such as intelligent interaction and human-computer dialogue to obtain the emotional category information of the target object's voice audio, thereby enabling the determination of interactive statements based on the emotional category information. The target object's voice audio can refer to the original audio collected by the server from the target object.

[0041] Optionally, preprocessing the acquired raw audio may include:

[0042] Perform a short-time Fourier transform on the original audio to obtain the short-time Fourier transform spectrum;

[0043] A preset mapping function is used to map the short-time Fourier spectrum, and the mapping result is determined to be the Mel spectrum;

[0044] The Mel frequency cepstral coefficients are obtained by performing a discrete cosine transform on the Mel spectrum.

[0045] The short-time Fourier transform (SFT) refers to windowing the audio signal corresponding to the original audio signal and then performing a Fourier transform on each window to obtain the SFT spectrum. The SFT spectrum can be used to characterize the frequency domain information of the original audio signal.

[0046] The preset mapping function can refer to the mapping function of the Mel scale. The preset mapping function contains the mapping relationship between the short-time Fourier spectrum and the Mel spectrum, that is, the mapping relationship between the frequencies in the short-time Fourier spectrum and the Mel scale in the Mel spectrum.

[0047] The Discrete Cosine Transform (DCT) can be used to remove the correlation between different frequency dimensions of an audio signal, mapping the audio signal to a lower-dimensional space, which also enhances speech audio.

[0048] Specifically, when performing a short-time Fourier transform on the original audio, the length of a single window during windowing can be set by the implementer. In this embodiment, the time length corresponding to a single window can be set to 0.5s.

[0049] The above Mel mapping function can be expressed as:

[0050]

[0051] Where f can refer to the frequency in the short-time Fourier spectrum, and m can refer to the Mel scale in the Mel spectrum.

[0052] This embodiment performs Mel-Cepstral Analysis on the original audio and retains the multimodal representations during the Mel-Cepstral Analysis process. The multimodal representations are the short-time Fourier transform spectrum, Mel spectrum, and Mel frequency cepstral coefficients, so as to facilitate the subsequent integration of the multimodal representations of the original audio to obtain richer audio feature information of the original audio.

[0053] Optionally, performing a short-time Fourier transform on the original audio includes:

[0054] The original audio is input into a preset filter for filtering, and the filtering result is divided into frames according to a preset frame length to obtain N initial frames.

[0055] Each initial frame is windowed using a preset window function, and the windowing result is then subjected to a short-time Fourier transform.

[0056] The preset filter can refer to a high-pass filter. The preset filter can be used to pre-emphasize the original audio, and the pre-emphasis can make the signal corresponding to the original audio flatter so that it can be processed later.

[0057] Frame segmentation processing refers to processing audio signals in units of frames based on the short-term stationary characteristics of audio signals. The preset frame length refers to the time length of one frame during frame segmentation processing. After the filtering result is processed into frames, N frame segmentation results are obtained, which are also N initial frames, where N is an integer greater than zero.

[0058] The preset window functions can include rectangular window, Hamming window, and Hanning window, etc. Windowing can be used to eliminate signal discontinuities at both ends of each initial frame.

[0059] Specifically, because the vocal cords and lips vibrate during the speaking process, the amplitude of the high-frequency formant in the original audio is lower than that of the low-frequency formant. Therefore, the original audio needs to be pre-emphasized, that is, filtered by a high-pass filter to eliminate the influence of vocal cord and lip vibration and to compensate for the high-frequency part of the original audio.

[0060] In this embodiment, the preset frame length can be set to 32ms. The implementer can adjust the value of the preset frame length according to the actual situation. It should be noted that in order to avoid missing the audio signal during subsequent windowing processing, it is necessary to set frame stacking. Frame stacking is the overlapping part between two adjacent initial frames. In this embodiment, the length of frame stacking is set to 16ms, that is, half of the preset frame length is taken as the length of frame stacking, so as to avoid excessive changes in signal characteristics between adjacent initial frames.

[0061] In this embodiment, the preset window function is a Hamming window. The Hamming window is used to perform windowing processing on N initial frames to obtain N windowing processing results. Then, a short-time Fourier transform is performed on each of the N windowing processing results.

[0062] This embodiment pre-emphasizes, frames, and windows the original audio before performing a short-time Fourier transform. This eliminates the noise effects of the original audio and improves the ability of the short-time Fourier transform spectrum obtained from the subsequent short-time Fourier transform to represent the original audio. This allows the multimodal representation extracted based on the short-time Fourier transform spectrum to provide more effective information for the emotion classification task, thereby improving the accuracy of emotion classification.

[0063] The above steps involve preprocessing the acquired raw audio, concatenating the preprocessed short-time Fourier transform spectrum, Mel spectrum, and Mel frequency cepstral coefficients to obtain the concatenated result. By obtaining the multimodal representation of the raw audio through preprocessing and concatenating the multimodal representations, the concatenated result more fully represents the information of the raw audio. This allows the sentiment classification model, which uses the concatenated result as input, to output a more accurate classification result, thereby improving the accuracy of sentiment classification.

[0064] Step S202: Input the splicing result into the trained self-attention layer for feature mapping, determine the feature mapping result as an attention vector, multiply the attention vector with the splicing result, input it into the trained convolutional layer for feature extraction, and obtain the audio features.

[0065] The trained self-attention layer can be used to extract the correlation between different parts of the input of the self-attention layer. The input of the self-attention layer is the concatenation result. The output of the trained self-attention layer is the attention vector, which can be used to characterize the degree of correlation between different parts of the input of the self-attention layer.

[0066] The trained convolutional layer can be used to extract features from the input of the convolutional layer to obtain deep features. The input of the trained convolutional layer is the result of multiplying the attention vector with the concatenation result, and the output of the trained convolutional layer is the audio feature.

[0067] Specifically, the trained self-attention layer can implement the self-attention mechanism using a 1*1 convolution. When the attention vector is multiplied by the concatenation result, the attention vector acts as a weight, enabling the subsequently trained convolutional layer to better focus on key information when extracting features.

[0068] In one implementation, the trained self-attention layer can also be implemented using the attention layer portion of the Transformer model to achieve the self-attention mechanism.

[0069] Optionally, inputting the concatenated result into the trained self-attention layer for feature mapping includes:

[0070] The concatenated result is input into the trained first feedforward network layer for feature aggregation to obtain the first feature aggregation result.

[0071] The first feature aggregation result and the concatenation result are superimposed and then input into the trained self-attention layer for feature mapping.

[0072] The first feedforward network layer, which is trained, can be used for feature aggregation. It is essentially a convolutional computation, but unlike the trained convolutional layer, the first feedforward network layer only has the ability to compute shallow features, while the trained convolutional layer contains multiple layers of convolutional processing and can obtain deeper features.

[0073] The first feature aggregation result can refer to the output result after inputting the spliced ​​result into the trained first feedforward network layer.

[0074] Specifically, the trained first feedforward network layer includes layer normalization, linear mapping, activation function processing, and random dropout processing. After the concatenation result is input into the trained first feedforward network layer, layer normalization is first performed. The purpose of layer normalization is to avoid gradient explosion and prevent gradient vanishing.

[0075] After layer normalization, a linear mapping is performed, which is the convolution calculation process described above. The result of the layer normalization is convolved with the parameters in the trained first feedforward network layer to obtain the linear mapping result. An activation function is then applied to the linear mapping result to improve its nonlinear representation capability. In this embodiment, the activation function can be a SWISH function, which can be expressed as f(x) = x * σ(x), where x can be the input of the activation function, i.e., the linear mapping result, and σ(x) can be the sigmoid function. Specifically, the sigmoid function can be expressed as...

[0076] Randomly discarding parameters can refer to randomly masking the parameters in the first feedforward network layer according to a preset ratio. In this embodiment, the preset ratio can be set to 0.5, which means masking half of the parameters in the trained first feedforward network layer. Masking can refer to setting the parameters to zero. Randomly discarding parameters can be used to improve the generalization ability of the model.

[0077] The first feature aggregation result and the splicing result can be superimposed using a weighted addition method. In this embodiment, when superimposing the first feature aggregation result and the splicing result, half of the first feature aggregation result is added to the splicing result. After obtaining the addition result, the addition result is input into the trained self-attention layer for feature mapping.

[0078] In this embodiment, before feature mapping is performed on the splicing result in the trained self-attention layer, the splicing result is pre-processed using a trained first feedforward neural network to obtain aggregated features of the splicing result. The aggregated features and the splicing result are then superimposed and used as input to the trained self-attention layer, thereby ensuring the reusability of features and avoiding degradation problems in the sentiment classification model, thus improving the accuracy of the sentiment classification model.

[0079] Optionally, the attention vector is multiplied by the concatenation result and then input into a trained convolutional layer for feature extraction, yielding audio features including:

[0080] The attention vector is multiplied by the concatenation result and input into the trained convolutional layer for feature extraction. The feature extraction result is then input into the trained second feedforward network layer for feature aggregation to obtain the second feature aggregation result.

[0081] The sum of the feature extraction result and the second feature aggregation result is determined as the audio feature.

[0082] The trained second feedforward network layer can also be used for feature aggregation. The second feature aggregation result can refer to the output result after the feature extraction result is input into the trained second feedforward network layer.

[0083] Specifically, the trained second feedforward network layer also includes layer normalization, linear mapping, activation function processing, and random dropout processing. After the feature extraction result is input into the trained second feedforward network layer, it first undergoes layer normalization. After layer normalization, linear mapping is performed, and activation function processing is applied to the linear mapping result to improve the nonlinear representation ability of the activation function processing result. The parameters in the second feedforward network layer are randomly masked according to a preset ratio.

[0084] The sum of the second feature aggregation result and the feature extraction result can be calculated by weighted addition. In this embodiment, when adding the second feature aggregation result and the feature extraction result, half of the second feature aggregation result is added to the feature extraction result. The sum result is then used as the audio feature.

[0085] In this embodiment, after the feature extraction results are obtained from the trained convolutional layer, the trained second feedforward neural network is used to post-process the feature extraction results to obtain aggregated features of the feature extraction results. Then, the aggregated features and the feature extraction results are added together as audio features, thereby ensuring the reusability of features, avoiding the degradation problem of the sentiment classification model, and thus improving the accuracy of the sentiment classification model.

[0086] Optionally, the trained attention layer includes a trained first embedding matrix, a trained second embedding matrix, and a trained third embedding matrix;

[0087] The concatenated result is input into the trained self-attention layer for feature mapping, including:

[0088] Multiply the concatenated result by the trained first embedding matrix to obtain the query vector; multiply the concatenated result by the trained second embedding matrix to obtain the key vector; and multiply the concatenated result by the trained third embedding matrix to obtain the value vector.

[0089] Multiply the transpose of the query vector and the key vector, normalize the result using a normalized exponential function, multiply the normalized result with the value vector, and determine the result as the feature mapping result.

[0090] Here, the trained first embedding matrix, the trained second embedding matrix, and the trained third embedding matrix can all refer to the parameters of the trained attention layer. The trained first embedding matrix, the trained second embedding matrix, and the trained third embedding matrix can all be used to perform linear transformation on the input concatenation result.

[0091] Specifically, the first, second, and third embedded matrices trained have the same matrix size. Therefore, the resulting query vector, key vector, and value vector all have the same size. According to the rules of vector operations, the final feature mapping result has the same size as the query vector, key vector, and value vector.

[0092] Since the query vector and the key vector have the same size, when the query vector and the key vector interact, the key vector needs to be transposed. The query vector and the transpose of the key vector are multiplied together. The result of the multiplication includes the attention weight of each part in the concatenation result to all parts. All parts include the part itself, where a single part can refer to the audio information corresponding to a single initial frame.

[0093] It should be noted that after obtaining the attention weights, the attention weights need to be scaled. The scaling process is to compare the attention weights with a preset value. In this embodiment, the preset value can be the square root of N to narrow the range of the multiplication result and determine the gradient stability of the subsequent normalization function processing.

[0094] The purpose of normalizing the exponential function is to ensure the non-negativity of attention weights and to increase the non-linearity of attention weights.

[0095] In this embodiment, the feature mapping result is calculated in the form of query vector, key-value vector and value vector, which can effectively capture the long-distance dependency relationship of the input, thereby improving the accuracy of subsequent sentiment classification.

[0096] The above steps involve inputting the concatenated result into a trained self-attention layer for feature mapping, determining the feature mapping result as an attention vector, multiplying the attention vector with the concatenated result, and inputting it into a trained convolutional layer for feature extraction to obtain audio features. By concatenating the trained attention layer and the trained convolutional layer, the sentiment classification model can effectively capture long-distance dependencies by relying on the trained attention layer, while also obtaining deeper features with stronger representation capabilities by relying on the trained convolutional layer, thereby improving the accuracy of sentiment classification in sentiment classification.

[0097] Step S203: Input the audio features into the trained speaker classifier and the trained gender classifier respectively for feature classification to obtain the reference speaker and reference gender.

[0098] Among them, the trained speaker classifier can be used to predict the speaker category of audio features. The speaker category can refer to the speaker's identity identifier. The trained gender classifier can be used to predict the speaker's gender of audio features. The speaker's gender can include male and female. The reference speaker can refer to the prediction result of the speaker's gender. The reference gender can refer to the prediction result of the speaker's gender.

[0099] Specifically, a speaker classifier and a gender classifier are added to the emotion classification model so that the audio features acquired by the trained emotion classification model can include the speaker's identity information and gender information, thereby improving the representation ability of the audio features.

[0100] It should be noted that by adding a speaker classifier and a gender classifier to the sentiment classification model, the loss function during the training process of the sentiment classification model includes sentiment classification loss, speaker classification loss, and gender classification loss. In other words, speaker classification loss and gender classification loss are added, which enables the sentiment classification model to focus on the ability of audio features to represent the speaker's identity information and gender information.

[0101] Optionally, the audio features are input into a pre-trained speaker classifier and a pre-trained gender classifier for feature classification, resulting in reference speaker and reference gender, including:

[0102] The audio features are input into the trained speaker classifier for feature classification, and the first predicted probability corresponding to each preset speaker category is obtained. All the first predicted probabilities and their corresponding preset speaker categories are determined to form a first probability distribution.

[0103] The audio features are input into the trained gender classifier for feature classification, and the second predicted probability corresponding to each preset gender category is obtained. All the second predicted probabilities and their corresponding preset gender categories are determined to form a second probability distribution.

[0104] The preset speaker category corresponding to the largest first predicted probability in the first probability distribution is determined as the reference speaker, and the preset gender category corresponding to the largest second predicted probability in the second probability distribution is determined as the reference gender;

[0105] Accordingly, the audio features, reference speaker, and reference gender are concatenated and then input into the trained sentiment classifier for classification, including:

[0106] The audio features, reference speaker, reference gender, first probability distribution, and second probability distribution are concatenated and then input into a trained sentiment classifier for classification.

[0107] The preset speaker category can refer to the speaker's identity identifier, which can be represented by a serial number, ID, etc. The first predicted probability can refer to the probability that the audio feature belongs to a preset speaker's category. The first probability distribution can refer to the probability distribution of the audio feature belonging to each preset speaker's category. The preset gender category can include male and female categories. The second predicted probability can refer to the probability that the audio feature belongs to a preset gender category. The second probability distribution can refer to the probability distribution of the audio feature belonging to each preset gender category.

[0108] Specifically, if there are C preset speaker categories, the output of the trained speaker classifier is the C first predicted probabilities corresponding to the preset speaker category. Similarly, the output of the trained gender classifier is the two second predicted probabilities corresponding to the preset gender category.

[0109] In this embodiment, the first probability distribution and the second probability distribution are also used as input to the trained sentiment classifier, so that the sentiment classification model has better generalization when faced with audio data corresponding to unprocessed speakers, thereby improving the accuracy of sentiment classification.

[0110] The steps described above, which input audio features into a trained speaker classifier and a trained gender classifier for feature classification to obtain a reference speaker and a reference gender, can provide speaker attribute information for subsequent emotion classification tasks, making the input of the trained emotion classifier richer and thus improving the accuracy of emotion classification.

[0111] Step S204: After concatenating the audio features, reference speaker, and reference gender, input the data into the trained emotion classifier for classification to obtain the classification result representing the emotion category of the original audio.

[0112] The trained emotion classifier can be implemented using models such as fully connected layers and decision trees. The classification result can refer to the emotion category of the original audio, which can include categories such as happy, sad, angry, and calm.

[0113] Specifically, the trained sentiment classifier outputs predicted values ​​for each preset sentiment category. A normalization exponential function is used to normalize all predicted values, and the preset sentiment category corresponding to the maximum value in the normalization result is determined as the classification result.

[0114] The steps described above, which concatenate audio features, reference speaker, and reference gender, and then input them into a trained emotion classifier for classification to obtain a classification result representing the emotion category of the original audio, integrate the reference speaker and reference gender with the audio features, thereby enriching the input of the trained emotion classifier and improving the accuracy of emotion classification.

[0115] This embodiment uses the multimodal representation of the original audio as input, which provides richer audio information and improves the accuracy of sentiment classification. In addition, it performs speaker and gender classification, and uses the classified reference speaker and reference gender as reference information to be concatenated with the audio features, which improves the generalization ability when performing sentiment classification processing on the original audio, thereby improving the accuracy of sentiment classification. This allows users to have a better user experience when using the financial service platform and reduces the probability of complaints against the financial service platform.

[0116] Corresponding to the AI-based sentiment classification method in the above embodiments, Figure 3 The diagram illustrates a structural block diagram of an artificial intelligence-based emotion classification device provided in Embodiment 2 of the present invention. This emotion classification device is applied to a client-side computer device connected to a server to obtain raw audio from the server. The raw audio can refer to audio that requires emotion classification. The client-side computer device is equipped with a pre-trained emotion classification model. This pre-trained model may include a pre-trained self-attention layer, a pre-trained convolutional layer, a pre-trained speaker classifier, a pre-trained gender classifier, and a pre-trained emotion classifier. The pre-trained emotion classification model can be used to classify the emotions of the input raw audio. For ease of explanation, only the parts relevant to the embodiments of the present invention are shown.

[0117] See Figure 3 The emotion classification device includes:

[0118] The preprocessing module 31 is used to preprocess the acquired raw audio, and to concatenate the preprocessed short-time Fourier transform spectrum, Mel spectrum and Mel frequency cepstral coefficients to obtain the concatenation result.

[0119] Feature extraction module 32 is used to input the splicing result into the trained self-attention layer for feature mapping, determine the feature mapping result as an attention vector, multiply the attention vector with the splicing result, input it into the trained convolutional layer for feature extraction, and obtain audio features;

[0120] The feature classification module 33 is used to input the audio features into the trained speaker classifier and the trained gender classifier respectively for feature classification, so as to obtain the reference speaker and reference gender.

[0121] The emotion classification module 34 is used to concatenate the audio features, reference speaker and reference gender, and then input them into the trained emotion classifier for classification to obtain the classification result representing the emotion category of the original audio.

[0122] Optionally, the preprocessing module 31 mentioned above includes:

[0123] The first transformation unit is used to perform a short-time Fourier transform on the original audio to obtain the short-time Fourier transform spectrum.

[0124] The spectrum mapping unit is used to map the short-time Fourier spectrum using a preset mapping function to determine the mapping result as the Mel spectrum. The mapping function contains the mapping relationship between the short-time Fourier spectrum and the Mel spectrum.

[0125] The second transformation unit is used to perform discrete cosine transform on the Mel spectrum to obtain the Mel frequency cepstral coefficients.

[0126] Optionally, the first transformation unit mentioned above includes:

[0127] The frame division subunit is used to filter the original audio input through a preset filter, and then divide the filtering result into frames according to a preset frame length to obtain N initial frames.

[0128] The windowing subunit is used to perform windowing processing on each initial frame using a preset window function, and to perform short-time Fourier transform on the windowing processing result.

[0129] Optionally, the feature extraction module 32 mentioned above includes:

[0130] The first feedforward unit is used to input the splicing result into the trained feedforward network layer for feature aggregation, and obtain the first feature aggregation result.

[0131] The feature mapping unit is used to superimpose the first feature aggregation result and the concatenation result, and then input them into the trained self-attention layer for feature mapping.

[0132] Optionally, the feature extraction module 32 mentioned above includes:

[0133] The second feedforward unit is used to multiply the attention vector with the concatenation result, input it into the trained convolutional layer for feature extraction, and input the feature extraction result into the trained feedforward network layer for feature aggregation to obtain the second feature aggregation result.

[0134] The feature determination unit is used to determine that the sum of the feature extraction result and the second feature aggregation result is the audio feature.

[0135] Optionally, the trained attention layer includes a trained first embedding matrix, a trained second embedding matrix, and a trained third embedding matrix;

[0136] The aforementioned feature extraction module 32 includes:

[0137] The matrix calculation unit is used to multiply the concatenated result with the trained first embedding matrix to obtain the query vector, multiply the concatenated result with the trained second embedding matrix to obtain the key vector, and multiply the concatenated result with the trained third embedding matrix to obtain the value vector.

[0138] The normalization unit is used to multiply the query vector and the transpose of the key value vector. The normalization exponent function is used to normalize the multiplication result. The normalized result is then multiplied with the value vector to determine the feature mapping result.

[0139] Optionally, the feature classification module 33 mentioned above includes:

[0140] The first distribution determination unit is used to input audio features into a trained speaker classifier for feature classification, obtain the first predicted probability corresponding to each preset speaker category, and determine that all the first predicted probabilities and their corresponding preset speaker categories form a first probability distribution.

[0141] The second distribution determination unit is used to input audio features into the trained gender classifier for feature classification, obtain the second prediction probability corresponding to each preset gender category, and determine that all the second prediction probabilities and their corresponding preset gender categories form the second probability distribution.

[0142] The category determination unit is used to determine the preset speaker category corresponding to the largest first predicted probability in the first probability distribution as the reference speaker, and to determine the preset gender category corresponding to the largest second predicted probability in the second probability distribution as the reference gender;

[0143] Accordingly, the aforementioned emotion classification module 34 includes:

[0144] The distribution splicing unit is used to splice audio features, reference speaker, reference gender, first probability distribution and second probability distribution, and then input them into the trained sentiment classifier for classification.

[0145] It should be noted that the information interaction and execution process between the above modules, units, and sub-units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0146] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the diagram), a memory, and a computer program stored in the memory and executable on at least one processor, which, when executed by the processor, implements the steps in any of the above-described emotion classification method embodiments.

[0147] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 4 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0148] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0149] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of a computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0150] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0151] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0152] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0153] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0154] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0155] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An artificial intelligence-based sentiment classification method, characterized in that, The sentiment classification method includes: The acquired raw audio is preprocessed, and the preprocessed short-time Fourier transform spectrum, Mel spectrum, and Mel frequency cepstral coefficients are concatenated to obtain the concatenated result. The splicing result is input into a trained self-attention layer for feature mapping. The feature mapping result is determined as an attention vector. The attention vector is multiplied by the splicing result and input into a trained convolutional layer for feature extraction to obtain audio features. The audio features are input into a trained speaker classifier and a trained gender classifier for feature classification to obtain a reference speaker and a reference gender. The audio features, the reference speaker, and the reference gender are concatenated and then input into a trained emotion classifier for classification to obtain a classification result representing the emotion category of the original audio. The step of inputting the concatenation result into the trained self-attention layer for feature mapping includes: The splicing result is input into the trained first feedforward network layer for feature aggregation to obtain the first feature aggregation result. The first feedforward network layer includes layer normalization processing, linear mapping processing, activation function processing, and random dropout processing. Half of the first feature aggregation result is added to the concatenation result to obtain the addition result, and the addition result is input into the trained self-attention layer for feature mapping; The step of multiplying the attention vector with the concatenation result and inputting it into a trained convolutional layer for feature extraction yields audio features including: The attention vector is multiplied by the concatenation result and input into the trained convolutional layer for feature extraction. The feature extraction result is then input into the trained second feedforward network layer for feature aggregation to obtain the second feature aggregation result. The second feedforward network layer includes layer normalization processing, linear mapping processing, activation function processing, and random dropout processing. The sum of the feature extraction result and half of the second feature aggregation result is determined as the audio feature.

2. The emotion classification method according to claim 1, characterized in that, The preprocessing of the acquired raw audio includes: Perform a short-time Fourier transform on the original audio to obtain the short-time Fourier transform spectrum; A preset mapping function is used to map the short-time Fourier transform spectrum, and the mapping result is determined to be the Mel spectrum. The mapping function contains the mapping relationship between the short-time Fourier transform spectrum and the Mel spectrum. The Mel frequency cepstral coefficients are obtained by performing a discrete cosine transform on the Mel spectrum.

3. The emotion classification method according to claim 2, characterized in that, The short-time Fourier transform of the original audio includes: The original audio is input into a preset filter for filtering, and the filtering result is divided into frames according to a preset frame length to obtain N initial frames. Each initial frame is windowed using a preset window function, and the windowing result is then subjected to a short-time Fourier transform.

4. The emotion classification method according to claim 1, characterized in that, The trained self-attention layer includes a trained first embedding matrix, a trained second embedding matrix, and a trained third embedding matrix; The step of inputting the concatenation result into the trained self-attention layer for feature mapping includes: Multiply the concatenated result by the trained first embedding matrix to obtain the query vector; multiply the concatenated result by the trained second embedding matrix to obtain the key vector; and multiply the concatenated result by the trained third embedding matrix to obtain the value vector. The query vector and the transpose of the key value vector are multiplied together, and the result of the multiplication is normalized using a normalized exponential function. The normalized result is then multiplied with the value vector to determine the feature mapping result.

5. The emotion classification method according to any one of claims 1 to 4, characterized in that, The step of inputting the audio features into a trained speaker classifier and a trained gender classifier for feature classification to obtain a reference speaker and reference gender includes: The audio features are input into the trained speaker classifier for feature classification to obtain the first predicted probability corresponding to each preset speaker category. All first predicted probabilities and their corresponding preset speaker categories are determined to form a first probability distribution. The audio features are input into the trained gender classifier for feature classification to obtain the second predicted probability corresponding to each preset gender category. All the second predicted probabilities and their corresponding preset gender categories are determined to form a second probability distribution. The preset speaker category corresponding to the largest first predicted probability in the first probability distribution is determined as the reference speaker, and the preset gender category corresponding to the largest second predicted probability in the second probability distribution is determined as the reference gender; Accordingly, the step of concatenating the audio features, the reference speaker, and the reference gender, and then inputting the result into the trained sentiment classifier for classification includes: The audio features, the reference speaker, the reference gender, the first probability distribution, and the second probability distribution are concatenated and then input into a trained sentiment classifier for classification.

6. An artificial intelligence-based emotion classification device, characterized in that, The emotion classification device includes: The preprocessing module is used to preprocess the acquired raw audio, and concatenate the preprocessed short-time Fourier transform spectrum, Mel spectrum and Mel frequency cepstral coefficients to obtain the concatenated result; The feature extraction module is used to input the splicing result into a trained self-attention layer for feature mapping, determine the feature mapping result as an attention vector, multiply the attention vector with the splicing result, and input it into a trained convolutional layer for feature extraction to obtain audio features; The feature classification module is used to input the audio features into a trained speaker classifier and a trained gender classifier respectively for feature classification to obtain a reference speaker and a reference gender. The emotion classification module is used to concatenate the audio features, the reference speaker, and the reference gender, and then input the result into a trained emotion classifier for classification to obtain a classification result representing the emotion category of the original audio. The feature extraction module includes: The first feedforward unit is used to input the splicing result into the trained first feedforward network layer for feature aggregation to obtain the first feature aggregation result. The first feedforward network layer includes layer normalization processing, linear mapping processing, activation function processing, and random discarding processing. The feature mapping unit is used to add half of the first feature aggregation result to the concatenation result to obtain the addition result, and input the addition result into the trained self-attention layer for feature mapping; The feature extraction module includes: The second feedforward unit is used to multiply the attention vector with the concatenation result, input the result into the trained convolutional layer for feature extraction, and input the feature extraction result into the trained second feedforward network layer for feature aggregation to obtain the second feature aggregation result. The second feedforward network layer includes layer normalization processing, linear mapping processing, activation function processing, and random dropout processing. The feature determination unit is used to determine that the sum of the feature extraction result and half of the second feature aggregation result is the audio feature.

7. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the emotion classification method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the emotion classification method as described in any one of claims 1 to 5.

Images