Audio emotion classification method and device, computer equipment and storage medium

Abstract

The invention discloses an audio emotion classification method and device, computer equipment and a storage medium, and relates to the technical field of data processing and the field of financial science and technology, and the method comprises the steps: obtaining target audio data to be subjected to emotion classification; performing feature extraction on the target audio data to obtain MFCC features; calculating a first-order derivative and a second-order derivative of the MFCC feature, and splicing the first-order derivative and the second-order derivative with the MFCC feature to obtain a 3D MFCC feature; and based on the 3D MFCC features, performing sentiment classification on the target audio data through an audio sentiment classification model. According to the method, the 3D MFCC features of the target audio data are obtained through processing, and then the emotion classification of the target audio data is realized by processing the 3D MFCC features through the audio emotion classification model, so that the calculation cost can be reduced and the processing efficiency can be improved while the audio emotion classification accuracy is ensured.

Description

Technical Field

[0001] The present invention relates to the fields of data processing technology and financial technology, and in particular to an audio emotion classification method, device, computer equipment and computer-readable storage medium. Background Art

[0002] At present, with the development of computer technology, more and more technologies are applied in the financial field. The traditional financial industry is gradually transforming into financial technology (Fintech), and audio sentiment classification technology is no exception. For example, audio sentiment classification technology can be applied to AI customer service application scenarios related to financial institutions (such as banks, etc.) to improve the service capabilities of AI customer service to customers. However, due to the security, real-time and other requirements of the financial industry, higher requirements are also placed on audio sentiment classification technology.

[0003] At present, with the rapid development of artificial intelligence technology, audio sentiment classification has become an important research field. Audio sentiment classification aims to automatically identify and classify the emotions expressed in audio signals, which has broad application prospects in many fields such as intelligent customer service, sentiment analysis, and health monitoring. At present, the commonly used audio sentiment classification methods in the industry mainly include the following two:

[0004] 1. Method based on speech features: This method mainly identifies emotions in audio by extracting direct features of audio signals, such as pitch, volume, speech speed and voice quality. These features can intuitively reflect the physical properties of speech signals, so this method is simple to operate and easy to implement; however, since the expression of emotions is not limited to the physical characteristics of sound, but is also affected by multiple factors such as the speaker's emotional state and context, this method has certain limitations in terms of the accuracy and robustness of emotion classification;

[0005] 2. Language model-based method: This method uses natural language processing (NLP) technology to perform speech recognition and semantic analysis on audio signals to identify emotions in the audio. By deeply analyzing the content and structure of the language, this method can more accurately capture the semantic information of emotions; however, due to differences in accents, tones, etc. among different individuals, these differences may affect the accuracy of speech recognition and semantic analysis, thereby affecting the effect of emotion classification.

[0006] It can be seen that among the existing audio emotion classification methods, most methods directly input audio signals for processing. Although the end-to-end method simplifies the classification process, its computational cost is significantly increased compared to the back-end model. Therefore, how to reduce the computational cost and improve the processing efficiency while ensuring the accuracy of audio emotion classification is a problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide an audio emotion classification method, apparatus, computer equipment and computer-readable storage medium, aiming to solve the problem of how to reduce computing costs and improve processing efficiency while ensuring the accuracy of audio emotion classification.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] In a first aspect, the present invention provides an audio emotion classification method, which comprises:

[0010] Obtain target audio data to be sentimentally classified;

[0011] Extracting features from the target audio data to obtain MFCC features;

[0012] Calculating the first-order derivative and the second-order derivative of the MFCC feature, and concatenating the first-order derivative and the second-order derivative with the MFCC feature to obtain a 3D MFCC feature;

[0013] Based on the 3D MFCC features, the target audio data is subjected to sentiment classification through an audio sentiment classification model.

[0014] In a second aspect, the present invention provides an audio emotion classification device, which includes:

[0015] A data acquisition module is used to acquire target audio data to be sentimentally classified;

[0016] A feature extraction module is used to extract features from the target audio data to obtain MFCC features;

[0017] A calculation and splicing module, used for calculating the first-order derivative and the second-order derivative of the MFCC feature, and splicing the first-order derivative and the second-order derivative with the MFCC feature to obtain a 3D MFCC feature;

[0018] The emotion classification module is used to perform emotion classification on the target audio data through an audio emotion classification model based on the 3D MFCC features.

[0019] In a third aspect, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the audio emotion classification method as described above when executing the computer program.

[0020] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program implements the audio emotion classification method as described above when executed by a processor.

[0021] Beneficial effects: Compared with the prior art, the present invention provides an audio emotion classification method, device, computer equipment and computer-readable storage medium, wherein the target audio data to be emotion classified is obtained; the target audio data is subjected to feature extraction to obtain MFCC features; the first-order derivative and the second-order derivative of the MFCC features are calculated, and the first-order derivative and the second-order derivative are concatenated with the MFCC features to obtain 3D MFCC features; based on the 3D MFCC features, the target audio data is subjected to emotion classification through an audio emotion classification model. In this way, the present invention first processes the 3D MFCC features of the target audio data, and then processes the 3D MFCC features through the audio emotion classification model to achieve emotion classification of the target audio data, so that the present invention can reduce the computational cost and improve the processing efficiency while ensuring the accuracy of audio emotion classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A schematic diagram of an application environment of an audio emotion classification method provided by an embodiment of the present invention.

[0024] Figure 2 A flowchart of an audio emotion classification method provided by an embodiment of the present invention.

[0025] Figure 3 A schematic diagram of a program module of an audio emotion classification device provided by an embodiment of the present invention.

[0026] Figure 4 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention.

[0027] Figure 5 Another structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0030] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0031] As used in the present specification and the appended claims, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0032] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0033] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0034] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0035] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0036] An audio emotion classification method provided by an embodiment of the present invention can be applied in Figure 1 In the application environment shown, the client communicates with the server through the network. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPC), netbooks, cloud computing devices, personal digital assistants (PDA) and other computer devices. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0037] See also Figure 2 An embodiment of the present invention provides an audio emotion classification method, wherein the method comprises the following steps:

[0038] S100, obtaining target audio data to be emotion classified;

[0039] S200, extracting features from the target audio data to obtain MFCC features;

[0040] S300, calculating the first-order derivative and the second-order derivative of the MFCC feature, and concatenating the first-order derivative and the second-order derivative with the MFCC feature to obtain a 3D MFCC feature;

[0041] S400: Based on the 3D MFCC features, sentiment classification is performed on the target audio data using an audio sentiment classification model.

[0042] In specific implementation, this embodiment first processes the 3D MFCC features of the target audio data, and then processes the 3D MFCC features through the audio emotion classification model to achieve emotion classification of the target audio data. Therefore, this embodiment can reduce computing costs and improve processing efficiency while ensuring the accuracy of audio emotion classification.

[0043] Furthermore, the audio emotion classification method, wherein the step S100, obtaining the target audio data to be emotion classified, specifically comprises the steps of:

[0044] Get the uploaded raw audio data to be sentimentally classified;

[0045] The original audio data is preprocessed to obtain target audio data.

[0046] In specific implementation, this embodiment can ensure that the quality of the target audio data obtained meets the needs of subsequent emotion classification and improve the availability and accuracy of the target audio data by obtaining the uploaded original audio data and preprocessing it; wherein the preprocessing step generally involves operations such as denoising, volume normalization, and format conversion, which help to eliminate background noise, adjust volume differences, and unify data formats, thereby providing clear and consistent input for the audio emotion classification model; in addition, preprocessing can also enhance the adaptability of the audio emotion classification model to different recording conditions and speaking styles, and improve the accuracy and robustness of emotion classification;

[0047] The specific implementation process of the steps in this embodiment is roughly as follows:

[0048] 1. Collect raw audio data

[0049] Purpose: Get the uploaded audio files to be sentimentally classified. These audio files will be used as input for sentiment classification, i.e. original audio data;

[0050] Operation: Provide an upload interface to allow uploading of audio files; support multiple audio formats, such as WAV, MP3, AAC, etc., to ensure compatibility; verify whether the uploaded audio files meet the preset format and quality standards;

[0051] 2. Audio data format conversion (optional)

[0052] Purpose: To ensure that all audio data is processed in a unified format to facilitate subsequent preprocessing and analysis;

[0053] Operation: Convert the uploaded audio file to a uniform sampling rate and bit depth, such as 16kHz sampling rate and 16-bit depth. You can use the audio processing library to perform format conversion.

[0054] 3. Denoising and background noise suppression

[0055] Purpose: To reduce the impact of non-speech components on sentiment classification and improve the quality of audio data;

[0056] Action: Apply denoising algorithms, such as spectral subtraction, Wiener filtering, etc., to remove background noise; use noise suppression algorithms, such as improved versions of spectral subtraction or deep learning methods, to more intelligently identify and suppress noise;

[0057] 4. Audio Segmentation

[0058] Purpose: To split long audio files into shorter audio clips containing independent emotional expressions;

[0059] Operation: Automatically or manually segment the audio file into appropriate segments based on the audio content and context; Voice Activity Detection (VAD) technology can be used to assist in identifying speech and non-speech segments;

[0060] 5. Volume normalization:

[0061] Purpose: To adjust the volume of audio data to a uniform volume level and reduce the impact of volume differences on subsequent processing;

[0062] Operation: Measure the peak or average volume of the audio data and use the gain control to adjust the volume; ensure that the volume adjustment does not introduce additional distortion or affect the audio quality;

[0063] Through the above process, this embodiment can obtain preprocessed target audio data, which is ready for subsequent feature extraction and emotion classification. The purpose of the preprocessing step is to ensure the quality of the audio data and provide accurate input for emotion classification.

[0064] Furthermore, the audio emotion classification method, wherein the step S200, extracting features from the target audio data to obtain MFCC features, specifically comprises the steps of:

[0065] Performing frame segmentation on the target audio data according to a preset frame length;

[0066] Performing a short-time Fourier transform on each frame of the target audio data to obtain a short-term power spectrum;

[0067] Filtering the short-term power spectrum using a Mel filter to obtain a Mel spectrum;

[0068] Taking the logarithm of the Mel spectrum to obtain a logarithmic Mel spectrum graph;

[0069] Perform discrete cosine transform on the logarithmic Mel spectrogram to obtain MFCC features.

[0070] Furthermore, the audio emotion classification method, wherein the short-time Fourier transform is performed on each frame of the target audio data to obtain a short-term power spectrum, specifically comprises the steps of:

[0071] Applying a Hanning window to each frame of the target audio data;

[0072] Performing a short-time Fourier transform on each frame of the target audio data to which the Hanning window is applied to obtain a short-term power spectrum.

[0073] In specific implementation, this embodiment can extract MFCC features representing key spectral features of the target audio data by dividing the target audio data into frames, applying a Hanning window, performing a short-time Fourier transform, filtering using a Mel filter, taking a logarithm, and performing a discrete cosine transform. These features can capture the dynamic and static characteristics of the target audio data and simulate the human ear's perception of sound, thereby providing rich and discriminative information for audio emotion classification. This embodiment can highlight emotion-related features in audio data and improve the accuracy and efficiency of emotion recognition.

[0074] The specific implementation process of the steps in this embodiment is roughly as follows:

[0075] 1. Frame segmentation

[0076] Purpose: To divide the continuous target audio data into shorter frames for local frequency domain analysis;

[0077] Operation: Determine the frame length, which is usually 20-40 milliseconds, corresponding to 256-1024 sample points of the audio sampling rate; split the target audio data into multiple overlapping frames according to the frame length and frame shift (usually half or 75% of the frame length);

[0078] 2. Apply window function

[0079] Purpose: Reduce spectral leakage at frame boundaries and smooth the start and end of frames;

[0080] Operation: Apply a Hanning window to each frame. This is a commonly used window function that is cosine-shaped in the time domain and close to zero at both ends, which helps reduce boundary effects.

[0081] 3. Short-time Fourier transform

[0082] Purpose: Convert the time domain signal of each frame into a frequency domain signal for frequency analysis;

[0083] Operation: Apply short-time Fourier transform to each frame to obtain the short-term power spectrum; the short-term power spectrum is a comprehensive representation of the sound characteristics, including two-dimensional spatial information in the frequency domain and time domain;

[0084] 4. Mel filter filtering

[0085] Purpose: To simulate the human ear's perception of sounds of different frequencies and convert the spectrum into Mel spectrum;

[0086] Operation: Use a set of Mel filters to filter the short-term power spectrum. These filters are usually designed based on the Mel scale to simulate the nonlinear perception of frequency by the human ear. Each filter outputs the accumulation of energy in the corresponding frequency range to obtain the Mel spectrum.

[0087] 5. Logarithmic Transform and Discrete Cosine Transform

[0088] Purpose: Extract MFCC features that capture the key spectral characteristics of the target audio data;

[0089] Operation: Take the logarithm of the output of the Mel filter (i.e., the Mel spectrum) to obtain the logarithmic Mel spectrum. This step is to simulate the nonlinear perception of the human ear to the sound intensity.

[0090] Perform discrete cosine transform (DCT) on the logarithmic Mel spectrum, and usually take the first 13 coefficients as MFCC features; DCT helps reduce data dimensions and extract the most important spectral features;

[0091] Through the above process, this embodiment can extract corresponding MFCC (Mel-Frequency Cepstral Coefficients) features from the target audio data. These features are helpful for subsequent audio processing tasks such as emotion classification. Among them, MFCC is the abbreviation of "Mel-Frequency Cepstral Coefficients", which is a feature description technology commonly used in the fields of speech recognition and audio analysis. MFCC features can capture important characteristics of human speech signals and simulate the nonlinear perceptual characteristics of the human auditory system.

[0092] Further, the audio emotion classification method, wherein the step S300, calculating the first-order derivative and the second-order derivative of the MFCC feature, and concatenating the first-order derivative and the second-order derivative with the MFCC feature to obtain a 3D MFCC feature, specifically comprises the steps of:

[0093] Differentiating the feature sequence of the MFCC features to obtain a first-order derivative;

[0094] Differentiating the first-order derivative again to obtain a second-order derivative;

[0095] The first-order derivative and the second-order derivative are concatenated with the MFCC feature to obtain a 3DM MFCC feature.

[0096] In specific implementation, this embodiment differentiates the feature sequence of MFCC features to obtain first-order and second-order derivatives, and then splices the first-order and second-order derivatives with the original MFCC features to form 3D MFCC features. The 3D MFCC features can capture the emotional changes and dynamic characteristics in the target audio data. This embodiment not only retains the spectral information of the target audio data, but also adds a description of the changing trend of the audio signal, thereby providing a richer and more detailed feature representation for audio emotion classification, and enhancing the sensitivity of audio emotion classification to emotional changes and the accuracy of classification; this embodiment can more accurately identify and distinguish different emotional states through this feature enhancement, especially when processing audio data with complex emotional expressions, the 3D MFCC features can provide more powerful support;

[0097] The specific implementation process of the steps in this embodiment is roughly as follows:

[0098] 1. Calculate the first-order derivative

[0099] Purpose: To capture the changes of MFCC features over time, that is, the dynamic characteristics of the target audio data;

[0100] Operation: For each MFCC coefficient in the feature sequence, calculate the difference between adjacent frames; for example, for the MFCC coefficient MFCC of the tth frame t , whose first-order derivative can be obtained by ΔMFCC t =MFCC t+1 -MFCC t Calculated;

[0101] Handle boundary cases, such as the first and last frames of a feature sequence, which do not have enough adjacent frames for differencing; typically, this can be done by using forward or backward differencing, or by padding with zeros at the beginning and end of the feature sequence;

[0102] 2. Calculate the second-order derivative

[0103] Purpose: To capture the change of the first-order derivative (i.e., the rate of change of MFCC) and provide information on the acceleration change of the target audio data;

[0104] Operation: Differentiate the first-order derivative again; for the first-order derivative ΔMFCC of the t-th frame t , whose second-order derivative can be obtained by Δ 2 MFCC t =ΔMFCC t+1 -ΔMFCC t Calculated;

[0105] Likewise, edge cases need to be handled, and padding or special processing may be required to avoid computational errors;

[0106] 3. Concatenate first-order derivatives, second-order derivatives and MFCC features

[0107] Purpose: Combine static features (MFCC features) with dynamic features (first-order derivatives and second-order derivatives) to provide a more comprehensive description of audio data;

[0108] Operation: Concatenate MFCC features, first-order derivatives, and second-order derivatives along the feature dimension to form 3D MFCC features. This three-dimensional feature contains static and dynamic information of the target audio data and is very useful for tasks such as audio sentiment classification.

[0109] Ensure that the frame numbers of all features remain consistent during the stitching process to facilitate subsequent processing;

[0110] 4. Normalization (optional)

[0111] Purpose: To improve the numerical stability of 3D MFCC features and reduce the impact of different speakers or recording conditions on the features;

[0112] Operation: Normalize the concatenated 3D MFCC features, for example, using maximum-minimum normalization or Z-score normalization;

[0113] Through the above process, this embodiment can effectively calculate the first-order derivative and the second-order derivative from the MFCC features, and combine them with the original MFCC features to form 3D MFCC features. The 3D MFCC features include not only the static information of the target audio data, but also the dynamic information, thereby providing strong support for subsequent tasks such as audio emotion classification.

[0114] Furthermore, the audio emotion classification method, wherein the step S400, based on the 3D MFCC features, performs emotion classification on the target audio data through an audio emotion classification model, specifically comprises the steps of:

[0115] Pre-built audio sentiment classification model;

[0116] Inputting the 3D MFCC features into the audio emotion classification model to generate an emotion classification result of the target audio data;

[0117] The sentiment classification result is output or displayed.

[0118] Furthermore, the audio emotion classification method, wherein the pre-constructing the audio emotion classification model specifically comprises the steps of:

[0119] Pre-build a deep learning model based on the residual network;

[0120] The deep learning model is trained and tested according to the collected audio emotion classification samples, and when the deep learning model meets the preset performance requirements, an audio emotion classification model is generated.

[0121] In specific implementation, this embodiment pre-builds a deep learning model based on a residual network, and uses the collected audio emotion classification samples for training and testing, so as to ensure that the deep learning model is used for actual audio emotion classification after meeting the preset performance requirements; by inputting 3D MFCC features into the trained audio emotion classification, the target audio data can be effectively emotion-classified, and the emotion classification results can be output or displayed. This embodiment provides an automated, efficient and accurate audio emotion classification solution, which can be widely used in scenarios such as intelligent voice assistants, emotion analysis tools and human-computer interaction systems, and enhances customer experience and system decision-making capabilities;

[0122] The specific implementation process of the steps in this embodiment is roughly as follows:

[0123] 1. Build a residual network model

[0124] Objective: To design and implement a deep learning model that can learn the characteristics of audio data and perform sentiment classification.

[0125] Operation: Select or design a residual network architecture suitable for audio processing; the residual network (ResNet) solves the gradient vanishing problem in deep network training by introducing skip connections;

[0126] Define network layers, including convolutional layers, activation functions (such as ReLU), batch normalization layers, skip connections, and fully connected layers and softmax output layers for classification;

[0127] Among them, the residual network (ResNet) is a deep convolutional neural network architecture that solves the common gradient vanishing and gradient exploding problems when training very deep networks by introducing skip connections (also called residual connections). These connections allow the signal in the network to bypass one or more layers and pass directly, so that the network can learn the identity mapping and facilitate the direct propagation of gradients. ResNet performs well in tasks such as speech processing and image recognition, so it has been widely used in the computer field.

[0128] 2. Training and testing deep learning models

[0129] Purpose: To train a model using collected audio sentiment classification samples and test it to evaluate its performance;

[0130] Operation: Prepare training and testing datasets, including audio samples and their corresponding emotion labels;

[0131] Use the training data set to train the deep learning model and optimize the model parameters through back propagation and gradient descent algorithms;

[0132] Evaluate the performance of deep learning models on the test dataset using metrics such as accuracy, recall, and F1 score;

[0133] 3. Generate audio sentiment classification model

[0134] Purpose: When the trained deep learning model meets the preset performance requirements, save it as an audio sentiment classification model that can be used for actual audio sentiment classification;

[0135] Operation: When it is determined that the performance of the deep learning model meets the preset accuracy, recall and other performance requirements, save the trained model parameters and obtain the corresponding audio emotion classification model;

[0136] 4. Sentiment Classification

[0137] Purpose: Input 3D MFCC features into the trained audio sentiment classification model to generate sentiment classification results for the target audio data;

[0138] Operation: Input the 3D MFCC features into the generated audio sentiment classification model, which will output the sentiment classification results of the target audio data;

[0139] 5. Output or display sentiment classification results

[0140] Purpose: To display the results of sentiment classification to users in an appropriate manner or for further processing;

[0141] Operation: Display the sentiment classification results to the user in a visual form; of course, the sentiment classification results can also be output to a specified location as needed;

[0142] If necessary, the sentiment classification results can be saved to a file or database for subsequent analysis or application;

[0143] Through the above process, this embodiment can construct and use a deep learning model based on a residual network (i.e., an audio sentiment classification model) to automatically classify the emotions in the target audio data, thereby realizing the automation and intelligence of audio sentiment analysis.

[0144] It can be seen from the above method embodiments that the audio emotion classification method provided by the present invention includes: obtaining target audio data to be emotion classified; extracting features from the target audio data to obtain MFCC features; calculating the first-order derivative and the second-order derivative of the MFCC features, and splicing the first-order derivative and the second-order derivative with the MFCC features to obtain 3D MFCC features; based on the 3D MFCC features, emotion classification is performed on the target audio data through an audio emotion classification model. In this way, the method of the present invention first processes to obtain the 3D MFCC features of the target audio data, and then processes the 3D MFCC features through the audio emotion classification model to achieve emotion classification of the target audio data, so that the method of the present invention can reduce the computational cost and improve the processing efficiency while ensuring the accuracy of audio emotion classification.

[0145] Specifically, the audio emotion classification method provided by the embodiment of the present invention can at least achieve the following beneficial technical effects:

[0146] 1. Low computational cost. At the front end of the network, extracting features in advance based on the method of the present invention can greatly reduce the number of parameters and computational cost of the back end model;

[0147] 2. The model input is the 3D MFCC feature of the audio data. The method of the present invention constructs a lighter-weighted deep residual network model (audio sentiment classification model) for this feature. The model has strong adaptability and generalization ability and performs well in audio sentiment classification.

[0148] 3. Strong robustness. The 3D MFCC features of the audio data obtained by the method of the present invention can effectively remove noise and other interferences, and also have good robustness to different speech speeds and pitch changes;

[0149] 4. Adaptable to different speech signals: 3D MFCC features and residual networks can adapt to different speech signals, including spoken language, dialects, and different music styles. The method of the present invention can well handle individual differences in speech data and differences between different speakers.

[0150] It can be understood that the audio sentiment classification method provided by the embodiment of the present invention can be applied to audio sentiment classification scenarios related to the field of financial technology. Specifically, the audio sentiment classification function can be applied to AI customer service application scenarios related to financial institutions (such as banks, etc.). Through the audio sentiment classification and recognition of customers by AI customer service, the service capabilities of AI customer service of financial institutions for customers can be improved.

[0151] Of course, the audio emotion classification method provided by the embodiment of the present invention can also have a variety of application scenarios in the field of financial technology. The following are some common examples:

[0152] 1. Customer Service Sentiment Analysis:

[0153] In the customer service hotline of financial institutions, by analyzing the emotions in the customer's voice, the customer's dissatisfaction or anger can be detected in real time, and the customer service staff can be notified in time to take corresponding measures to improve customer satisfaction;

[0154] 2. Intelligent Voice Assistant:

[0155] In the intelligent voice assistants of financial institutions, through sentiment analysis, intelligent voice assistants can provide more personalized services based on customers' emotions, such as providing comfort or advice when customers are depressed;

[0156] 3. Transaction monitoring:

[0157] In financial transactions at financial institutions, by analyzing the voice emotions of traders or customers, abnormal emotions such as excessive excitement or panic can be detected, which may indicate potential improper trading behavior or other behaviors;

[0158] 4. Loan approval:

[0159] In the loan approval process of financial institutions, analyzing the applicant's voice emotion can help assess their psychological state and willingness to repay, which serves as an auxiliary factor in credit risk assessment;

[0160] 5. Market sentiment analysis:

[0161] By analyzing audio content from sources such as financial news, investor conference calls, and social media, market sentiment can be captured to provide reference for investment decisions;

[0162] 6. Employee Emotion Monitoring:

[0163] In the financial industry, by monitoring employees’ voice emotions in the workplace, work stress and employee satisfaction can be assessed, thereby optimizing the work environment and improving employee efficiency;

[0164] 7. Financial risk detection:

[0165] In the case of telephone risk or network risk of financial institutions, analyzing emotional features in voice can help identify financial risk behaviors, especially in situations involving financial transactions;

[0166] 8. Customer feedback analysis:

[0167] By analyzing voice sentiment in customer feedback, financial institutions can better understand customer needs and preferences, thereby improving products and services;

[0168] These application scenarios demonstrate the broad potential of audio sentiment classification technology in the field of financial technology, which can help improve service quality, enhance risk management, optimize decision-making processes, and ultimately enhance customer experience and business efficiency.

[0169] It should be understood that, although the present application provides method operation steps as described in the embodiments or flowcharts, more or less operation steps may be included based on conventional or non-creative labor, and these operation steps are not necessarily performed in sequence according to the embodiment or flowchart. The order of steps listed in the embodiment or flowchart is only one way of executing the order of many steps, and does not represent the only execution order. It should be noted that there is not necessarily a certain order between the above steps. A person of ordinary skill in the art can understand from the description of the embodiment of the present invention that in different embodiments, the above steps may have different execution orders, that is, they may be executed in parallel, or they may be executed in exchange, etc. Moreover, at least a part of the steps in the embodiment or flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but may be executed in turn, alternately or synchronously with other steps or at least a part of the sub-steps or stages of other steps.

[0170] Based on the above method embodiment, please refer to Figure 3 Another embodiment of the present invention further provides an audio emotion classification device, wherein the device comprises:

[0171] A data acquisition module 11 is used to acquire target audio data to be emotion classified;

[0172] A feature extraction module 12 is used to extract features from the target audio data to obtain MFCC features;

[0173] A calculation and splicing module 13, used for calculating the first-order derivative and the second-order derivative of the MFCC feature, and splicing the first-order derivative and the second-order derivative with the MFCC feature to obtain a 3D MFCC feature;

[0174] The emotion classification module 14 is used to perform emotion classification on the target audio data through an audio emotion classification model based on the 3D MFCC features.

[0175] Furthermore, in the audio emotion classification device, the data acquisition module 11 is specifically used for:

[0176] Get the uploaded raw audio data to be sentimentally classified;

[0177] The original audio data is preprocessed to obtain target audio data.

[0178] Furthermore, in the audio emotion classification device, the feature extraction module 12 is specifically used for:

[0179] Performing frame segmentation on the target audio data according to a preset frame length;

[0180] Performing a short-time Fourier transform on each frame of the target audio data to obtain a short-term power spectrum;

[0181] Filtering the short-term power spectrum using a Mel filter to obtain a Mel spectrum;

[0182] Taking the logarithm of the Mel spectrum to obtain a logarithmic Mel spectrum graph;

[0183] Perform discrete cosine transform on the logarithmic Mel spectrogram to obtain MFCC features.

[0184] Furthermore, the audio emotion classification device, wherein the short-time Fourier transform is performed on each frame of the target audio data to obtain a short-term power spectrum, comprises:

[0185] Applying a Hanning window to each frame of the target audio data;

[0186] Performing a short-time Fourier transform on each frame of the target audio data to which the Hanning window is applied to obtain a short-term power spectrum.

[0187] Furthermore, in the audio emotion classification device, the calculation and splicing module 13 is specifically used for:

[0188] Differentiating the feature sequence of the MFCC features to obtain a first-order derivative;

[0189] Differentiating the first-order derivative again to obtain a second-order derivative;

[0190] The first-order derivative and the second-order derivative are concatenated with the MFCC feature to obtain a 3DM MFCC feature.

[0191] Furthermore, in the audio emotion classification device, the emotion classification module 14 is specifically used for:

[0192] Pre-built audio sentiment classification model;

[0193] Inputting the 3D MFCC features into the audio emotion classification model to generate an emotion classification result of the target audio data;

[0194] The sentiment classification result is output or displayed.

[0195] Furthermore, in the audio emotion classification device, the pre-constructed audio emotion classification model comprises:

[0196] Pre-build a deep learning model based on the residual network;

[0197] The deep learning model is trained and tested according to the collected audio emotion classification samples, and when the deep learning model meets the preset performance requirements, an audio emotion classification model is generated.

[0198] It should be noted that in the embodiment of the device of the present invention, the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the aforementioned method embodiment part and will not be repeated here.

[0199] Based on the above method embodiment, another embodiment of the present invention further provides a computer device, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of the audio emotion classification method in any of the above method embodiments are implemented.

[0200] Based on the above method embodiment, another embodiment of the present invention further provides a computer device, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the functions or steps of the client side of the audio emotion classification method in any of the above method embodiments are implemented.

[0201] Those skilled in the art will understand that Figure 4 and Figure 5The structural schematic diagram shown in the figure is only a schematic diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0202] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0203] Among them, the memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be the memory of a computer device, and the internal memory provides an environment for the operation of an operating system and computer-readable instructions in the readable storage medium. The readable storage medium can be a hard disk of a computer device, and in other embodiments, it can also be an external storage device of a computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on a computer device. Further, the memory can also include both an internal storage unit of a computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory can also be used to temporarily store data that has been output or is to be output.

[0204] Based on the above method embodiments, another embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the audio emotion classification method in any of the above method embodiments is implemented. The computer-readable storage medium may be non-volatile or volatile.

[0205] It should be noted that the above-mentioned functions or steps that can be implemented by the computer-readable storage medium or computer device, and the technical effects brought about by the functions / steps, can be found in the relevant descriptions in the aforementioned method embodiments. To avoid repetition, they will not be described one by one here.

[0206] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM). The disclosed memory components or memories of the operating environments described herein are intended to comprise one or more of these and / or any other suitable types of memory.

[0207] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0208] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0209] In the embodiments provided by the present invention, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0210] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0211] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. An audio emotion classification method, characterized in that: include: Obtain target audio data to be sentimentally classified; Extracting features from the target audio data to obtain MFCC features; Calculating the first-order derivative and the second-order derivative of the MFCC feature, and concatenating the first-order derivative and the second-order derivative with the MFCC feature to obtain a 3D MFCC feature; Based on the 3D MFCC features, the target audio data is subjected to sentiment classification through an audio sentiment classification model.

2. The audio emotion classification method according to claim 1, characterized in that: The step of obtaining target audio data to be emotion classified includes: Get the uploaded raw audio data to be sentimentally classified; The original audio data is preprocessed to obtain target audio data.

3. The audio emotion classification method according to claim 1, characterized in that: The extracting features of the target audio data to obtain MFCC features includes: Performing frame segmentation on the target audio data according to a preset frame length; Performing a short-time Fourier transform on each frame of the target audio data to obtain a short-term power spectrum; Filtering the short-term power spectrum using a Mel filter to obtain a Mel spectrum; Taking the logarithm of the Mel spectrum to obtain a logarithmic Mel spectrum graph; Perform discrete cosine transform on the logarithmic Mel spectrogram to obtain MFCC features.

4. The audio emotion classification method according to claim 3, characterized in that: The performing a short-time Fourier transform on each frame of the target audio data to obtain a short-term power spectrum includes: Applying a Hanning window to each frame of the target audio data; Performing a short-time Fourier transform on each frame of the target audio data to which the Hanning window is applied to obtain a short-term power spectrum.

5. The audio emotion classification method according to claim 1, characterized in that: The calculating the first-order derivative and the second-order derivative of the MFCC feature, and concatenating the first-order derivative and the second-order derivative with the MFCC feature to obtain a 3DM MFCC feature, includes: Differentiating the feature sequence of the MFCC features to obtain a first-order derivative; Differentiating the first-order derivative again to obtain a second-order derivative; The first-order derivative and the second-order derivative are concatenated with the MFCC feature to obtain a 3DM MFCC feature.

6. The audio emotion classification method according to any one of claims 1 to 5, characterized in that: The step of performing emotion classification on the target audio data based on the 3DM FCC feature by using an audio emotion classification model includes: Pre-built audio sentiment classification model; Inputting the 3D MFCC features into the audio emotion classification model to generate an emotion classification result of the target audio data; The sentiment classification result is output or displayed.

7. The audio emotion classification method according to claim 6, characterized in that: The pre-built audio sentiment classification model includes: Pre-build a deep learning model based on the residual network; The deep learning model is trained and tested according to the collected audio emotion classification samples, and when the deep learning model meets the preset performance requirements, an audio emotion classification model is generated.

8. An audio emotion classification device, characterized in that: include: A data acquisition module is used to acquire target audio data to be sentimentally classified; A feature extraction module is used to extract features from the target audio data to obtain MFCC features; A calculation and splicing module, used for calculating the first-order derivative and the second-order derivative of the MFCC feature, and splicing the first-order derivative and the second-order derivative with the MFCC feature to obtain a 3D MFCC feature; The emotion classification module is used to perform emotion classification on the target audio data through an audio emotion classification model based on the 3D MFCC features.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the audio emotion classification method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the audio emotion classification method according to any one of claims 1 to 7 is implemented.

Images