Audio emotion recognition method and device, equipment and medium

Abstract

The invention belongs to the field of artificial intelligence, and relates to an audio emotion recognition method, which comprises the following steps: acquiring original audio data of a target object; preprocessing the original audio data to obtain preprocessed audio data; performing feature extraction on the audio data by adopting a pre-trained self-supervised model to obtain audio feature information of the audio data; performing dimension reduction processing on the audio feature information to obtain an emotion feature vector of the audio data; based on the emotion feature vector, performing emotion classification on the audio data by adopting a preset emotion classification model to obtain an emotion classification result; and determining an emotion type represented by the audio data according to an emotion classification result. The invention further provides a device, equipment and a medium. In addition, the invention also relates to a block chain technology, and the original audio data can be stored in a block chain. The accuracy of audio emotion recognition can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an audio emotion recognition method, device, equipment and medium. Background Art

[0002] In the field of speech emotion recognition technology, traditional machine learning methods have been widely used. Traditional machine learning methods mainly rely on manually extracted features, such as Mel frequency cepstral coefficients, pitch, and zero crossing rate. However, the extraction process of these features is not only complicated and cumbersome, but also has relatively limited ability to capture emotions, making it difficult to meet the high-precision requirements for emotion recognition in practical applications.

[0003] In recent years, with the rise of deep learning technology, models such as convolutional neural networks and recurrent neural networks have shown significant advantages in feature extraction. Despite this, these deep learning models still face many challenges. On the one hand, they mainly focus on capturing local features and have limited ability to evaluate long-term dependencies; on the other hand, these models usually require a large amount of labeled data for training and are not robust enough when dealing with noise interference and complex environmental changes. These limitations result in the performance of deep learning models in actual speech emotion recognition applications often failing to meet expectations, especially in noisy environments or scenarios with complex and changing emotional expressions, where recognition accuracy and stability are greatly reduced.

[0004] In summary, there is an urgent need for a method that can solve the problems of traditional methods' complex feature extraction and limited emotion capture capabilities, while deep learning models are limited by local feature capture, long-term dependency evaluation, and insufficient robustness. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose an audio emotion recognition method, device, equipment and medium to solve the problem that the feature extraction using existing methods is complex and the emotion capture ability is limited, while the deep learning model is limited by local feature capture, long-term dependency evaluation and insufficient robustness.

[0006] In order to solve the above technical problems, the present application embodiment provides an audio emotion recognition method, which adopts the following technical solutions:

[0007] Obtain the original audio data of the target object; preprocess the original audio data to obtain preprocessed audio data; use a pre-trained self-supervised model to extract features from the audio data to obtain audio feature information of the audio data; perform dimensionality reduction processing on the audio feature information to obtain a sentiment feature vector of the audio data; based on the sentiment feature vector, use a preset sentiment classification model to sentimentally classify the audio data to obtain a sentiment classification result; determine the sentiment type represented by the audio data based on the sentiment classification result.

[0008] Further, the step of preprocessing the original audio data to obtain preprocessed audio data specifically includes:

[0009] The original audio data is sampled and converted to a uniform sampling rate to obtain audio data with a target sampling rate. The audio data is denoised using a preset denoising algorithm to obtain preprocessed audio data.

[0010] Furthermore, before the step of using the pre-trained self-supervisory model to extract features from the audio data to obtain audio feature information of the audio data, the method further includes:

[0011] Acquire audio training data, preprocess the audio training data, and obtain preprocessed target audio data; use a preset self-supervised model to perform self-supervised learning training on the target audio data to obtain a pre-trained self-supervised model;

[0012] The steps of extracting features from audio data using a pre-trained self-supervised model to obtain audio feature information of the audio data specifically include:

[0013] Input the audio data into the pre-trained self-supervised model and output the audio feature information of the audio data.

[0014] Further, the step of performing dimensionality reduction processing on the audio feature information to obtain the emotional feature vector of the audio data specifically includes:

[0015] Calculate the statistics of the audio feature information, including the mean and standard deviation; map the audio feature information to a low-dimensional space based on the mean and standard deviation to obtain the emotional feature vector of the audio data.

[0016] Furthermore, before the step of performing emotion classification on the audio data based on the emotion feature vector and using a preset emotion classification model to obtain the emotion classification result, the method further includes:

[0017] An audio training data set is obtained, the audio training data set includes multiple audio samples, each audio sample has a corresponding emotion label; a pre-trained audio processing model is used to extract audio features of the audio sample to obtain a first audio feature vector of the audio sample, and a preset principal component analysis method is used to perform dimensionality reduction processing on the audio feature vector to obtain a first audio feature vector after dimensionality reduction; a preset support vector machine model is trained according to the first audio feature vector and the emotion label to obtain an initial emotion classification model; a preset data enhancement technology is used to perform data enhancement processing on the audio training data set to obtain an enhanced target audio data set, and the target audio data set includes multiple audio enhancement samples; an audio processing model is used to extract audio features of multiple audio enhancement samples to obtain a second audio feature vector of multiple audio enhancement samples, and a principal component analysis method is used to perform dimensionality reduction processing on the second audio feature vector to obtain a second audio feature vector after dimensionality reduction; the initial emotion classification model is fine-tuned according to the second audio feature vector after dimensionality reduction and the emotion label to obtain a preset emotion classification model.

[0018] Furthermore, based on the emotion feature vector, a preset emotion classification model is used to perform emotion classification on the audio data to obtain the emotion classification result, which specifically includes:

[0019] The emotion feature vector is input into the preset emotion classification model to obtain the probability values ​​of multiple emotion types of the audio data; the emotion type with a probability value greater than a preset probability threshold among the multiple emotion types is determined as the preliminary target emotion type of the audio data; based on the probability value, the emotion classification confidence of the target emotion type is calculated; if the emotion classification confidence is greater than the preset confidence threshold, the target emotion type is determined as the final emotion type of the audio data to obtain the emotion classification result.

[0020] Furthermore, after the step of determining the emotion type represented by the audio data according to the emotion classification result, the method further includes:

[0021] Obtain the historical emotion type corresponding to the historical audio data of the target object, where the historical audio data and the original audio data are continuous audio data; generate an emotion trend curve representing the emotion change trend of the target object based on the historical emotion type and the target emotion type; generate communication strategy recommendations based on the emotion trend curve.

[0022] In order to solve the above technical problems, the embodiment of the present application also provides an audio emotion recognition device, which adopts the following technical solutions:

[0023] An acquisition module, used to acquire the original audio data of the target object;

[0024] A preprocessing module, used to preprocess the original audio data to obtain preprocessed audio data;

[0025] A feature extraction module, used to extract features from audio data using a pre-trained self-supervised model to obtain audio feature information of the audio data;

[0026] A dimension reduction module is used to perform dimension reduction processing on audio feature information to obtain the emotional feature vector of audio data;

[0027] The classification module is used to perform emotion classification on the audio data based on the emotion feature vector and adopt a preset emotion classification model to obtain the emotion classification result;

[0028] The type determination module is used to determine the emotion type represented by the audio data based on the emotion classification result.

[0029] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the above-mentioned audio emotion recognition method are implemented.

[0030] In order to solve the above technical problems, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable at least one processor to perform the steps of the above-mentioned audio emotion recognition method.

[0031] Compared with the prior art, the embodiment of the present application has the following beneficial effects: through a series of innovative steps, the accuracy and robustness of audio emotion recognition are significantly improved. First, by acquiring and preprocessing the original audio data of the target object, a solid foundation is laid for subsequent processing. The pre-trained self-supervised model is used for feature extraction. This step abandons the complex manual feature extraction method with limited emotion capture ability in the traditional method, and instead uses the self-supervised learning method to automatically mine the deep emotional features in the speech signal. Not only does it simplify the feature extraction process, but it also greatly broadens the feature dimension, so that the model can capture more delicate and comprehensive emotional information. At the same time, the self-supervised model is pre-trained on an unlabeled large-scale data set, which gives it a strong generalization ability and effectively alleviates the problem of deep learning models relying on a large amount of labeled data. Further, by performing dimensionality reduction processing on the extracted audio feature information, a refined emotional feature vector is obtained, which helps to improve the efficiency and accuracy of emotion classification. Finally, based on the feature vector, the preset emotion classification model can accurately and stably output the emotion classification result, and maintain a high recognition performance even in a noisy environment or in a scene with complex and changeable emotional expression. In summary, this technical solution realizes the automatic and efficient extraction of emotional features in the speech signal of the original audio data by adopting the self-supervised learning method, thereby improving the accuracy of audio emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0034] Figure 2 It is a flowchart of an audio emotion recognition method provided by this application;

[0035] Figure 3 It is a structural schematic diagram of an audio emotion recognition device provided by the present application;

[0036] Figure 4 It is a structural schematic diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0038] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0039] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0040] like Figure 1As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0041] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0042] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0043] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0044] It should be noted that the audio emotion recognition method provided in the embodiment of the present application is generally executed by a server, and accordingly, the audio emotion recognition device is generally set in the server.

[0045] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0046] Continue to refer Figure 2 , shows a flow chart of an embodiment of an audio emotion recognition method according to the present application. The audio emotion recognition method comprises the following steps:

[0047] Step S201, obtaining original audio data of the target object.

[0048] In this embodiment, the audio emotion recognition method is executed on an electronic device (eg Figure 1The server shown in the figure) can obtain the original audio data through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0049] The target object refers to the subject or individual that needs to be subjected to audio emotion recognition, and the audio data generated by it is the direct source of emotion recognition. The target object can be any entity that can make a sound, such as a person, an animal, or a mechanical device that simulates sound. For example, in an intelligent voice interaction system, the target object is usually a user, whose voice data is captured by the system and emotion recognition is performed.

[0050] Raw audio data refers to audio data directly obtained from the target object without any processing. This data may contain various noises and different sampling rates and needs to be processed later. For example, a user's voice recorded from a mobile phone microphone is raw audio data, which may contain background noise and different sampling rates.

[0051] Step S202: preprocess the original audio data to obtain preprocessed audio data.

[0052] Preprocessing refers to a series of processing operations on raw audio data, which aims to improve the data quality and make it more suitable for subsequent feature extraction and emotion recognition. Preprocessing can include sampling rate conversion, denoising, etc. For example, converting the sampling rate of raw audio data to 16kHz and using a denoising algorithm to remove background noise is the preprocessing process.

[0053] The audio data refers to the audio data that has been pre-processed, and its quality has been improved, making it more suitable for subsequent feature extraction and emotion recognition. For example, the user's voice data that has been processed by sampling rate conversion and denoising is the audio data that can be used for subsequent processing.

[0054] Step S203: Use the pre-trained self-supervisory model to extract features from the audio data to obtain audio feature information of the audio data.

[0055] The pre-trained self-supervised model refers to a model trained by self-supervised learning on a large amount of unlabeled data, which can capture the intrinsic structure and characteristics of audio data. It can be used to predict masked audio segments in audio data. For example, a pre-trained self-supervised model trained with a large amount of unlabeled audio data can predict missing or masked parts in audio data.

[0056] The audio feature information refers to information extracted from the audio data that can characterize the audio characteristics. Such feature information may include spectrum features, rhythm features, etc. For example, features such as Mel frequency cepstral coefficients and pitch extracted from the audio data are audio feature information.

[0057] Step S204: perform dimensionality reduction processing on the audio feature information to obtain an emotional feature vector of the audio data.

[0058] Among them, the emotional feature vector refers to the vector obtained by reducing the dimension of the audio feature information and can represent the emotional characteristics of the audio data. The emotional feature vector is an important basis for emotional classification. For example, the vector obtained by mapping the mean and standard deviation of the audio feature information to a low-dimensional space is the emotional feature vector, which can represent the emotional characteristics of the audio data.

[0059] Step S205: Based on the emotion feature vector, a preset emotion classification model is used to perform emotion classification on the audio data to obtain an emotion classification result.

[0060] The preset sentiment classification model refers to a model trained on a training data set and can be used for sentiment classification. The model can receive a sentiment feature vector as input and output a sentiment classification result. For example, a support vector machine model trained using audio data with sentiment labels is a preset sentiment classification model.

[0061] The emotion classification result refers to the result obtained after the emotion classification model classifies the audio data. It can include the probability values ​​of multiple emotion types and the final determined emotion type. For example, the probability values ​​of the two emotion types "happy" and "sad" output by the emotion classification model, and the final determined emotion type "happy" are the emotion classification results.

[0062] Step S206: Determine the emotion type represented by the audio data according to the emotion classification result.

[0063] Among them, emotion type refers to the emotion types that the emotion classification model can identify, such as happiness, sadness, anger, etc. Emotion type is an important part of the emotion classification result. For example, emotion types include "happiness", "sadness", "anger" and many other emotions.

[0064] The embodiment of the present application can significantly improve the accuracy and robustness of audio emotion recognition through a series of innovative steps. First, by acquiring and preprocessing the original audio data of the target object, a solid foundation is laid for subsequent processing. The pre-trained self-supervised model is used for feature extraction. This step abandons the complex manual feature extraction method with limited emotion capture ability in the traditional method, and instead uses the self-supervised learning method to automatically mine the deep emotional features in the speech signal. Not only does it simplify the feature extraction process, but it also greatly broadens the feature dimension, allowing the model to capture more delicate and comprehensive emotional information. At the same time, the self-supervised model is pre-trained on an unlabeled large-scale data set, which gives it a strong generalization ability and effectively alleviates the problem of deep learning models relying on a large amount of labeled data. Further, by performing dimensionality reduction processing on the extracted audio feature information, a refined emotional feature vector is obtained, which helps to improve the efficiency and accuracy of emotion classification. Finally, based on the feature vector, the preset emotion classification model can accurately and stably output the emotion classification result, and maintain a high recognition performance even in a noisy environment or in a scene with complex and changeable emotional expression. In summary, this technical solution realizes the automatic and efficient extraction of emotional features in the speech signal of the original audio data by adopting the self-supervised learning method, thereby improving the accuracy of audio emotion recognition.

[0065] In some optional implementations of this embodiment, step 202, preprocessing the original audio data to obtain preprocessed audio data, specifically includes the following steps:

[0066] The original audio data is sampled and converted to a uniform sampling rate to obtain audio data with a target sampling rate. The audio data is denoised using a preset denoising algorithm to obtain preprocessed audio data.

[0067] The sampling rate refers to the number of times the audio data is sampled per second, which determines the resolution and quality of the audio data. Different sampling rates may result in differences in audio data. For example, common sampling rates include 44.1kHz, 16kHz, etc. The higher the sampling rate, the higher the resolution and quality of the audio data.

[0068] The target sampling rate refers to converting different sampling rates of the original audio data to the same sampling rate during the preprocessing process. The uniform sampling rate can ensure the consistency and accuracy of the subsequent audio data processing. For example, the target sampling rate is set to 16kHz to ensure that all audio data has the same sampling rate.

[0069] In one example, the implementation process of this embodiment is described in detail by taking the financial customer service hotline scenario as an example. In a certain bank customer service hotline system, the original audio data is a customer's call audio, and the sampling rate ranges from 8kHz to 48kHz. First, the call audio can be sampled at a rate conversion to convert the sampling rate of the call audio to a uniform 16kHz. Then, a denoising algorithm based on spectrum analysis can be used to denoise the call audio after the sampling rate conversion to remove background noise, echo and other interference factors. The signal-to-noise ratio of the pre-processed call audio is significantly improved, providing high-quality input for subsequent speech recognition and sentiment analysis.

[0070] The embodiment of the present application can ensure that audio data with different sampling rates can be unified to the target sampling rate by converting the sampling rate, thereby eliminating audio distortion or incompatibility problems caused by sampling rate differences, and providing a stable data basis for subsequent audio processing and analysis. The use of a denoising algorithm to denoise the audio data effectively reduces the impact of factors such as background noise and environmental interference on the audio quality, and improves the clarity and recognizability of the audio data. This series of processing steps is crucial to improving the accuracy and efficiency of applications such as sentiment analysis.

[0071] In some optional implementations of this embodiment, in step S203, before extracting features from the audio data using a pre-trained self-supervisory model to obtain audio feature information of the audio data, the following steps are specifically included:

[0072] Acquire audio training data, preprocess the audio training data, and obtain preprocessed target audio data; use a preset self-supervised model to perform self-supervised learning training on the target audio data to obtain a pre-trained self-supervised model.

[0073] The audio training data is unlabeled audio data. It can be used to train the audio data of the preset self-supervised model. During the training process, the self-supervised model can capture the inherent structure and characteristics of the audio training data and predict the masked audio segments in the audio training data.

[0074] The target audio data refers to the audio data that has been pre-processed and used for self-supervised learning training. For example, the audio training data that has been processed by sampling rate conversion and denoising can be used as the target audio data for training the preset self-supervised model.

[0075] Among them, the preset self-supervised model is used as a model for self-supervised learning. The model can capture the intrinsic structure of the target audio data and improve the ability of feature extraction by predicting the masked audio segments in the target audio data. The goal of self-supervised learning is to predict the masked audio segments in the target audio data. For example, a self-supervised model trained with a large amount of unlabeled audio training data can predict the missing or masked parts of the target audio data, thereby improving the accuracy of feature extraction.

[0076] An audio segment refers to a part of the target audio data, which may be a complete sentence, a word, or a syllable. In self-supervised learning, audio segments are usually masked or replaced so that the model can learn the intrinsic structure of the target audio data. For example, in a user's voice, a complete sentence or a word is an audio segment.

[0077] In one example, in the financial field, a large number of telephone recordings containing conversations between customers and customer service personnel, transaction confirmations, etc. can be selected as audio training data. These data cover various transaction scenarios and contexts, providing a rich source of information for subsequent self-supervised learning. The acquired audio training data is preprocessed, including removing silent segments, noise reduction, volume standardization, etc., to obtain preprocessed target audio data. This step ensures the clarity and consistency of the audio data, providing a high-quality data foundation for subsequent learning and training. A preset self-supervised model (such as Wave2Vec, CPC, etc.) is used to perform self-supervised learning training on the preprocessed target audio data. By simulating the contextual relationship of audio data, the model can automatically learn the intrinsic characteristics of audio data, thereby obtaining a pre-trained self-supervised model. Subsequently, the audio data to be processed, such as a new telephone recording, can be input into the pre-trained self-supervised model, and the model can output audio feature information of the audio data, such as voice content, emotional tendency, speech speed, etc.

[0078] The embodiment of the present application can ensure the quality and consistency of the data by acquiring and preprocessing the audio training data, laying a solid foundation for subsequent learning and training. The preset self-supervised model is used for self-supervised learning and training, which makes full use of the contextual relationship within the audio data and can automatically learn the intrinsic characteristics of the audio data without manual annotation, thereby greatly improving the learning efficiency and generalization ability of the model. The complex process of audio feature extraction is simplified, the labor cost is reduced, and the accuracy and efficiency of audio data processing are significantly improved. In addition, because the self-supervised model can adaptively capture subtle changes in audio data, this solution can more accurately identify and analyze key information in audio data.

[0079] Step S203, using a pre-trained self-supervisory model to extract features from the audio data to obtain audio feature information of the audio data, specifically includes the following steps:

[0080] Input the audio data into the pre-trained self-supervised model and output the audio feature information of the audio data.

[0081] In one example, in the financial field, a large number of telephone recordings containing conversations between customers and customer service personnel, transaction confirmations, etc. can be selected as audio training data, and the preset self-supervised model can be trained for self-supervised learning to obtain a pre-trained self-supervised model. Subsequently, the audio data to be processed, such as new telephone recordings, can be input into the pre-trained self-supervised model, and the pre-trained self-supervised model can output audio feature information of the audio data, such as voice content, emotional tendency, speech speed, etc.

[0082] The embodiment of the present application can achieve accurate extraction of audio feature information by inputting audio data into a pre-trained self-supervised model. The pre-trained self-supervised model, with its powerful feature learning ability, can deeply explore the implicit features in the audio data, such as speech rhythm, pitch changes, emotional tendencies, etc. These features are crucial for understanding the audio content. This solution can not only improve the accuracy and efficiency of audio feature extraction, but also make the extracted feature information richer and more comprehensive.

[0083] In some optional implementations of this embodiment, step S204, performing dimensionality reduction processing on the audio feature information to obtain an emotional feature vector of the audio data, specifically includes the following steps:

[0084] Calculate the statistics of the audio feature information, including the mean and standard deviation; map the audio feature information to a low-dimensional space based on the mean and standard deviation to obtain the emotional feature vector of the audio data.

[0085] Among them, statistics are used to describe the numerical value of the overall characteristics of audio feature information, such as mean, standard deviation, etc. Statistics can reflect the distribution and change trend of audio feature information.

[0086] The mean refers to the average of all values ​​of the audio feature information, and is used to describe the average level of the audio feature information. For example, in the audio feature information of a user's voice, the average of all values ​​is the mean, which can reflect the overall level of the audio feature information.

[0087] The standard deviation refers to the square root of the sum of the squares of the deviations between each value of the audio feature information and the mean, and is used to describe the degree of dispersion of the audio feature information.

[0088] In one example, the statistics of the audio feature information are calculated, including the mean and standard deviation. The mean reflects the overall level of the audio feature information, while the standard deviation reflects the degree of fluctuation of the audio feature information. These two statistics together constitute the basic description of the audio feature information. Next, based on the calculated mean and standard deviation, an appropriate mapping method such as principal component analysis, linear discriminant analysis, etc. is used to map the audio feature information to a low-dimensional space. In this process, by retaining key information and removing redundant information, effective dimensionality reduction of the audio feature information is achieved. Finally, the emotional feature vector of the audio data is obtained, which can accurately reflect the emotional information in the audio data.

[0089] The embodiment of the present application can achieve a quantitative description of the emotional characteristics of audio data by calculating statistics such as the mean and standard deviation of audio feature information. This step not only simplifies the complex process of audio emotion analysis, but also improves the accuracy and stability of emotion feature extraction. Subsequently, based on the calculated statistics, the audio feature information is mapped to a low-dimensional space to obtain the emotional feature vector of the audio data. This mapping process not only retains the key emotional information in the audio data, but also achieves effective dimensionality reduction of the data, thereby improving the efficiency of subsequent emotion analysis.

[0090] In some optional implementations of this embodiment, in step S205, based on the emotion feature vector, a preset emotion classification model is used to perform emotion classification on the audio data, and before obtaining the emotion classification result, the following steps are specifically included:

[0091] An audio training data set is obtained, the audio training data set includes multiple audio samples, each audio sample has a corresponding emotion label; a pre-trained audio processing model is used to extract audio features of the audio sample to obtain a first audio feature vector of the audio sample, and a preset principal component analysis method is used to perform dimensionality reduction processing on the audio feature vector to obtain a first audio feature vector after dimensionality reduction; a preset support vector machine model is trained according to the first audio feature vector and the emotion label to obtain an initial emotion classification model; a preset data enhancement technology is used to perform data enhancement processing on the audio training data set to obtain an enhanced target audio data set, and the target audio data set includes multiple audio enhancement samples; an audio processing model is used to extract audio features of multiple audio enhancement samples to obtain a second audio feature vector of multiple audio enhancement samples, and a principal component analysis method is used to perform dimensionality reduction processing on the second audio feature vector to obtain a second audio feature vector after dimensionality reduction; the initial emotion classification model is fine-tuned according to the second audio feature vector after dimensionality reduction and the emotion label to obtain a preset emotion classification model.

[0092] The audio training dataset is a collection of a large number of audio samples with emotion labels, which are used to train the support vector machine model to obtain the emotion classification model. These audio samples cover different emotional expressions and are the basis for the model to learn emotional features. For example, a dataset containing 1,000 audio samples, each of which is labeled with the corresponding emotion (such as happiness, sadness, anger, etc.), is used to train the support vector machine model to recognize these emotions.

[0093] The multiple audio samples refer to audio data obtained from multiple different sources or in different situations, and each sample represents an independent emotional expression. These samples are used to train and optimize the emotion classification model.

[0094] Among them, the emotion label refers to the text or numerical identifier associated with the audio sample, which is used to indicate the type of emotion expressed by the audio sample. Emotion labels are important supervisory information when training emotion classification models. For example, in an audio training dataset, each audio sample may have a corresponding emotion label, such as "happy", "sad", "angry", etc. These labels are used to guide the model to learn emotional features.

[0095] Among them, the audio processing model is a machine learning model that can process audio data and extract useful information. In the sentiment classification task, the audio processing model is used to extract audio features of audio samples, such as spectral features, rhythm features, etc. For example, a pre-trained convolutional neural network model can be used to extract spectral features and rhythm features from audio samples, which are then used for sentiment classification.

[0096] Among them, the principal component analysis method is a data dimensionality reduction technology that reduces the dimension of the data by retaining the most important features in the data set. In the sentiment classification task, it is used to reduce the dimension of audio features and improve the computational efficiency of the model.

[0097] Data augmentation technology is a technology that generates more training samples by transforming the original data or adding noise. In sentiment classification tasks, data augmentation technology is used to increase the diversity and richness of audio training data sets and improve the generalization ability of the model. For example, performing operations such as time stretching, frequency transformation, or adding background noise to audio samples to generate more training samples is the application of data augmentation technology in sentiment classification tasks.

[0098] The target audio dataset is an audio dataset obtained after data enhancement technology processing and contains more diverse and rich training samples. It is used to fine-tune or further optimize the sentiment classification model.

[0099] In one example, audio data in the financial field can be collected and sorted to construct an audio training data set. The data set includes multiple audio samples, each of which has a corresponding emotional label, such as "satisfied", "dissatisfied", "neutral", etc. These emotional labels are obtained by manual or automatic labeling, providing a reliable data basis for subsequent model training. The pre-trained audio processing model is used to extract the audio features of the audio sample to obtain a first audio feature vector. Then, the first audio feature vector is subjected to dimensionality reduction processing by a preset principal component analysis method to remove redundant information and obtain the first audio feature vector after dimensionality reduction. According to the first audio feature vector after dimensionality reduction and the emotional label, a preset support vector machine model is trained to obtain an initial emotional classification model. The model can preliminarily identify the emotional information in the audio data, but there may be certain errors and deviations. In order to improve the accuracy and generalization ability of the model, the preset data enhancement technology is used to perform data enhancement processing on the audio training data set to obtain an enhanced target audio data set. Then, the same audio processing model and principal component analysis method are used to extract the audio features of the audio enhancement sample, and the dimensionality reduction processing is performed to obtain a second audio feature vector after dimensionality reduction. Finally, the initial emotion classification model is fine-tuned according to the second audio feature vector and emotion label after dimensionality reduction to obtain the preset emotion classification model.

[0100] The embodiment of the present application can accurately capture the emotional information of audio data by constructing an audio training data set containing emotional tags and extracting audio features using a pre-trained audio processing model. The audio feature vector is reduced in dimension by the principal component analysis method, which effectively reduces the data dimension and reduces the computational complexity while retaining key emotional information. The support vector machine model is trained using the reduced-dimensional audio feature vector to obtain an initial emotion classification model that can preliminarily identify the emotional tendency in the audio. Furthermore, the audio training data set is expanded through data enhancement technology, and the audio features of the enhanced samples are extracted for dimensionality reduction, and the model is fine-tuned, which significantly improves the accuracy and generalization ability of the emotion classification model.

[0101] In some optional implementations of this embodiment, step S205, based on the emotion feature vector, using a preset emotion classification model to perform emotion classification on the audio data to obtain an emotion classification result, may specifically include the following steps:

[0102] The emotion feature vector is input into the preset emotion classification model to obtain the probability values ​​of multiple emotion types of the audio data; the emotion type with a probability value greater than a preset probability threshold among the multiple emotion types is determined as the preliminary target emotion type of the audio data; based on the probability value, the emotion classification confidence of the target emotion type is calculated; if the emotion classification confidence is greater than the preset confidence threshold, the target emotion type is determined as the final emotion type of the audio data to obtain the emotion classification result.

[0103] The probability values ​​of multiple emotion types refer to the probability of each emotion type given by the emotion classification model when the emotion classification model classifies the audio sample. These probability values ​​are used to determine the final emotion type of the audio sample. For example, the emotion classification model gives the probability values ​​of the audio sample belonging to the emotion types such as "happy", "sad", and "angry", such as 0.7 (happy), 0.2 (sad), and 0.1 (angry).

[0104] The preset probability threshold is a threshold used to determine the final emotion type of the audio sample. When the probability value of a certain emotion type given by the emotion classification model exceeds the threshold, the emotion type is considered to be the preliminary target emotion type of the audio sample. For example, in the emotion classification task, a preset probability threshold may be set to 0.5, that is, when the probability value of a certain emotion type exceeds 0.5, the emotion type is considered to be the preliminary target emotion type of the audio sample.

[0105] The preliminary target emotion type is the most likely emotion type determined based on the probability values ​​of multiple emotion types given by the emotion classification model and the preset probability threshold. It needs further verification or confirmation before it can be used as the final emotion type. For example, in the emotion classification task, when the emotion classification model gives a probability value of 0.8 (exceeding the preset probability threshold of 0.5) that the audio sample belongs to the "happy" emotion type, "happy" is considered to be the preliminary target emotion type of the audio sample.

[0106] In one example, an emotional feature vector that accurately reflects the emotional information in the audio data can be input into a preset emotional classification model, which is built based on a machine learning or deep learning algorithm and can output probability values ​​of multiple emotional types corresponding to the audio data. Then, the emotional type with a probability value greater than a preset probability threshold is determined as the preliminary target emotional type of the audio data. This step realizes the preliminary screening and identification of the emotional type of the audio data. Based on the probability value of the preliminarily determined target emotional type, the emotional classification confidence is calculated. The emotional classification confidence reflects the relative credibility of the target emotional type among multiple emotional types and is an important indicator for evaluating the accuracy of the emotional classification results. If the emotional classification confidence is greater than the preset confidence threshold, the target emotional type is determined as the final emotional type of the audio data, and the emotional classification result is obtained. This step further improves the accuracy and reliability of emotional classification.

[0107] The embodiment of the present application can process the emotion feature vector through the emotion classification model, and can accurately calculate the probability value of audio data corresponding to multiple emotion types, thereby realizing the quantitative evaluation of emotion information. By comparing the probability value with the preset probability threshold, the preliminary target emotion type is effectively screened out, thereby improving the accuracy of emotion recognition. Furthermore, the emotion classification confidence is calculated based on the probability value, providing a reliable evaluation index for the emotion classification result. When the emotion classification confidence exceeds the preset threshold, the final emotion type is determined, thereby ensuring the stability and credibility of the emotion classification result.

[0108] In some optional implementations of this embodiment, after determining the emotion type represented by the audio data according to the emotion classification result in step S206, the following steps may be specifically included:

[0109] Obtain the historical emotion type corresponding to the historical audio data of the target object, where the historical audio data and the original audio data are continuous audio data; generate an emotion trend curve representing the emotion change trend of the target object based on the historical emotion type and the target emotion type; generate communication strategy recommendations based on the emotion trend curve.

[0110] Among them, historical audio data refers to the audio data generated by the target object in the past and continuous with the current audio data. It is used to analyze the emotional change trend of the target object. For example, in an intelligent voice interaction system, the user's voice data over a period of time is historical audio data, which can be used to analyze the user's emotional change trend.

[0111] Among them, the historical emotion type refers to the emotion type generated by the target object in the past according to the historical audio data. It is used to compare with the current emotion type to analyze the emotion change trend of the target object.

[0112] Among them, the emotional trend curve is a graphical representation of the target object's emotional change trend generated based on the target object's historical emotional type and current emotional type. It helps to analyze the target object's emotional state changes. For example, in an intelligent voice interaction system, a line graph or curve graph generated based on the user's historical emotional type and current emotional type is an emotional trend curve, which can be used to analyze the user's emotional state change trend.

[0113] Among them, the communication strategy suggestion is a suggestion or strategy generated based on the target object's emotional trend curve to guide effective communication with the target object. It helps to improve the communication effect and the satisfaction of the target object. For example, in an intelligent voice interaction system, the communication strategy suggestion generated based on the user's emotional trend curve may include adjusting the tone, choosing the appropriate communication time, or providing personalized services, etc., to improve the communication effect with the user.

[0114] In one example, historical audio data of a target object (such as a financial customer) can be obtained, and these data are continuous audio data with the original audio data. By performing sentiment analysis on the historical audio data, the corresponding historical sentiment types, such as satisfaction, neutrality, dissatisfaction, etc., are obtained. Then, based on the historical sentiment types and the target sentiment types obtained by the current analysis (also obtained by sentiment analysis of the original audio data), an emotional trend curve is generated to characterize the trend of emotional changes of the target object. This curve uses time as the axis and shows the changes in customer emotions from the past to the present, which helps financial institutions to fully understand the emotional state of customers. Finally, based on the emotional trend curve, personalized communication strategy recommendations are generated. For example, when the emotional trend curve shows that the customer's emotion changes from satisfaction to dissatisfaction, it is recommended to take timely measures to understand the reasons for the customer's dissatisfaction and provide corresponding solutions.

[0115] The embodiment of the present application can ensure the consistency and accuracy of emotion recognition by accurately obtaining the historical emotion types corresponding to the historical audio data of the target object, and combining it with the continuous emotion analysis formed by the current original audio data. The generation of the emotion trend curve not only intuitively shows the changing trend of the target object's emotions from the past to the present, but also indicates the possible emotional direction, providing a scientific basis for understanding its inner world. Based on this refined emotion analysis, the proposal of communication strategy suggestions becomes more personalized and efficient, and the communication method and content can be adjusted in time to adapt to the current and expected emotional state of the target object, effectively promoting the emotional communication between the two parties.

[0116] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned original audio data, the above-mentioned original audio data can also be stored in a blockchain node.

[0117] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0118] The embodiments of the present application can build and optimize related models and networks based on artificial intelligence technology, such as pre-trained self-supervised models, sentiment classification models, etc. Among them, artificial intelligence (AI) models are the crystallization of theory and practice that simulates the decision-making process of human intelligence through algorithms and data analysis to solve complex problems, predict future trends, or realize automated tasks. These models use a large amount of historical data and real-time information, and are trained and optimized through a specific algorithm framework to achieve efficient, accurate and reliable performance.

[0119] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0120] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0121] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of an audio emotion recognition device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0122] like Figure 3 As shown, the audio emotion recognition device 400 of this embodiment includes: an acquisition module 401, a preprocessing module 402, a feature extraction module 403, a dimension reduction module 404, a classification module 405 and a type determination module 406. Among them:

[0123] An acquisition module 401 is used to acquire original audio data of a target object;

[0124] The preprocessing module 402 is used to preprocess the original audio data to obtain preprocessed audio data;

[0125] A feature extraction module 403 is used to extract features from the audio data using a pre-trained self-supervisory model to obtain audio feature information of the audio data;

[0126] A dimension reduction module 404 is used to perform dimension reduction processing on the audio feature information to obtain an emotional feature vector of the audio data;

[0127] The classification module 405 is used to perform emotion classification on the audio data based on the emotion feature vector and adopt a preset emotion classification model to obtain an emotion classification result;

[0128] The type determination module 406 is used to determine the emotion type represented by the audio data according to the emotion classification result.

[0129] The embodiment of the present application can significantly improve the accuracy and robustness of audio emotion recognition through a series of innovative steps. First, by acquiring and preprocessing the original audio data of the target object, a solid foundation is laid for subsequent processing. The pre-trained self-supervised model is used for feature extraction. This step abandons the complex manual feature extraction method with limited emotion capture ability in the traditional method, and instead uses the self-supervised learning method to automatically mine the deep emotional features in the speech signal. Not only does it simplify the feature extraction process, but it also greatly broadens the feature dimension, allowing the model to capture more delicate and comprehensive emotional information. At the same time, the self-supervised model is pre-trained on an unlabeled large-scale data set, which gives it a strong generalization ability and effectively alleviates the problem of deep learning models relying on a large amount of labeled data. Further, by performing dimensionality reduction processing on the extracted audio feature information, a refined emotional feature vector is obtained, which helps to improve the efficiency and accuracy of emotion classification. Finally, based on the feature vector, the preset emotion classification model can accurately and stably output the emotion classification result, and maintain a high recognition performance even in a noisy environment or in a scene with complex and changeable emotional expression. In summary, this technical solution realizes the automatic and efficient extraction of emotional features in the speech signal of the original audio data by adopting the self-supervised learning method, thereby improving the accuracy of audio emotion recognition.

[0130] In one embodiment, the pre-processing module 402 includes:

[0131] The conversion submodule is used to perform sampling rate conversion on the original audio data, convert different sampling rates of the original audio data into a unified sampling rate, and obtain audio data with a sampling rate of a target sampling rate;

[0132] The denoising submodule is used to perform denoising processing on the audio data using a preset denoising algorithm to obtain pre-processed audio data.

[0133] The embodiment of the present application can ensure that audio data with different sampling rates can be unified to the target sampling rate by converting the sampling rate, thereby eliminating audio distortion or incompatibility problems caused by sampling rate differences, and providing a stable data basis for subsequent audio processing and analysis. The use of a denoising algorithm to denoise the audio data effectively reduces the impact of factors such as background noise and environmental interference on the audio quality, and improves the clarity and recognizability of the audio data. This series of processing steps is crucial to improving the accuracy and efficiency of applications such as sentiment analysis.

[0134] In one embodiment, the feature extraction module 403 includes:

[0135] The first input submodule is used to input audio data into a pre-trained self-supervised model and output audio feature information of the audio data.

[0136] The embodiment of the present application can achieve accurate extraction of audio feature information by inputting audio data into a pre-trained self-supervised model. The pre-trained self-supervised model, with its powerful feature learning ability, can deeply explore the implicit features in the audio data, such as speech rhythm, pitch changes, emotional tendencies, etc. These features are crucial for understanding the audio content. This solution can not only improve the accuracy and efficiency of audio feature extraction, but also make the extracted feature information richer and more comprehensive.

[0137] In one embodiment, the dimension reduction module 404 includes:

[0138] A first calculation submodule, used to calculate the statistics of the audio feature information, the statistics including the mean and the standard deviation;

[0139] The mapping submodule is used to map the audio feature information to a low-dimensional space according to the mean and standard deviation to obtain the emotional feature vector of the audio data.

[0140] The embodiment of the present application can achieve a quantitative description of the emotional characteristics of audio data by calculating statistics such as the mean and standard deviation of audio feature information. This step not only simplifies the complex process of audio emotion analysis, but also improves the accuracy and stability of emotion feature extraction. Subsequently, based on the calculated statistics, the audio feature information is mapped to a low-dimensional space to obtain the emotional feature vector of the audio data. This mapping process not only retains the key emotional information in the audio data, but also achieves effective dimensionality reduction of the data, thereby improving the efficiency of subsequent emotion analysis.

[0141] In one embodiment, the classification module 405 includes:

[0142] The second input submodule is used to input the emotion feature vector into a preset emotion classification model to obtain probability values ​​of multiple emotion types of the audio data;

[0143] A first determination submodule is used to determine an emotion type whose probability value is greater than a preset probability threshold among multiple emotion types as a preliminary target emotion type of the audio data;

[0144] The second calculation submodule is used to calculate the sentiment classification confidence of the target sentiment type based on the probability value;

[0145] The second determination submodule is used to determine the target emotion type as the final emotion type of the audio data if the emotion classification confidence is greater than a preset confidence threshold, so as to obtain the emotion classification result.

[0146] The embodiment of the present application can process the emotion feature vector through the emotion classification model, and can accurately calculate the probability value of audio data corresponding to multiple emotion types, thereby realizing the quantitative evaluation of emotion information. By comparing the probability value with the preset probability threshold, the preliminary target emotion type is effectively screened out, thereby improving the accuracy of emotion recognition. Furthermore, the emotion classification confidence is calculated based on the probability value, providing a reliable evaluation index for the emotion classification result. When the emotion classification confidence exceeds the preset threshold, the final emotion type is determined, thereby ensuring the stability and credibility of the emotion classification result.

[0147] In one embodiment, the audio emotion recognition device 400 further includes:

[0148] A first data acquisition module is used to acquire audio training data, preprocess the audio training data, and obtain preprocessed target audio data;

[0149] The first training module is used to use a preset self-supervised model to perform self-supervised learning training on the target audio data to obtain a pre-trained self-supervised model.

[0150] The embodiment of the present application can ensure the quality and consistency of the data by acquiring and preprocessing the audio training data, laying a solid foundation for subsequent learning and training. The preset self-supervised model is used for self-supervised learning and training, which makes full use of the contextual relationship within the audio data and can automatically learn the intrinsic characteristics of the audio data without manual annotation, thereby greatly improving the learning efficiency and generalization ability of the model. The complex process of audio feature extraction is simplified, the labor cost is reduced, and the accuracy and efficiency of audio data processing are significantly improved. In addition, because the self-supervised model can adaptively capture subtle changes in audio data, this solution can more accurately identify and analyze key information in audio data.

[0151] In one embodiment, the audio emotion recognition device 400 may further include:

[0152] A second data acquisition module is used to acquire an audio training data set, where the audio training data set includes multiple audio samples, each audio sample has a corresponding emotion label;

[0153] The second module is used to extract the audio features of the audio sample by using the pre-trained audio processing model to obtain the first audio feature vector of the audio sample, and perform dimensionality reduction processing on the audio feature vector by using a preset principal component analysis method to obtain the first audio feature vector after dimensionality reduction;

[0154] A second training module is used to train a preset support vector machine model according to the first audio feature vector and the emotion label to obtain an initial emotion classification model;

[0155] An enhancement module is used to perform data enhancement processing on the audio training data set using a preset data enhancement technology to obtain an enhanced target audio data set, where the target audio data set includes multiple audio enhancement samples;

[0156] An extraction module is used to extract audio features of multiple audio enhancement samples by using an audio processing model to obtain second audio feature vectors of the multiple audio enhancement samples, and to perform dimensionality reduction processing on the second audio feature vector by using a principal component analysis method to obtain a second audio feature vector after dimensionality reduction;

[0157] The fine-tuning module is used to fine-tune the initial emotion classification model according to the second audio feature vector and the emotion label after dimensionality reduction to obtain a preset emotion classification model.

[0158] The embodiment of the present application can accurately capture the emotional information of audio data by constructing an audio training data set containing emotional tags and extracting audio features using a pre-trained audio processing model. The audio feature vector is reduced in dimension by the principal component analysis method, which effectively reduces the data dimension and reduces the computational complexity while retaining key emotional information. The support vector machine model is trained using the reduced-dimensional audio feature vector to obtain an initial emotion classification model that can preliminarily identify the emotional tendency in the audio. Furthermore, the audio training data set is expanded through data enhancement technology, and the audio features of the enhanced samples are extracted for dimensionality reduction, and the model is fine-tuned, which significantly improves the accuracy and generalization ability of the emotion classification model.

[0159] In one embodiment, the audio emotion recognition device 400 may further include:

[0160] A type acquisition module is used to obtain the historical emotion type corresponding to the historical audio data of the target object, where the historical audio data and the original audio data are continuous audio data;

[0161] A curve generation module, used to generate an emotion trend curve representing the emotion change trend of the target object based on the historical emotion type and the target emotion type;

[0162] The suggestion generation module is used to generate communication strategy suggestions based on the sentiment trend curve.

[0163] The embodiment of the present application can ensure the consistency and accuracy of emotion recognition by accurately obtaining the historical emotion types corresponding to the historical audio data of the target object, and combining it with the continuous emotion analysis formed by the current original audio data. The generation of the emotion trend curve not only intuitively shows the changing trend of the target object's emotions from the past to the present, but also indicates the possible emotional direction, providing a scientific basis for understanding its inner world. Based on this refined emotion analysis, the proposal of communication strategy suggestions becomes more personalized and efficient, and the communication method and content can be adjusted in time to adapt to the current and expected emotional state of the target object, effectively promoting the emotional communication between the two parties.

[0164] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0165] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 6 having a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0166] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0167] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions of the audio emotion recognition method, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0168] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run computer-readable instructions stored in the memory 61 or process data, such as computer-readable instructions for running the audio emotion recognition method.

[0169] The network interface 63 may include a wireless network interface or a wired network interface, and the network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0170] The embodiment of the present application can significantly improve the accuracy and robustness of audio emotion recognition through a series of innovative steps. First, by acquiring and preprocessing the original audio data of the target object, a solid foundation is laid for subsequent processing. The pre-trained self-supervised model is used for feature extraction. This step abandons the complex manual feature extraction method with limited emotion capture ability in the traditional method, and instead uses the self-supervised learning method to automatically mine the deep emotional features in the speech signal. Not only does it simplify the feature extraction process, but it also greatly broadens the feature dimension, allowing the model to capture more delicate and comprehensive emotional information. At the same time, the self-supervised model is pre-trained on an unlabeled large-scale data set, which gives it a strong generalization ability and effectively alleviates the problem of deep learning models relying on a large amount of labeled data. Further, by performing dimensionality reduction processing on the extracted audio feature information, a refined emotional feature vector is obtained, which helps to improve the efficiency and accuracy of emotion classification. Finally, based on the feature vector, the preset emotion classification model can accurately and stably output the emotion classification result, and maintain a high recognition performance even in a noisy environment or in a scene with complex and changeable emotional expression. In summary, this technical solution realizes the automatic and efficient extraction of emotional features in the speech signal of the original audio data by adopting the self-supervised learning method, thereby improving the accuracy of audio emotion recognition.

[0171] The present application also provides another implementation, namely, providing a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the audio emotion recognition method as described above.

[0172] The embodiment of the present application can significantly improve the accuracy and robustness of audio emotion recognition through a series of innovative steps. First, by acquiring and preprocessing the original audio data of the target object, a solid foundation is laid for subsequent processing. The pre-trained self-supervised model is used for feature extraction. This step abandons the complex manual feature extraction method with limited emotion capture ability in the traditional method, and instead uses the self-supervised learning method to automatically mine the deep emotional features in the speech signal. Not only does it simplify the feature extraction process, but it also greatly broadens the feature dimension, allowing the model to capture more delicate and comprehensive emotional information. At the same time, the self-supervised model is pre-trained on an unlabeled large-scale data set, which gives it a strong generalization ability and effectively alleviates the problem of deep learning models relying on a large amount of labeled data. Further, by performing dimensionality reduction processing on the extracted audio feature information, a refined emotional feature vector is obtained, which helps to improve the efficiency and accuracy of emotion classification. Finally, based on the feature vector, the preset emotion classification model can accurately and stably output the emotion classification result, and maintain a high recognition performance even in a noisy environment or in a scene with complex and changeable emotional expression. In summary, this technical solution realizes the automatic and efficient extraction of emotional features in the speech signal of the original audio data by adopting the self-supervised learning method, thereby improving the accuracy of audio emotion recognition.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0174] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. An audio emotion recognition method, characterized in that: The steps include: Get the original audio data of the target object; Preprocessing the original audio data to obtain preprocessed audio data; Using a pre-trained self-supervised model, extracting features from the audio data to obtain audio feature information of the audio data; Performing dimensionality reduction processing on the audio feature information to obtain an emotional feature vector of the audio data; Based on the emotion feature vector, a preset emotion classification model is used to perform emotion classification on the audio data to obtain an emotion classification result; The emotion type represented by the audio data is determined according to the emotion classification result.

2. The method according to claim 1, characterized in that The step of preprocessing the original audio data to obtain preprocessed audio data specifically includes: Performing sampling rate conversion on the original audio data, converting different sampling rates of the original audio data into a unified sampling rate, and obtaining audio data with a sampling rate of a target sampling rate; The audio data is denoised using a preset denoising algorithm to obtain pre-processed audio data.

3. The method according to claim 1, characterized in that Before the step of using the pre-trained self-supervisory model to extract features from the audio data to obtain audio feature information of the audio data, the method further includes: Acquire audio training data, and preprocess the audio training data to obtain preprocessed target audio data; Using a preset self-supervised model, performing self-supervised learning training on the target audio data to obtain a pre-trained self-supervised model; The step of using a pre-trained self-supervisory model to extract features from the audio data to obtain audio feature information of the audio data specifically includes: The audio data is input into the pre-trained self-supervised model, and audio feature information of the audio data is output.

4. The method according to claim 1, characterized in that: The step of performing dimensionality reduction processing on the audio feature information to obtain the emotional feature vector of the audio data specifically includes: Calculating statistics of the audio feature information, wherein the statistics include a mean and a standard deviation; The audio feature information is mapped to a low-dimensional space according to the mean and the standard deviation to obtain an emotional feature vector of the audio data.

5. The method according to claim 1, characterized in that Before the step of performing emotion classification on the audio data based on the emotion feature vector and using a preset emotion classification model to obtain an emotion classification result, the method further includes: Acquire an audio training data set, wherein the audio training data set includes a plurality of audio samples, each of the audio samples having a corresponding emotion label; Using a pre-trained audio processing model to extract audio features of the audio sample to obtain a first audio feature vector of the audio sample, and using a preset principal component analysis method to perform dimensionality reduction processing on the audio feature vector to obtain a first audio feature vector after dimensionality reduction; According to the first audio feature vector and the emotion label, a preset support vector machine model is trained to obtain an initial emotion classification model; Using a preset data enhancement technology, the audio training data set is subjected to data enhancement processing to obtain an enhanced target audio data set, wherein the target audio data set includes a plurality of audio enhancement samples; Using the audio processing model, extracting audio features of the multiple audio enhancement samples to obtain second audio feature vectors of the multiple audio enhancement samples, and using the principal component analysis method to perform dimensionality reduction processing on the second audio feature vector to obtain a second audio feature vector after dimensionality reduction; The initial emotion classification model is fine-tuned according to the second audio feature vector after dimensionality reduction and the emotion label to obtain a preset emotion classification model.

6. The method according to claim 1, characterized in that The step of performing emotion classification on the audio data based on the emotion feature vector and using a preset emotion classification model to obtain an emotion classification result specifically includes: Inputting the emotion feature vector into a preset emotion classification model to obtain probability values ​​of multiple emotion types of the audio data; Determine the emotion type whose probability value is greater than a preset probability threshold among the multiple emotion types as the preliminary target emotion type of the audio data; Based on the probability value, calculating the sentiment classification confidence of the target sentiment type; If the emotion classification confidence is greater than a preset confidence threshold, the target emotion type is determined as the final emotion type of the audio data to obtain an emotion classification result.

7. The method according to claim 6, characterized in that After the step of determining the emotion type represented by the audio data according to the emotion classification result, the method further includes: Acquire a historical emotion type corresponding to the historical audio data of the target object, wherein the historical audio data and the original audio data are continuous audio data; Based on the historical emotion type and the target emotion type, generating an emotion trend curve representing the emotion change trend of the target object; Based on the sentiment trend curve, a communication strategy suggestion is generated.

8. An audio emotion recognition device, characterized in that: include: An acquisition module, used to acquire the original audio data of the target object; A preprocessing module, used to preprocess the original audio data to obtain preprocessed audio data; A feature extraction module, used to extract features from the audio data using a pre-trained self-supervised model to obtain audio feature information of the audio data; A dimension reduction module, used for performing dimension reduction processing on the audio feature information to obtain an emotional feature vector of the audio data; A classification module, used to perform emotion classification on the audio data based on the emotion feature vector and using a preset emotion classification model to obtain an emotion classification result; The type determination module is used to determine the emotion type represented by the audio data according to the emotion classification result.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the audio emotion recognition method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the audio emotion recognition method according to any one of claims 1 to 7 are implemented.

Images