Speech emotion analysis method and system based on artificial intelligence

Abstract

The present invention provides a method and system for speech emotion analysis based on artificial intelligence, wherein the method comprises the following steps: in response to receiving an acquisition trigger instruction at any triggering moment, establishing an audio acquisition task, so as to acquire audio of a corresponding target user based on the audio acquisition task, and obtaining audio acquisition data; preprocessing the audio acquisition data to acquire audio optimization data; performing emotion analysis on the audio optimization data based on a preset emotion analysis strategy to acquire emotion information corresponding to the audio optimization data; determining the emotion state corresponding to the target user based on the emotion information, and determining whether it is necessary to perform emotion intervention on the target user based on the emotion state. The present invention at least improves customer satisfaction.

Description

Technical Field

[0001] The present invention relates to data processing technology, and in particular to a speech emotion analysis method and system based on artificial intelligence. Background Art

[0002] In modern customer service and mental health assessment, understanding the user's emotional state is crucial to providing personalized services and timely intervention. Currently, sentiment analysis mainly relies on text data, but ignores the emotional information in speech. With the development of artificial intelligence and deep learning technology, speech sentiment analysis has become possible, but the existing system still has many shortcomings in real-time and accuracy. Summary of the invention

[0003] Based on the above problems, the present invention is proposed to provide a speech emotion analysis method and system based on artificial intelligence that overcomes the above problems or at least partially solves the above problems.

[0004] According to one aspect of the present invention, a method for speech emotion analysis based on artificial intelligence is provided, comprising the following steps:

[0005] In response to receiving a collection trigger instruction at any triggering moment, establishing an audio collection task to collect audio of a corresponding target user based on the audio collection task to obtain audio collection data;

[0006] Preprocessing the audio acquisition data to obtain audio optimization data;

[0007] Performing sentiment analysis on the audio optimization data based on a preset sentiment analysis strategy to obtain sentiment information corresponding to the audio optimization data;

[0008] An emotional state corresponding to the target user is determined based on the emotional information, and whether emotional intervention on the target user is required is determined based on the emotional state.

[0009] Optionally, in the method according to the present invention, in response to receiving a collection trigger instruction at any triggering moment, establishing an audio collection task to collect audio of a corresponding target user based on the audio collection task to obtain audio collection data, including:

[0010] In response to receiving a collection trigger instruction at any triggering moment, an audio collection task is established, and a preset collection instruction audio corresponding to the audio collection task is retrieved for output;

[0011] In response to completing the output of the preset collection indication audio, triggering the preset audio collection plane to collect audio corresponding to the target user, and obtaining test audio data corresponding to the target user;

[0012] Performing a test score on the target user based on the test audio data, and determining an audio attribute corresponding to the target user based on the obtained audio test score;

[0013] In response to the audio attribute of the target user being a positive evaluation attribute, calling the preset collection determined audio corresponding to the audio collection task for output;

[0014] In response to completing the output of the preset collection indication audio, triggering the preset audio collection plane to perform audio collection corresponding to the target user, and obtaining audio collection data corresponding to the target user;

[0015] or,

[0016] In response to the audio attribute of the target user being a negative evaluation attribute, a current adjusted audio is created based on the test audio data to be output again as the updated preset acquisition indication audio.

[0017] Optionally, in the method according to the present invention, performing a test score on the target user based on the test audio data, and determining an audio attribute corresponding to the target user based on the obtained audio test score, comprises:

[0018] Determine an audio output angle between the target user and the preset audio collection plane based on the test audio data, and determine an offset score corresponding to the target user based on the audio output angle;

[0019] Determine an audio output decibel corresponding to the target user based on the test audio data, and determine a volume score corresponding to the target user based on the audio output decibel;

[0020] Performing a fusion calculation on the offset score and the volume score, and comparing the obtained audio test score corresponding to the target user with the reference audio score;

[0021] When the audio test score is less than the reference audio score, determining the output attribute of the target user as a negative evaluation attribute;

[0022] When the audio test score is greater than or equal to the reference audio score, the output attribute of the target user is determined as a positive evaluation attribute.

[0023] Optionally, in the method according to the present invention, in response to the audio attribute of the target user being a negative evaluation attribute, creating a current adjustment audio based on the test audio data to output again as an updated preset acquisition indication audio includes:

[0024] In response to the audio attribute of the target user being a negative evaluation attribute, acquiring an audio output angle and an audio output decibel corresponding to the target user;

[0025] Calculate the difference between the audio output angle and the audio output decibel and the retrieved reference audio angle and reference audio decibel, respectively, to obtain the audio angle difference and the audio decibel difference;

[0026] Retrieving an initial adjustment audio, wherein the initial adjustment audio includes a reference audio segment and an angle filling slot and a decibel filling slot interspersed in the reference audio segment;

[0027] Fill the angle filling slot and the decibel filling slot respectively based on the audio angle difference and the audio decibel difference to obtain the current adjusted audio;

[0028] The currently adjusted audio is used as the updated preset collection indication audio, and the preset collection indication audio is outputted corresponding to the reference audio decibel based on the preset audio collection surface.

[0029] Optionally, in the method according to the present invention, preprocessing the audio acquisition data to obtain audio optimization data includes:

[0030] Performing noise removal on the audio acquisition data based on a preset frequency domain filtering strategy;

[0031] Normalize the audio data after noise removal;

[0032] In response to completing the normalization processing of the audio collection data, feature extraction is performed on the audio collection data after the normalization processing to obtain audio optimization data corresponding to the audio collection data.

[0033] Optionally, in the method according to the present invention, removing noise from the audio acquisition data based on a preset frequency domain filtering strategy comprises:

[0034] Determine the noise type of the audio collection data based on voice activity detection to obtain noise characteristics corresponding to the audio collection data;

[0035] Performing signal preprocessing including data segmentation and signal windowing on the audio acquisition data to obtain audio acquisition data of corresponding time domain;

[0036] Performing frequency domain conversion on the audio acquisition data after signal preprocessing to obtain audio acquisition data corresponding to the frequency domain;

[0037] Performing filtering processing on the audio collection data corresponding to the frequency domain, and performing time domain conversion on the audio collection data after the filtering processing to obtain audio collection data corresponding to the time domain;

[0038] A performance evaluation is performed on the audio collection data corresponding to the time domain to obtain a perceptual quality corresponding to the audio collection data in the time domain, and in response to the perceptual quality satisfying a preset output condition, noise removal of the audio collection data is completed.

[0039] Optionally, in the method according to the present invention, normalizing the audio acquisition data after noise removal includes:

[0040] Performing frame processing on the audio acquisition data after noise removal, and performing window processing on the audio acquisition data after frame processing;

[0041] The mean and standard deviation of the audio collection data after windowing are calculated, and the mean and standard deviation of the audio collection data are substituted into the following formula to normalize the audio collection data:

[0042] ;

[0043] Among them, ( X ) is the audio data after windowing, ( \mu ) is the mean, ( \sigma ) is the standard deviation, and ( X_{\text{norm}} ) is the audio data after normalization.

[0044] Optionally, in the method according to the present invention, in response to completing the normalization processing of the audio collection data, performing feature extraction on the audio collection data after the normalization processing to obtain audio optimization data corresponding to the audio collection data includes:

[0045] In response to completing the normalization processing of the audio collection data, performing frame processing on the normalized audio collection data, and performing window processing on the frame-processed audio collection data;

[0046] Performing a fast Fourier transform on the audio acquisition data after the windowing process to convert the audio acquisition data corresponding to the time domain into audio acquisition data corresponding to the frequency domain;

[0047] Feature extraction is performed on the audio acquisition data corresponding to the frequency domain to obtain audio optimization data corresponding to the multi-dimensional feature matrix.

[0048] Optionally, in the method according to the present invention, performing sentiment analysis on the audio optimization data based on a preset sentiment analysis strategy to obtain sentiment information corresponding to the audio optimization data includes:

[0049] Build an initial sentiment analysis model based on recurrent neural networks and convolutional neural networks;

[0050] Acquire a training data set, wherein the training data set includes speech data each labeled with different emotion categories;

[0051] Training the initial sentiment analysis model based on the training data set to obtain a trained current sentiment analysis model;

[0052] Inputting the audio optimization data into the current sentiment analysis model to perform sentiment analysis, and obtaining confidences corresponding to different sentiment categories;

[0053] The emotional state with the highest corresponding confidence is determined as the emotional information corresponding to the audio optimization data.

[0054] According to another aspect of the present invention, there is provided a speech emotion analysis system based on artificial intelligence, comprising:

[0055] The audio collection module is configured to establish an audio collection task in response to receiving a collection trigger instruction at any triggering moment, so as to collect audio of a corresponding target user based on the audio collection task and obtain audio collection data;

[0056] A data optimization module is configured to pre-process the audio acquisition data to obtain audio optimization data;

[0057] A sentiment analysis module, configured to perform sentiment analysis on the audio optimization data based on a preset sentiment analysis strategy to obtain sentiment information corresponding to the audio optimization data;

[0058] The emotion intervention module is configured to determine the emotion state corresponding to the target user based on the emotion information, and determine whether emotion intervention is required for the target user based on the emotion state.

[0059] According to the scheme of the present invention, the present invention can obtain the audio collection data of the target user, and further perform subsequent preprocessing on the audio collection data, so as to perform emotional analysis on the target user according to the obtained audio optimization data, thereby determining the emotional state of the target user. At the same time, according to the acquired emotional state, it can also determine whether corresponding emotional intervention is needed to help the target user to be in a positive emotion, thereby improving the psychological correction effect on the target user; in addition, in terms of customer service, the present invention can monitor customer emotions in real time, optimize customer service communication strategies, and improve customer satisfaction; in terms of mental health, the present invention can monitor and analyze user emotional changes, and assist in mental health diagnosis and intervention; in the field of education, the present invention can optimize teaching methods and improve teaching effects by analyzing students' emotional states; in terms of smart homes, the present invention can improve the naturalness and affinity of human-computer interaction by understanding and responding to users' emotions; and in terms of market research, the present invention can improve products and services by analyzing consumers' emotional responses. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A flow chart of a speech emotion analysis method based on artificial intelligence according to an embodiment of the present invention is shown;

[0061] Figure 2 The application scenario diagram of this embodiment is shown;

[0062] Figure 3 A structural block diagram of a speech emotion analysis system based on artificial intelligence according to another embodiment of the present invention is shown. DETAILED DESCRIPTION

[0063] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0064] In modern customer service and mental health assessment, understanding the user's emotional state is crucial to providing personalized services and timely intervention. Currently, sentiment analysis mainly relies on text data, but ignores the emotional information in speech. With the development of artificial intelligence and deep learning technology, speech sentiment analysis has become possible, but the existing system still has many shortcomings in real-time and accuracy.

[0065] In order to solve the problems existing in the above-mentioned prior art, the inventor proposes the solution of the present invention. One embodiment of the present invention provides a speech emotion analysis method based on artificial intelligence, which can be executed in a computing device, wherein the computing device can be understood as a terminal with data processing function, such as a mobile phone or a computer.

[0066] like Figure 1 As shown, the speech emotion analysis method based on artificial intelligence proposed in this embodiment begins with step S102. In step S102, the following contents are included:

[0067] In response to receiving a collection trigger instruction at any triggering moment, an audio collection task is established to perform audio collection of a corresponding target user based on the audio collection task to obtain audio collection data.

[0068] For example, in this embodiment, the sentiment analysis for any target user is mainly performed based on the voice of the target user. Therefore, when the corresponding sentiment analysis is required, the voice of the target user needs to be collected accordingly. In order to ensure the accuracy of the voice collection, before the collection, the target user can send a corresponding collection trigger instruction to establish a corresponding audio collection task, so that audio collection can be further performed based on the audio collection task, thereby obtaining corresponding audio collection data.

[0069] It can be understood that, in this embodiment, the speech emotion analysis method of this embodiment can be sintered into the corresponding processing chip and integrated into the corresponding device unit. The device unit can be specifically a portable speaker, so as to facilitate use and storage; the above-mentioned sending of the corresponding collection trigger instruction can be specifically a voice instruction or a key instruction. When the collection trigger instruction is a voice instruction, the voice wake-up word to be responded to can be pre-set, so as to facilitate waking up the processing chip and performing corresponding voice collection; and when the collection trigger instruction is a key instruction, the processing chip can be woken up and corresponding voice collection can be performed by setting the corresponding physical button.

[0070] For example, Figure 2 The application scenario diagram of this embodiment is shown. Figure 2 It can be seen from the content that the audio output by the user is received by the device unit.

[0071] Further, in this embodiment, the above-mentioned “in response to receiving a collection trigger instruction at any triggering moment, establishing an audio collection task, and performing audio collection of a corresponding target user based on the audio collection task to obtain audio collection data” may also include the following steps:

[0072] In response to receiving a collection trigger instruction at any triggering moment, an audio collection task is established, and a preset collection instruction audio corresponding to the audio collection task is retrieved for output;

[0073] In response to completing the output of the preset collection indication audio, triggering the preset audio collection plane to collect audio corresponding to the target user, and obtaining test audio data corresponding to the target user;

[0074] Performing a test score on the target user based on the test audio data, and determining an audio attribute corresponding to the target user based on the obtained audio test score;

[0075] In response to the audio attribute of the target user being a positive evaluation attribute, calling the preset collection determined audio corresponding to the audio collection task for output;

[0076] In response to completing the output of the preset collection indication audio, triggering the preset audio collection plane to perform audio collection corresponding to the target user, and obtaining audio collection data corresponding to the target user;

[0077] or,

[0078] In response to the audio attribute of the target user being a negative evaluation attribute, a current adjusted audio is created based on the test audio data to be output again as the updated preset acquisition indication audio.

[0079] For example, in this embodiment, when performing the corresponding emotion analysis, the processing chip will continue to be in a working state, and when receiving the corresponding collection trigger instruction at any triggering moment, the processing chip will establish an audio collection task accordingly, which means that the corresponding emotion analysis work is ready to be performed; after completing the establishment of the audio collection task, the corresponding preset collection instruction audio can be retrieved based on the audio collection task for output, where the preset collection instruction audio can be understood as being used to instruct the target user to make a corresponding sound to determine whether the current environment or the user's current position is suitable for the corresponding emotion evaluation, so the preset collection instruction audio can include the audio content corresponding to "please make a test speech";

[0080] After the output of the preset collection indication audio is completed, the corresponding preset audio collection surface can start audio collection, thereby obtaining the test audio data corresponding to the target user. From the above content, it can be known that the processing chip can be integrated in the corresponding device unit, so the preset audio collection surface can be understood as the collection surface of the corresponding device unit;

[0081] After obtaining the test audio data corresponding to the target user, a corresponding test score can be performed based on the content. Here, the test audio data can be understood as the voice content formed by the target user's random speech, which is mainly used to test whether the acquired audio data is clear or accurate. It can be explained that since the propagation of sound is directional, when the target user is too far away from the preset audio collection surface, or too far away, it will affect the corresponding audio collection effect;

[0082] After completing the corresponding test scoring, the audio attributes corresponding to the target user can be determined according to the obtained audio test scores, wherein the audio attributes can include positive evaluation attributes and negative evaluation attributes, where the positive evaluation attributes can be understood as clear or accurate, and the negative evaluation attributes can be understood as unclear or inaccurate audio data;

[0083] When the corresponding audio attribute is a positive evaluation attribute, it indicates that the test audio data output by the target user has met the standard and formal audio collection can be started. At this time, the preset audio collection surface can be triggered to collect the audio of the corresponding target user to obtain audio collection data for emotional evaluation;

[0084] When the corresponding audio attribute is a negative evaluation attribute, it indicates that the corresponding audio quality of the test audio data output by the target user does not meet the standard. At this time, the current adjustment audio can be created based on the test audio data to be output again as the updated preset acquisition indication audio to help the target user perform another audio test.

[0085] Furthermore, in this embodiment, the above-mentioned “testing and scoring the target user based on the test audio data, and determining the audio attribute corresponding to the target user based on the obtained audio test score” may also include the following steps:

[0086] Determine an audio output angle between the target user and the preset audio collection plane based on the test audio data, and determine an offset score corresponding to the target user based on the audio output angle;

[0087] Determine an audio output decibel corresponding to the target user based on the test audio data, and determine a volume score corresponding to the target user based on the audio output decibel;

[0088] Performing a fusion calculation on the offset score and the volume score, and comparing the obtained audio test score corresponding to the target user with the reference audio score;

[0089] When the audio test score is less than the reference audio score, determining the output attribute of the target user as a negative evaluation attribute;

[0090] When the audio test score is greater than or equal to the reference audio score, the output attribute of the target user is determined as a positive evaluation attribute.

[0091] For example, in this embodiment, it can be explained that the influencing conditions of audio quality include audio distance and audio offset, wherein the audio distance affects the corresponding audio decibels, and the audio offset affects the stereoscopic sense of the audio. Based on this, when the audio attribute of the target user is a negative evaluation attribute, the audio output angle and audio output decibel corresponding to the target user can be obtained. Here, the corresponding audio output angle and audio output decibel can be obtained based on the test audio data obtained above;

[0092] After obtaining the corresponding audio output angle and audio output decibel, the offset and volume scores can be respectively performed based on the audio output angle and the audio output decibel, thereby obtaining the corresponding offset score and volume score, and further performing fusion calculation to obtain the volume score corresponding to the target user. Here, the offset score and volume score process can be scored based on the rule that as the offset increases, as the volume increases, the corresponding score increases;

[0093] After the audio test score is obtained, the output attribute corresponding to the target user is determined by calling the pre-stored reference audio score and comparing the audio test score with the reference audio score.

[0094] Furthermore, in this embodiment, the above-mentioned “in response to the audio attribute of the target user being a negative evaluation attribute, creating the current adjusted audio based on the test audio data to output it again as the updated preset acquisition indication audio” may also include the following steps:

[0095] In response to the audio attribute of the target user being a negative evaluation attribute, acquiring an audio output angle and an audio output decibel corresponding to the target user;

[0096] Calculate the difference between the audio output angle and the audio output decibel and the retrieved reference audio angle and reference audio decibel, respectively, to obtain the audio angle difference and the audio decibel difference;

[0097] Retrieving an initial adjustment audio, wherein the initial adjustment audio includes a reference audio segment and an angle filling slot and a decibel filling slot interspersed in the reference audio segment;

[0098] Fill the angle filling slot and the decibel filling slot respectively based on the audio angle difference and the audio decibel difference to obtain the current adjusted audio;

[0099] The currently adjusted audio is used as the updated preset collection indication audio, and the preset collection indication audio is outputted corresponding to the reference audio decibel based on the preset audio collection surface.

[0100] For example, in this embodiment, when the output attribute corresponding to the target user is a negative evaluation attribute, it indicates that the test audio data output by the target user does not meet the standard. At this time, it is necessary to give corresponding instructions to the target user to ensure that the subsequent sentiment analysis has a high accuracy, and the corresponding instruction method is as follows:

[0101] First, the obtained audio output angle and audio output decibel can be used to perform differences with the retrieved reference audio angle and reference audio decibel respectively, so as to obtain the corresponding audio angle difference and audio decibel difference. It should be noted that, under normal circumstances, when the target user is outputting audio to the preset audio collection surface, the stereoscopic sense of the acquired audio data is optimal. Therefore, the corresponding reference audio angle can be set based on this. Similarly, the corresponding reference audio decibel can be set to an audio value at a normal level, such as 50 decibels.

[0102] Next, after obtaining the corresponding audio angle difference and audio decibel difference, the audio range that needs to be adjusted by the target user can be determined, so that the corresponding initial adjustment audio can be retrieved, wherein the initial adjustment audio mainly includes the reference audio segment and the angle filling slot and decibel filling slot interspersed in the reference audio segment. For example, the initial adjustment audio can be "Dear user, hello, please adjust your audio position to "Slot A" and adjust the audio decibel to "Slot B"", slot A represents the angle filling slot, and slot B represents the decibel filling slot. By filling the audio angle difference and the audio decibel difference into the corresponding slots, customized processing of the audio can be achieved to obtain the current adjusted audio;

[0103] Finally, after obtaining the corresponding current adjusted audio through customized processing, the current adjusted audio can be output based on the preset audio acquisition surface, and the corresponding volume during the output process can be based on the benchmark audio decibel for the target user to refer to, thereby facilitating the target user to make corresponding test adjustments to quickly obtain the corresponding audio acquisition data.

[0104] In addition, the above-mentioned processing chips can be configured in the following ways:

[0105] (1) Hardware configuration

[0106] Choose a suitable microphone and sound card to ensure that the acquisition device can capture the user's voice signal with high quality.

[0107] Microphone: Choose a highly sensitive microphone, such as a condenser microphone or a dynamic microphone.

[0108] Sound card: Choose a high-performance sound card to ensure low latency and high-precision analog-to-digital conversion.

[0109] (2) Software preparation

[0110] Install and configure related audio acquisition libraries, such as the PyAudio library in Python, to provide interfaces to audio devices.

[0111] Install PyAudio:

[0112] Install the PyAudio library in the Python environment

[0113] (3) Realize real-time audio acquisition

[0114] Initialize the audio stream:

[0115] Use the PyAudio library to initialize the audio stream and set parameters such as sampling rate, number of channels, and buffer size.

[0116] (4) Data storage and management

[0117] Real-time storage and processing:

[0118] The collected audio data is stored in real time and simple pre-processing is performed, such as noise reduction, framing and windowing.

[0119] Save to file:

[0120] The collected audio data is saved as an audio file for subsequent processing and analysis.

[0121] In step S104, the following contents are included:

[0122] The audio acquisition data is preprocessed to obtain audio optimization data.

[0123] For example, in this embodiment, since it is necessary to perform corresponding sentiment analysis based on the collected audio data, it is necessary to perform a preprocessing operation on the audio data to obtain corresponding audio optimization data.

[0124] For example, in this embodiment, the above-mentioned “preprocessing the audio acquisition data to obtain audio optimization data” may include the following contents:

[0125] Performing noise removal on the audio acquisition data based on a preset frequency domain filtering strategy;

[0126] Normalize the audio data after noise removal;

[0127] In response to completing the normalization processing of the audio collection data, feature extraction is performed on the audio collection data after the normalization processing to obtain audio optimization data corresponding to the audio collection data.

[0128] Below, the above preprocessing process can be described in detail as follows:

[0129] First, for the above-mentioned “noise removal of the audio acquisition data based on a preset frequency domain filtering strategy”, the following method can be used:

[0130] Determine the noise type of the audio collection data based on voice activity detection to obtain noise characteristics corresponding to the audio collection data;

[0131] Performing signal preprocessing including data segmentation and signal windowing on the audio acquisition data to obtain audio acquisition data of corresponding time domain;

[0132] Performing frequency domain conversion on the audio acquisition data after signal preprocessing to obtain audio acquisition data corresponding to the frequency domain;

[0133] Performing filtering processing on the audio collection data corresponding to the frequency domain, and performing time domain conversion on the audio collection data after the filtering processing to obtain audio collection data corresponding to the time domain;

[0134] A performance evaluation is performed on the audio collection data corresponding to the time domain to obtain a perceptual quality corresponding to the audio collection data in the time domain, and in response to the perceptual quality satisfying a preset output condition, noise removal of the audio collection data is completed.

[0135] For example, the above process can be implemented by the following method steps:

[0136] (1) Identification of noise types

[0137] When processing speech signals, different types of noise will have different effects on the quality of the signal. Common noises include:

[0138] Environmental noise: such as background talking, traffic noise, etc.

[0139] Equipment noise: such as the noise generated by the recording equipment itself.

[0140] Power supply noise: such as 50Hz or 60Hz power supply noise (power frequency noise).

[0141] Transient noise: such as coughing, keyboard typing, etc.

[0142] Identifying the specific noise type helps to select the appropriate denoising method. At the same time, noise characteristics can also be obtained through noise estimation methods, such as noise estimation technology based on voice activity detection (VAD).

[0143] (2) Signal preprocessing

[0144] Before frequency domain processing, it is usually necessary to perform some basic preprocessing operations on the speech signal, such as framing and windowing:

[0145] Framing: Split long speech signals into several frames, each of which is usually 20-40 ms in size and 10-20 ms in frame shift. This can reduce the dynamic range and facilitate the calculation of short-time Fourier transform (STFT).

[0146] Windowing: Apply a window function (such as Hamming window or Hanning window) to each frame signal to reduce spectrum leakage. The formula of the window function is:

[0147] ;

[0148] (3) Frequency domain transformation

[0149] Convert time domain signals to the frequency domain to more effectively identify and process noise:

[0150] Short-time Fourier transform (STFT):

[0151] ;

[0152] in, is the input signal, is the window function, is the number of FFT points, is the frame shift, and are the frequency and time indexes respectively.

[0153] (4) Filter design and application

[0154] Based on the frequency characteristics of the noise, design and apply appropriate filters to remove the noise:

[0155] Bandpass filter: It is used to retain the main frequency band of the speech signal (usually 300Hz to 3400Hz) and remove low-frequency (<300Hz) and high-frequency (>3400Hz) noise. The transfer function of the bandpass filter is:

[0156] ;

[0157] in, are the low and high cutoff frequencies of the bandpass filter respectively.

[0158] Adaptive filter: such as Wiener filter, which adaptively adjusts the filter coefficients according to the statistical characteristics of noise and signal. The frequency domain expression of Wiener filter is:

[0159] ;

[0160] in, are the power spectral densities of signal and noise, respectively.

[0161] (5) Time domain restoration

[0162] Convert the filtered frequency domain signal back to the time domain signal:

[0163] Inverse Short Time Fourier Transform (ISTFT):

[0164] ;

[0165] Windowing and overlap-add: During the inverse transform process, each frame is windowed and the overlapping parts are added to restore the complete time domain signal.

[0166] (6) Performance evaluation

[0167] Perform performance evaluation on the denoised signal to ensure signal quality:

[0168] Signal-to-noise ratio (SNR): A common measure of denoising effectiveness. It is defined as the ratio of the power of the denoised signal to the power of the residual noise:

[0169] ;

[0170] Speech quality assessment (such as PESQ): Objective speech quality assessment methods are used to evaluate the perceptual quality of speech signals.

[0171] User auditory evaluation: Through subjective auditory testing, ensure that the denoised speech signal meets the user's perceived quality.

[0172] Through the above steps, various types of noise in the speech signal can be effectively reduced, ensuring the clarity and accuracy of the speech signal, and providing high-quality input data for subsequent feature extraction and emotion recognition.

[0173] Secondly, for the above-mentioned "normalization of the audio data after noise removal", the following methods can be used:

[0174] Performing frame processing on the audio acquisition data after noise removal, and performing window processing on the audio acquisition data after frame processing;

[0175] The mean and standard deviation of the audio collection data after windowing are calculated, and the mean and standard deviation of the audio collection data are substituted into the following formula to normalize the audio collection data:

[0176] ;

[0177] Among them, ( X ) is the audio data after windowing, ( \mu ) is the mean, ( \sigma ) is the standard deviation, and ( X_{\text{norm}} ) is the audio data after normalization.

[0178] For example, the above process can be implemented by the following method steps:

[0179] (1) Data preparation

[0180] Before normalization, you need to ensure that the input data is suitable for processing and analysis. The specific steps are as follows:

[0181] Framing: Split long speech signals into several frames. Each frame is usually 20-40 milliseconds long and the frame shift is 10-20 milliseconds. This step can avoid processing too long signal segments.

[0182] Windowing: Apply a window function (such as Hamming window or Hanning window) to each frame of the signal to reduce spectrum leakage. The formula of the window function is:

[0183] ;

[0184] (2) Calculation of mean and standard deviation

[0185] Calculating the mean and standard deviation of speech signals is a basic step in normalization processing.

[0186] Calculate the mean: The mean is the average value of the signal samples, indicating the collective center position of the signal. The calculation formula is:

[0187] ;

[0188] in, For the samples, (N) is the total number of samples.

[0189] Calculate the standard deviation: The standard deviation indicates the degree of dispersion of the signal samples. The calculation formula is:

[0190] ;

[0191] (3) Application of normalization formula

[0192] Each speech sample is normalized using the previously calculated mean and standard deviation. Normalization adjusts the amplitude of the signal to a standard range, usually zero mean and unit variance.

[0193] Normalization: For each sample, the following formula is applied for normalization:

[0194] ;

[0195] is the original speech signal sample.

[0196] is the mean.

[0197] is the standard deviation.

[0198] ( X_{\text{norm}} ) is the normalized sample.

[0199] The normalized signal will have zero mean and unit variance, which helps in subsequent feature extraction and efficient model training.

[0200] (4) Verification and performance evaluation

[0201] After normalization, the processing results need to be verified and performance evaluated to ensure that the signal quality meets the requirements.

[0202] Signal restoration verification: Check whether the normalized signal can be restored to the original signal through denormalization processing.

[0203] ;

[0204] Performance evaluation indicators: Use statistical indicators such as skewness and kurtosis to evaluate whether the distribution characteristics of the signal are in line with expectations.

[0205] Skewness: Evaluates the symmetry of the signal distribution. The calculation formula is:

[0206] ;

[0207] Kurtosis: Evaluates the flatness of the signal distribution. The calculation formula is:

[0208] ;

[0209] Normalization can improve the stability and consistency of signal processing and analysis, ensuring that speech features have good performance in feature extraction and model training.

[0210] Finally, in response to the above-mentioned “in response to completing the normalization processing of the audio collection data, extracting features from the audio collection data after the normalization processing to obtain audio optimization data corresponding to the audio collection data”, the following method can be used:

[0211] In response to completing the normalization processing of the audio collection data, performing frame processing on the normalized audio collection data, and performing window processing on the frame-processed audio collection data;

[0212] Performing a fast Fourier transform on the audio acquisition data after the windowing process to convert the audio acquisition data corresponding to the time domain into audio acquisition data corresponding to the frequency domain;

[0213] Feature extraction is performed on the audio acquisition data corresponding to the frequency domain to obtain audio optimization data corresponding to the multi-dimensional feature matrix.

[0214] For example, the above process can be implemented by the following method steps:

[0215] Extract MFCC (Mel-Frequency Cepstral Coefficients) and other effective speech features.

[0216] It can be divided into the following key points:

[0217] Framing and Windowing

[0218] Fast Fourier Transform (FFT)

[0219] Mel-Frequency Cepstral Coefficients (MFCC)

[0220] Chroma Features

[0221] Tuned Frequency Oscillation Coefficient (Tonnetz) (Tonnetz Features)

[0222] Other feature extraction (such as ZCR, STFT, etc.)

[0223] Feature Extraction in Detail

[0224] (1) Framing and Windowing

[0225] Framing and windowing are the first steps in feature extraction, which are used to divide long speech signals into small segments for further processing.

[0226] Framing: Speech signals are divided into several frames, each of which is usually 20-40 milliseconds long and has a frame shift of 10-20 milliseconds. Each frame contains enough information but is not too long to avoid complex processing.

[0227] ;

[0228] Where (s[n]) is the original signal, (M) is the frame shift, and (N) is the window length.

[0229] Windowing: Apply a window function (such as a Hamming window or a Hanning window) to each frame of the signal to reduce spectrum leakage. The formula for the window function is:

[0230] ;

[0231] (2) Fast Fourier Transform (FFT)

[0232] FFT converts time domain signals into frequency domain signals and is used to analyze the frequency components of the signal.

[0233] Calculate FFT: Apply a Fast Fourier Transform to each frame to obtain the frequency spectrum.

[0234] ;

[0235] in, is the spectrum, is the input signal, is the number of FFT points.

[0236] Spectrum calculation: The obtained spectrum is used for further feature extraction.

[0237] 3. Mel-Frequency Cepstral Coefficients (MFCC)

[0238] MFCC is the main feature that captures the characteristics of human hearing. The steps are as follows:

[0239] Mel frequency conversion: Convert the spectrum into Mel scale, which is more in line with the auditory characteristics of the human ear.

[0240] ;

[0241] Mel filter bank: The spectrum energy within the Mel frequency range is weighted and summed through a group of triangular filters to obtain the filtered spectrum energy.

[0242] Logarithmic compression: Logarithmic compression is performed on the filtered spectrum energy to enhance small signals and reduce the dynamic range.

[0243] ;

[0244] in, is the spectral energy after filtering, is the energy after logarithmic compression.

[0245] Discrete Cosine Transform (DCT): DCT is applied to the logarithmically compressed energy to obtain MFCC features.

[0246] ;

[0247] (4) Chroma Features

[0248] Chroma characteristics reflect the energy distribution of notes and help capture harmonic information.

[0249] Calculate Chroma spectrum: Convert the spectrum into 12 equal pitch class energy values.

[0250] ;

[0251] in, It is Pitch-like energy, is the spectrum, It belongs to A collection of frequency indices for each pitch class.

[0252] Normalization: The 12-dimensional Chroma features are normalized so that the sum of the energy of all pitch classes is 1.

[0253] (5) Tuned frequency oscillation coefficient (Tonnetz Features)

[0254] The Tonnetz feature is used to analyze pitch perception by capturing the tuning frequency relationship.

[0255] Calculate the simple harmonic oscillation frequency: Get the simple harmonic oscillation frequency through spectrum analysis.

[0256] Mapping to time series: Map the simple harmonic oscillation frequency to time series features to capture pitch change information.

[0257] (6) Other feature extraction (such as ZCR, STFT, etc.)

[0258] Zero-Crossing Rate (ZCR): It is used to describe the rate at which a signal changes and count the number of times a signal crosses zero on the zero horizontal line.

[0259] ;

[0260] Short-time Fourier Transform (STFT): used to analyze the time-frequency characteristics of signals.

[0261] ;

[0262] In step S106, the following contents are included:

[0263] Based on a preset sentiment analysis strategy, sentiment analysis is performed on the audio optimization data to obtain sentiment information corresponding to the audio optimization data.

[0264] For example, in this embodiment, after completing the preprocessing of the audio acquisition data to obtain the corresponding audio optimization data, corresponding emotion analysis can be performed on the corresponding audio optimization data based on the pre-stored preset emotion analysis strategy to obtain emotion information corresponding to the audio optimization data.

[0265] Further, in this embodiment, the above-mentioned “performing sentiment analysis on the audio optimization data based on a preset sentiment analysis strategy to obtain sentiment information corresponding to the audio optimization data” may include the following steps:

[0266] Build an initial sentiment analysis model based on recurrent neural networks and convolutional neural networks;

[0267] Acquire a training data set, wherein the training data set includes speech data each labeled with different emotion categories;

[0268] Training the initial sentiment analysis model based on the training data set to obtain a trained current sentiment analysis model;

[0269] Inputting the audio optimization data into the current sentiment analysis model to perform sentiment analysis, and obtaining confidences corresponding to different sentiment categories;

[0270] The emotional state with the highest corresponding confidence is determined as the emotional information corresponding to the audio optimization data.

[0271] For example, the above process can be implemented by the following method steps:

[0272] Based on the above content, after extracting MFCC, Chroma, Tonnetz and other features, we get a multi-dimensional feature matrix. This feature matrix usually contains the following dimensions:

[0273] Time Frames: The number of frames the signal is divided into.

[0274] Feature dimension: The dimension of each feature vector, such as the dimension of MFCC.

[0275] Assuming that we have extracted N frames and the feature vector dimension of each frame is D, the shape of the feature matrix input to the neural network model is (N, D).

[0276] Next, we input the feature matrix into the constructed deep learning model. Here we take a hybrid model containing CNN and RNN as an example.

[0277] LSTM Layer

[0278] Input: feature map after pooling.

[0279] LSTM operation: Process time series features and capture long-term dependencies.

[0280] Output: hidden state vector of time series, shape (num_lstm_units).

[0281] (1) Model processing

[0282] Convolutional Layer (Conv1D Layer)

[0283] Input: Feature matrix (N, D).

[0284] Convolution operation: Use multiple convolution kernels to scan the input feature matrix and extract local features. The size and number of convolution kernels can be adjusted.

[0285] Activation function: Apply the ReLU activation function to the convolution result to increase nonlinearity.

[0286] Output: Convolutional feature map, shape is (N, num_filters).

[0287] Pooling Layer

[0288] Input: Feature map after convolution.

[0289] Pooling operation: Downsampling operation usually uses maximum pooling (MaxPooling) to reduce the size of feature maps and improve computational efficiency.

[0290] Output: Pooled feature map, shape is (N / / 2, num_filters) (if the pooling window size is 2).

[0291] Dense Layer

[0292] Input: Output vector of LSTM.

[0293] Fully connected operation: Map the input features to a high-dimensional space.

[0294] Activation function: Apply the ReLU activation function.

[0295] Output: The output vector of the fully connected layer.

[0296] Output Layer

[0297] Input: The output vector of the fully connected layer.

[0298] Softmax function: maps the output to the probability distribution of sentiment categories.

[0299] Output: The probability of each sentiment category, shape is (num_classes).

[0300] (2) Output and emotion recognition results

[0301] Probability distribution and sentiment categories

[0302] The output of the model is a probability distribution that represents the probability of each emotion category. The current emotion state is determined by selecting the category with the highest probability.

[0303] Confidence calculation

[0304] Confidence is the highest probability value and represents the confidence of the model in the predicted result.

[0305] (3) Emotional judgment logic

[0306] The user's emotional state can be determined by the output emotion category and confidence level. For example, Table 1 (confidence determination table) shown below:

[0307] Table 1 Confidence determination table

[0308] ;

[0309] The confidence level can be used to understand the model's confidence in the current prediction results. A high confidence level means that the model is more certain about the current sentiment judgment.

[0310] Furthermore, the construction and training process of the initial sentiment analysis model can be as follows:

[0311] For recurrent neural networks, these include:

[0312] (1) Basic structure of RNN

[0313] Recurrent Neural Networks (RNNs) are neural networks that can process sequential data because they retain information about previous inputs and use that information to influence the processing of the current input. Its basic structure includes:

[0314] Input layer: receives input sequence data.

[0315] Hidden layer: The core component of an RNN, consisting of multiple neurons, each of which has an internal state (memory) and uses recurrent connections to share parameters across time steps.

[0316] Output layer: generates the output of the current time step.

[0317] The core part of RNN is its hidden layer. The state of the hidden layer is updated by the following formula:

[0318] ;

[0319] in:

[0320] is the hidden state at the current time step,

[0321] is the input of the current time step,

[0322] is the hidden state at the previous time step,

[0323] and is the weight matrix to be learned,

[0324] is the bias,

[0325] is the activation function, such as tanh or ReLU.

[0326] (2) Long Short-Term Memory (LSTM) Unit

[0327] The LSTM unit is an extended version of the RNN that can better capture long-term dependencies and control the flow of information by introducing a gate mechanism.

[0328] The structure of LSTM includes the following main components:

[0329] Forget Gate: Controls the retention or discarding of the previous state information.

[0330] ;

[0331] Input Gate: Controls the writing of current input information.

[0332] ;

[0333] Cell State Candidate: Generates new candidate memory cell states.

[0334] ;

[0335] Output Gate: Controls the information output to the hidden state.

[0336] ;

[0337] The update formula of the LSTM unit is:

[0338] Update cell status:

[0339] ;

[0340] Update hidden state:

[0341] ;

[0342] (3) Gated Recurrent Unit (GRU)

[0343] GRU is another improved RNN unit, similar to LSTM but with a simpler structure, containing two gates:

[0344] Update Gate: controls the update of information.

[0345] ;

[0346] Reset Gate: Controls the degree of resetting of the previous state information.

[0347] ;

[0348] The update formula of GRU is:

[0349] Candidate version of the current hidden state:

[0350] ;

[0351] Update hidden state:

[0352] ;

[0353] (4) Data input and processing steps

[0354] The data processed by the RNN part is a series of sequence data, with one data point input at each time step.

[0355] Input data: usually features extracted by the preprocessing module, such as MFCC. The input of each time step is represented as .

[0356] Initialize hidden state: RNN needs to initialize hidden state , usually initialized to an all-zero vector.

[0357] Forward propagation: For each time step, calculate the hidden state and output .

[0358] The specific steps are as follows:

[0359] Initialization: Initial hidden state .

[0360] Time step 1: Calculation , and get the output .

[0361] Time step 2: Calculation , and get the output . ...

[0362] Time step T: Calculation , and get the final output .

[0363] (5) Model training and optimization process

[0364] Loss Function: The cross entropy loss function is used to measure the difference between the model prediction and the actual label.

[0365] ;

[0366] in, is the actual label (one-hot encoding), is the model predicted probability.

[0367] Gradient calculation: Use the backpropagation algorithm to calculate the gradient of the loss to each parameter through time backpropagation (BPTT).

[0368] ;

[0369] Optimization algorithm: Optimization algorithms such as Adam or RMSProp are commonly used to update model parameters.

[0370] ;

[0371] in, is the learning rate, is the parameter at step t.

[0372] Regularization technology: In order to prevent overfitting, regularization techniques such as Dropout are often used.

[0373] ;

[0374] where ( p ) is the probability of retaining an activated unit.

[0375] For convolutional neural networks, including:

[0376] (1) Basic concepts and structure of CNN

[0377] Convolutional Neural Network (CNN) is a neural network designed specifically for processing the spatial structure of data, which mainly includes convolutional layer, pooling layer and fully connected layer.

[0378] Convolutional layer: Local features are extracted through convolution operations. Weight sharing and sparse connections make the convolutional layer suitable for processing high-dimensional data.

[0379] Pooling layer: reduces feature dimensions, improves computational efficiency, and has a certain degree of noise resistance.

[0380] Fully connected layer: maps the convolutional and pooled features to the target space, such as the sentiment labels in sentiment classification tasks.

[0381] (2) Detailed process of each part of CNN

[0382] Convolutional Layer

[0383] The convolution layer extracts the local features of the input data through convolution operation, and its formula is:

[0384] ;

[0385] in:

[0386] is the convolution output.

[0387] are input features.

[0388] are input features.

[0389] is bias.

[0390] The feature map generation process includes:

[0391] Convolution operation: Use multiple convolution kernels to scan the input data and generate a set of feature maps.

[0392] Activation function: Applying an activation function (such as ReLU) to the convolution result introduces nonlinearity:

[0393] ;

[0394] Pooling Layer

[0395] The pooling layer is used to downsample the feature map to improve the computational efficiency and noise resistance of the feature representation.

[0396] Common pooling operations include Max Pooling and Average Pooling.

[0397] Max Pooling:

[0398] ;

[0399] in, is the element within the pooling window.

[0400] Average Pooling:

[0401] ;

[0402] Where (Z) is the number of elements in the pooling window.

[0403] Fully Connected Layer

[0404] The fully connected layer maps the convolution and pooling features to the target space.

[0405] Linear transformation:

[0406] ;

[0407] in, is the weight, is the pooling output, is bias.

[0408] Activation function:

[0409] Apply an activation function (such as ReLU or Softmax) to the result of the linear transformation.

[0410] (3) Data input and processing steps

[0411] In order for CNN to process speech data, the input data usually needs to be preprocessed and format converted.

[0412] Input data: usually features obtained by preprocessing and feature extraction modules, such as MFCC, Chroma, etc. These features are usually represented in two-dimensional matrices (such as time and frequency dimensions).

[0413] Input format conversion: Convert the input features into a data format suitable for convolution processing. For speech data, the input dimension is usually (number of channels, height, width), such as a single-channel two-dimensional matrix.

[0414] The specific steps are as follows:

[0415] Data preprocessing: Convert the original speech signal into a feature matrix.

[0416] Convolution layer processing: Perform convolution operation on the input feature matrix to extract local features.

[0417] Pooling layer processing: downsample the convolutional features to improve computational efficiency.

[0418] Fully connected layer output: Map the pooled features to the sentiment classification output.

[0419] (4) Model training and optimization process

[0420] Loss Function

[0421] The commonly used loss function is the cross entropy loss function, which is used to measure the difference between the predicted result and the actual label:

[0422] ;

[0423] in, is the actual label (one-hot encoding), is the model predicted probability.

[0424] Gradient calculation and back propagation

[0425] CNN model training relies on the back-propagation algorithm to update model parameters by calculating the gradient of loss with respect to each parameter.

[0426] Forward propagation: Calculate the output of each layer after the input passes through it.

[0427] Back propagation: Calculate the gradient of loss with respect to each layer parameter and update the parameters.

[0428] ;

[0429] Optimization Algorithm

[0430] Commonly used optimization algorithms include SGD (stochastic gradient descent), Adam, etc., which update model parameters by adjusting the learning rate.

[0431] SGD:

[0432] ;

[0433] in, is the learning rate, is the parameter of step t.

[0434] Adam: Adaptive Learning Rate Optimization Algorithm:

[0435] ;

[0436] Regularization Techniques

[0437] To prevent overfitting, commonly used regularization techniques include Dropout and L2 regularization.

[0438] Dropout: Randomly discard some neurons during training to prevent overfitting.

[0439] ;

[0440] where (p) is the probability of retaining an activated unit.

[0441] L2 regularization: Add the L2 norm of the weights to the loss function to prevent the model from overfitting.

[0442] ;

[0443] where (\lambda) is the regularization coefficient.

[0444] In step S108, the following contents are included:

[0445] An emotional state corresponding to the target user is determined based on the emotional information, and whether emotional intervention on the target user is required is determined based on the emotional state.

[0446] For example, in this embodiment, after obtaining the above-mentioned emotional information, the emotional state of the target user can be determined using the emotional information, wherein the emotional state can include positive emotions, negative emotions, and neutral emotions, positive emotions can include happiness, negative emotions can include sadness and anger, and neutral emotions can include calmness; generally speaking, when the emotional state of the target user is in a negative state, it is necessary to perform emotional intervention on the target user's emotions based on the bad state to help improve the emotional state of the target user.

[0447] In summary, according to the scheme of the present invention, the present invention can obtain the audio collection data of the target user, and further perform subsequent preprocessing on the audio collection data, so as to perform emotional analysis on the target user according to the obtained audio optimization data, thereby determining the emotional state of the target user. At the same time, according to the acquired emotional state, it can also determine whether corresponding emotional intervention is needed to help the target user to be in a positive emotion and improve the psychological correction effect on the target user; in addition, for customer service, the present invention can monitor customer emotions in real time, optimize customer service communication strategies, and improve customer satisfaction; for mental health, the present invention can monitor and analyze user's emotional changes, and assist in mental health diagnosis and intervention; for the field of education, the present invention can optimize teaching methods and improve teaching effects by analyzing students' emotional states; for smart homes, the present invention can improve the naturalness and affinity of human-computer interaction by understanding and responding to users' emotions; and for market research, the present invention can improve products and services by analyzing consumers' emotional responses.

[0448] Another embodiment of the present invention provides a speech emotion analysis system based on artificial intelligence, Figure 3 As the corresponding system block diagram, the system includes:

[0449] The audio collection module is configured to establish an audio collection task in response to receiving a collection trigger instruction at any triggering moment, so as to collect audio of a corresponding target user based on the audio collection task and obtain audio collection data;

[0450] A data optimization module is configured to pre-process the audio acquisition data to obtain audio optimization data;

[0451] A sentiment analysis module, configured to perform sentiment analysis on the audio optimization data based on a preset sentiment analysis strategy to obtain sentiment information corresponding to the audio optimization data;

[0452] The emotion intervention module is configured to determine the emotion state corresponding to the target user based on the emotion information, and determine whether emotion intervention is required for the target user based on the emotion state.

[0453] In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the examples of the present invention. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to implement the content of the present invention described herein, and the above description of specific languages ​​is for the purpose of disclosing the preferred embodiment of the present invention.

[0454] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0455] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof.

[0456] Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in the devices described in the embodiment, or alternatively may be located in one or more devices different from the devices in the examples. The modules in the foregoing examples may be combined into one module or may be divided into multiple submodules.

[0457] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and furthermore may be divided into a plurality of submodules or subunits or subcomponents.

[0458] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present invention and to form different embodiments.

[0459] In addition, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform the functions. Therefore, a processor with necessary instructions for implementing the method or method elements forms a device for implementing the method or method elements. In addition, the elements described herein of the device embodiments are examples of devices for implementing the functions performed by the elements for the purpose of implementing the invention.

[0460] As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe common objects merely indicates that different instances of similar objects are involved, and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner.

[0461] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is primarily selected for readability and instructional purposes, rather than for explaining or limiting the subject matter of the present invention.

Claims

1. A speech emotion analysis method based on artificial intelligence, characterized in that: The following steps are involved: In response to receiving a collection trigger instruction at any triggering moment, an audio collection task is established to collect audio of a corresponding target user based on the voice collection task to obtain audio collection data, including: In response to receiving a collection trigger instruction at any triggering moment, an audio collection task is established, and a preset collection instruction audio corresponding to the audio collection task is retrieved for output; In response to completing the output of the preset collection indication audio, triggering the preset audio collection plane to collect audio corresponding to the target user, and obtaining test audio data corresponding to the target user; Performing a test score on the target user based on the test audio data, and determining an audio attribute corresponding to the target user based on the obtained audio test score; Determine an audio output angle between the target user and the preset audio collection plane based on the test audio data, and determine an offset score corresponding to the target user based on the audio output angle; Determine an audio output decibel corresponding to the target user based on the test audio data, and determine a volume score corresponding to the target user based on the audio output decibel; Performing a fusion calculation on the offset score and the volume score, and comparing the obtained audio test score corresponding to the target user with the reference audio score; When the audio test score is less than the reference audio score, determining the output attribute of the target user as a negative evaluation attribute; When the audio test score is greater than or equal to the reference audio score, determining the output attribute of the target user as a positive evaluation attribute; In response to the audio attribute of the target user being a positive evaluation attribute, calling the preset collection determined audio corresponding to the audio collection task for output; In response to completing the output of the preset collection indication audio, triggering the preset audio collection plane to perform audio collection corresponding to the target user, and obtaining audio collection data corresponding to the target user; or, In response to the audio attribute of the target user being a negative evaluation attribute, creating a current adjustment audio based on the test audio data to output again as an updated preset acquisition indication audio; Preprocessing the audio acquisition data to obtain audio optimization data; Performing sentiment analysis on the audio optimization data based on a preset sentiment analysis strategy to obtain sentiment information corresponding to the audio optimization data; An emotional state corresponding to the target user is determined based on the emotional information, and whether emotional intervention on the target user is required is determined based on the emotional state.

2. The method for speech emotion analysis based on artificial intelligence according to claim 1, characterized in that: In response to the audio attribute of the target user being a negative evaluation attribute, creating a current adjustment audio based on the test audio data to output it again as an updated preset acquisition indication audio, including: In response to the audio attribute of the target user being a negative evaluation attribute, acquiring an audio output angle and an audio output decibel corresponding to the target user; Calculate the difference between the audio output angle and the audio output decibel and the retrieved reference audio angle and reference audio decibel, respectively, to obtain the audio angle difference and the audio decibel difference; Retrieving an initial adjustment audio, wherein the initial adjustment audio includes a reference audio segment and an angle filling slot and a decibel filling slot interspersed in the reference audio segment; Fill the angle filling slot and the decibel filling slot respectively based on the audio angle difference and the audio decibel difference to obtain the current adjusted audio; The currently adjusted audio is used as the updated preset collection indication audio, and the preset collection indication audio is outputted corresponding to the reference audio decibel based on the preset audio collection surface.

3. The method for speech emotion analysis based on artificial intelligence according to claim 1, characterized in that: Preprocessing the audio acquisition data to obtain audio optimization data includes: Performing noise removal on the audio acquisition data based on a preset frequency domain filtering strategy; Normalize the audio data after noise removal; In response to completing the normalization processing of the audio collection data, feature extraction is performed on the audio collection data after the normalization processing to obtain audio optimization data corresponding to the audio collection data.

4. The method for speech emotion analysis based on artificial intelligence according to claim 3 is characterized in that: The audio acquisition data is subjected to noise removal based on a preset frequency domain filtering strategy, including: Determine the noise type of the audio collection data based on voice activity detection to obtain noise characteristics corresponding to the audio collection data; Performing signal preprocessing including data segmentation and signal windowing on the audio acquisition data to obtain audio acquisition data of corresponding time domain; Performing frequency domain conversion on the audio acquisition data after signal preprocessing to obtain audio acquisition data corresponding to the frequency domain; Performing filtering processing on the audio collection data corresponding to the frequency domain, and performing time domain conversion on the audio collection data after the filtering processing to obtain audio collection data corresponding to the time domain; A performance evaluation is performed on the audio collection data corresponding to the time domain to obtain a perceptual quality corresponding to the audio collection data in the time domain, and in response to the perceptual quality satisfying a preset output condition, noise removal of the audio collection data is completed.

5. The method for speech emotion analysis based on artificial intelligence according to claim 3 is characterized in that: The audio data after noise removal is normalized, including: Performing frame processing on the audio acquisition data after noise removal, and performing window processing on the audio acquisition data after frame processing; The mean and standard deviation of the audio collection data after the windowing process are calculated, and the mean and standard deviation corresponding to the audio collection data are substituted into a formula to normalize the audio collection data.

6. The method for speech emotion analysis based on artificial intelligence according to claim 3 is characterized in that: In response to completing the normalization processing of the audio collection data, performing feature extraction on the audio collection data after the normalization processing to obtain audio optimization data corresponding to the audio collection data, including: In response to completing the normalization processing of the audio collection data, performing frame processing on the normalized audio collection data, and performing window processing on the frame-processed audio collection data; Performing a fast Fourier transform on the audio acquisition data after the windowing process to convert the audio acquisition data corresponding to the time domain into audio acquisition data corresponding to the frequency domain; Feature extraction is performed on the audio acquisition data corresponding to the frequency domain to obtain audio optimization data corresponding to the multi-dimensional feature matrix.

7. The method for speech emotion analysis based on artificial intelligence according to claim 1, characterized in that: Performing sentiment analysis on the audio optimization data based on a preset sentiment analysis strategy to obtain sentiment information corresponding to the audio optimization data includes: Build an initial sentiment analysis model based on recurrent neural networks and convolutional neural networks; Acquire a training data set, wherein the training data set includes speech data each labeled with different emotion categories; Training the initial sentiment analysis model based on the training data set to obtain a trained current sentiment analysis model; Inputting the audio optimization data into the current sentiment analysis model to perform sentiment analysis, and obtaining confidences corresponding to different sentiment categories; The emotional state with the highest corresponding confidence is determined as the emotional information corresponding to the audio optimization data.

8. A speech emotion analysis system based on artificial intelligence, characterized in that: include: The audio collection module is configured to establish an audio collection task in response to receiving a collection trigger instruction at any trigger time, so as to collect audio of the corresponding target user based on the voice collection task and obtain audio collection data, including: In response to receiving a collection trigger instruction at any triggering moment, an audio collection task is established, and a preset collection instruction audio corresponding to the audio collection task is retrieved for output; In response to completing the output of the preset collection indication audio, triggering the preset audio collection plane to collect audio corresponding to the target user, and obtaining test audio data corresponding to the target user; Performing a test score on the target user based on the test audio data, and determining an audio attribute corresponding to the target user based on the obtained audio test score; Determine an audio output angle between the target user and the preset audio collection plane based on the test audio data, and determine an offset score corresponding to the target user based on the audio output angle; Determine an audio output decibel corresponding to the target user based on the test audio data, and determine a volume score corresponding to the target user based on the audio output decibel; Performing a fusion calculation on the offset score and the volume score, and comparing the obtained audio test score corresponding to the target user with the reference audio score; When the audio test score is less than the reference audio score, determining the output attribute of the target user as a negative evaluation attribute; When the audio test score is greater than or equal to the reference audio score, determining the output attribute of the target user as a positive evaluation attribute; In response to the audio attribute of the target user being a positive evaluation attribute, calling the preset collection determined audio corresponding to the audio collection task for output; In response to completing the output of the preset collection indication audio, triggering the preset audio collection plane to perform audio collection corresponding to the target user, and obtaining audio collection data corresponding to the target user; or, In response to the audio attribute of the target user being a negative evaluation attribute, creating a current adjustment audio based on the test audio data to output again as an updated preset acquisition indication audio; A data optimization module is configured to pre-process the audio acquisition data to obtain audio optimization data; A sentiment analysis module, configured to perform sentiment analysis on the audio optimization data based on a preset sentiment analysis strategy to obtain sentiment information corresponding to the audio optimization data; The emotion intervention module is configured to determine the emotion state corresponding to the target user based on the emotion information, and determine whether emotion intervention is required for the target user based on the emotion state.

Images