Emotion Recognition Method and Its Device, Computer Device, Storage Medium

Abstract

An embodiment of the present disclosure provides an emotion recognition method, its device, a computer device, and a storage medium, belonging to the technical field of computer information processing. The method includes: obtaining an audio signal; performing frame division processing on the audio signal to obtain a plurality of frame signals, and extracting acoustic features of the plurality of frame signals; constructing a first feature matrix according to the acoustic features; obtaining an emotion scoring value corresponding to the acoustic features; constructing a second feature matrix according to the emotion scoring value; selecting target acoustic features from the plurality of acoustic features according to the first feature matrix and the second feature matrix; constructing an emotion prediction model according to the target acoustic features, and performing emotion recognition according to the emotion prediction model. The principle is simple, the calculation amount is small, the emotion recognition of the audio signal does not require manual operation, the manual emotion annotation work of the audio is reduced, and the accuracy of emotion recognition is high.

Description

Technical Field

[0001] The present invention relates to the technical field of computer information processing, and particularly to an emotion recognition method, an apparatus thereof, a computer device, and a storage medium. Background Art

[0002] Currently, emotion recognition methods based on audio features all extract underlying audio features, ignoring the characteristics of emotion changing over time and not considering the emotion distribution, resulting in weak correlation between acoustic features and emotions. Therefore, how to extract acoustic features related to emotions plays a crucial role in music emotion recognition. Summary of the Invention

[0003] The main purpose of the embodiments of the present disclosure is to propose an emotion recognition method, an apparatus thereof, a computer device, and a storage medium, which can extract acoustic features related to emotions for emotion recognition.

[0004] To achieve the above object, a first aspect of the embodiments of the present disclosure proposes an emotion recognition method, the method including:

[0005] Obtain an audio signal;

[0006] Perform frame splitting on the audio signal to obtain a plurality of frame signals, and extract acoustic features of the plurality of frame signals;

[0007] Construct a first feature matrix according to the acoustic features;

[0008] Obtain an emotion score value corresponding to the acoustic features;

[0009] Construct a second feature matrix according to the emotion score value;

[0010] Select target acoustic features from the plurality of acoustic features according to the first feature matrix and the second feature matrix;

[0011] Construct an emotion prediction model according to the target acoustic features, and perform emotion recognition according to the emotion prediction model.

[0012] In some embodiments, the step of selecting target acoustic features from the plurality of acoustic features according to the first feature matrix and the second feature matrix includes:

[0013] Calculate a first score according to the first feature matrix and the second feature matrix, and calculate a second score according to the first feature matrix;

[0014] Select target acoustic features from the plurality of acoustic features according to the first score and the second score.

[0015] In some embodiments, calculating a first score according to a first feature matrix and a second feature matrix, and calculating a second score according to the first feature matrix includes:

[0016] Calculating a similarity parameter between the first feature matrix and the second feature matrix;

[0017] Calculating a first score according to the similarity parameter;

[0018] Calculating a dissimilarity parameter according to the first feature matrix;

[0019] Calculating a second score according to the dissimilarity parameter.

[0020] In some embodiments, selecting a target acoustic feature from a plurality of acoustic features according to the first score and the second score includes:

[0021] Obtaining a first preset parameter and a second preset parameter;

[0022] Obtaining a third score according to the first score and the first preset parameter;

[0023] Obtaining a fourth score according to the second score and the second preset parameter;

[0024] Obtaining a target score according to the third score and the fourth score;

[0025] Selecting a target acoustic feature from a plurality of acoustic features according to the target score.

[0026] In some embodiments, performing frame segmentation on the audio signal to obtain a plurality of segmented signals, and extracting acoustic features of the plurality of segmented signals includes:

[0027] Performing frame segmentation on the audio signal through overlapping segmentation and a preset window function to obtain a plurality of segmented signals;

[0028] Extracting short-time features of the plurality of segmented signals, and obtaining acoustic features of the plurality of segmented signals according to the short-time features;

[0029] Wherein, the short-time features include short-time average amplitude, short-time threshold crossing rate, and short-time power spectrum.

[0030] In some embodiments, constructing an emotion prediction model according to the target acoustic feature includes:

[0031] Constructing an optimal feature set according to the target acoustic feature;

[0032] Constructing a first regression equation and a second regression equation according to the optimal feature set;

[0033] Constructing an emotion prediction model according to the first regression equation and the second regression equation.

[0034] In some embodiments, constructing an optimal feature set according to the target acoustic features includes:

[0035] Obtain the target acoustic features and store the target acoustic features in an array;

[0036] Construct an optimal feature set according to the elements in the array.

[0037] A second aspect of the embodiments of the present disclosure provides an emotion recognition device, and the device includes:

[0038] A first acquisition module, configured to acquire an audio signal;

[0039] A first feature extraction module, configured to perform frame splitting on the audio signal to obtain a plurality of frame-split signals, and extract acoustic features of the plurality of frame-split signals;

[0040] A first feature calculation module, configured to construct a first feature matrix according to the acoustic features;

[0041] A second acquisition module, configured to acquire an emotion scoring value corresponding to the acoustic features;

[0042] A second feature calculation module, configured to construct a second feature matrix according to the emotion scoring value;

[0043] A second feature extraction module, configured to select target acoustic features from the plurality of acoustic features according to the first feature matrix and the second feature matrix;

[0044] An emotion recognition module, configured to construct an emotion prediction model according to the target acoustic features, and perform emotion recognition according to the emotion prediction model.

[0045] A third aspect of the embodiments of the present disclosure provides a computer device, and the computer device includes a memory and a processor. Among them, a program is stored in the memory, and when the program is executed by the processor, the processor is configured to execute the method according to any one of the embodiments of the first aspect of the present application.

[0046] A fourth aspect of the embodiments of the present disclosure provides a storage medium, which is a computer-readable storage medium, and the storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to execute the method according to any one of the embodiments of the first aspect of the present application.

[0047] The emotion recognition method, device, computer device, and storage medium provided by the embodiments of the present disclosure obtain an audio signal, perform frame segmentation processing on the audio signal to obtain a plurality of segmented signals, extract the acoustic features of the plurality of segmented signals, reflect the characteristics of the change of emotion over time by extracting the time-related acoustic features in the audio signal through frame segmentation, construct a first feature matrix according to the acoustic features, obtain the emotion scoring value corresponding to the acoustic features, construct a second feature matrix according to the emotion scoring value, select target acoustic features from the plurality of acoustic features according to the first feature matrix and the second feature matrix, make the acoustic features related to the emotion, and be able to extract the acoustic features related to the emotion, that is, the target acoustic features. Establish an emotion prediction model according to the target acoustic features, and perform emotion recognition according to the emotion prediction model. The principle is simple, the calculation amount is small, and the emotion recognition of the audio signal does not require manual operation, reducing the manual emotion annotation work of the audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is the first flowchart of the emotion recognition method provided by the embodiments of the present disclosure;

[0049] Figure 2 is Figure 1 the flowchart of step S160 in

[0050] Figure 3 is Figure 2 the flowchart of step S210 in

[0051] Figure 4 is Figure 2 the flowchart of step S220 in

[0052] Figure 5 is Figure 1 the flowchart of step S120 in

[0053] Figure 6 is Figure 1 the flowchart of step S170 in

[0054] Figure 7 [[ID=P40]]is the second flowchart of the emotion recognition method provided by the embodiments of the present disclosure;

[0055] Figure 8 is the module structure diagram of the emotion recognition device provided by the embodiments of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the following further describes the present application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] It should be noted that although functional modules are divided in the schematic diagram of the device and the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different module division from that in the device or a different order from that in the flowchart. Terms such as "first" and "second" in the specification, claims, and the above-mentioned drawings are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this invention belongs. The terms used herein are only for the purpose of describing the embodiments of this invention and are not intended to limit this invention.

[0059] Music is an art that expresses emotions through sounds. Compared with ordinary language, it can convey emotions more directly. Music expresses emotions from aspects such as melody, rhythm, sound, harmony, and timbre, and conveys different emotions contained in it through differences in melody, speed of rhythm, pitch of sound, changes in harmony, and differences in timbre. In order to achieve music emotion retrieval and recommendation, it is often necessary to manually annotate the emotions of music works, which not only involves a huge amount of work but is also easily affected by the subjective opinions of annotators.

[0060] To conform to the situation where humans perceive dynamic music emotions, in the dynamic music emotion recognition task, to recognize the emotion at a certain moment, only the music signal before that moment can be relied on. Currently, most music emotion recognition methods based on audio features extract low-level audio features, ignoring the characteristics of music emotions changing over time; in addition, these methods do not consider the emotion distribution, that is, the correlation between acoustic features and emotions is not strong, and the accuracy of music emotion recognition is low. Accurately analyzing the emotion features contained in music is an important basis for studying human emotion activities. Therefore, there is an urgent need for an automatic, objective, effective dynamic music emotion recognition method that considers time-series emotion-related features.

[0061] Based on this, the main purpose of the embodiments of this disclosure is to propose an emotion recognition method. To reflect the characteristics of music emotions changing over time, short-time features, which are acoustic features related to time in the audio signal, are extracted by frame division. To enhance the correlation between acoustic features and emotions, an emotion score value of the acoustic features is introduced. According to the first feature matrix constructed from the acoustic features and the second feature matrix constructed from the emotion score value, target acoustic features are selected from multiple acoustic features to make the acoustic features related to emotions, and acoustic features related to emotions, that is, target acoustic features, can be extracted. An emotion prediction model is constructed through the target acoustic features, and emotion recognition is automatically performed according to the emotion prediction model. Since the characteristics of emotions changing over time and the correlation between emotions and acoustic features are considered, the emotion recognition accuracy of the emotion prediction model is improved.

[0062] The emotion recognition method, device, computer device, and storage medium provided by the embodiments of the present disclosure will be specifically described through the following embodiments. First, the emotion recognition method in the embodiments of the present disclosure will be described.

[0063] Referring to Figure 1 , according to the emotion recognition method of the first aspect embodiment of the present disclosure, it includes but is not limited to steps S110 to S170.

[0064] S110, obtaining an audio signal;

[0065] S120, performing frame division processing on the audio signal to obtain a plurality of frame signals, and extracting the acoustic features of the plurality of frame signals;

[0066] S130, constructing a first feature matrix according to the acoustic features;

[0067] S140, obtaining an emotion scoring value corresponding to the acoustic features;

[0068] S150, constructing a second feature matrix according to the emotion scoring value;

[0069] S160, selecting target acoustic features from a plurality of acoustic features according to the first feature matrix and the second feature matrix;

[0070] S170, constructing an emotion prediction model according to the target acoustic features, and performing emotion recognition according to the emotion prediction model.

[0071] In step S110, the audio signal may be a song in a music dataset.

[0072] In step S120, the audio signal is a random signal that changes with time and has short-term stationarity. When extracting the acoustic features of the audio signal, the audio signal is subjected to frame division processing to obtain a plurality of short segments, that is, a plurality of frame signals, and one short segment is one frame. The acoustic features include fundamental frequency, fundamental frequency perturbation, amplitude perturbation, harmonic-to-noise ratio, spectral centroid, spectral flux, linear prediction coding coefficients, Mel frequency cepstral coefficients, short-time average amplitude, short-time threshold crossing rate, and short-time power spectrum. The dimensions of these acoustic features are a total of 36 dimensions. Among them, the fundamental frequency reflects the vibration frequency of the vocal cords when pronouncing voiced sounds, which is the reciprocal of the pitch period. Its dimension is 1 dimension, denoted as F0. In the embodiments of the present disclosure, the autocorrelation method is used to calculate the pitch period, and the calculation method of the pitch period is shown in formula (1):

[0073]

[0074] Where x s(m) is the s-th frame audio signal; N is the frame length, and N>0. The embodiments of the present disclosure do not limit the specific value of N, and those skilled in the art can set it according to actual needs; k is the offset. When (-N + 1) ≤ k ≤ (N - 1), R s (k) is not 0. R s (k) has a peak at an integer multiple of the fundamental period. Take the first maximum peak point as the fundamental period point, and take the reciprocal of the fundamental period as the fundamental frequency feature, that is, the F0 feature, to calculate the fundamental frequency value of the s-th frame audio signal.

[0075] It should be noted that one opening and closing of the vocal cords is one fundamental period.

[0076] Fundamental frequency perturbation refers to the small change in the fundamental frequency between adjacent periods of the audio signal, which reflects the periodic vibration of the human vocal cords. Its dimension is 1D, denoted as jitter, and the calculation method is shown in formula (2):

[0077]

[0078] Among them, represents the fundamental frequency of the i-th frame audio signal; S is the total number of frames after the audio signal is framed. When the fundamental frequencies between adjacent frames are mostly the same and only some fundamental frequencies are different, the value of fundamental frequency perturbation is small.

[0079] Amplitude perturbation refers to the small change in amplitude between adjacent periods of the audio signal, which is another important indicator reflecting the periodic vibration of the human vocal cords. Its dimension is 1D, denoted as shimmer, and the calculation method is shown in formula (3):

[0080]

[0081] Among them, A i represents the amplitude of the i-th frame audio signal; S is the total number of frames after the audio signal is framed. When the amplitudes between adjacent frames are mostly the same and only some amplitudes are different, the value of amplitude perturbation is small.

[0082] Harmonic-to-noise ratio refers to the ratio of the harmonic component to the noise component in the audio signal. Its dimension is 1, denoted as HNR, and the calculation method is shown in formula (4):

[0083]

[0084] Among them, E p is the energy of the harmonic component, denoted as S is the total number of frames after the audio signal is framed, N max is the maximum frame length of multiple framed signals, x a (m) is the average wave, that is, the average audio signal, x s(m) is the audio signal of the s-th frame, and the average wave is expressed as E ap is the energy of the noise component, and N s is the frame length of the s-th frame audio signal, then the energy of the noise component is expressed as

[0085] The spectral centroid refers to the centroid of the spectrum, which reflects the brightness of the sound and is calculated using the weighted value of the audio signal frequency. A large value of the spectral centroid indicates that the sound is loud and cheerful. On the contrary, a small value indicates that the sound is low and gloomy. The unit is Hz, and its dimension is 1, denoted as centroid. The calculation method is shown in formula (5):

[0086]

[0087] where centroid[s] is the spectral centroid of the s-th frame audio signal, and X s (k) is the frequency corresponding to the k-th point of the s-th frame audio signal.

[0088] The spectral flux reflects the spectral change between adjacent frames and is calculated based on the power spectra between two adjacent frames. Its dimension is 1, denoted as SF. The calculation method is shown in formula (6):

[0089]

[0090] where SF[s] is the spectral flux of the s-th frame audio signal, and P s (i) is the power of the i-th frequency of the s-th frame audio signal.

[0091] The linear prediction coding coefficient is the signal value at the current moment obtained by linearly combining the signal values at previous moments, with a dimension of 8, denoted as LPC. Let a i be the i-th order linear prediction coding coefficient, and {a i} defines a p-th order linear predictor F(z), then F(z) is expressed as:

[0092]

[0093] Using the first p points of F(z) to predict the sample value at the current point, the signal value at the n-th moment of the current frame is obtained, that is:

[0094]

[0095] In the embodiments of the present disclosure, the value of p is 8, and the linear prediction coding coefficient is calculated through the signal values at previous moments and the current moment.

[0096] Mel-frequency cepstral coefficients are Mel-frequency features proposed based on the cochlear filtering characteristics of the human ear. The Mel-frequency is a non-linear scale obtained based on the sensory judgment of the human ear on the equidistant fundamental frequency change. It follows a linear distribution when the frequency is less than 1000 Hz and a logarithmic distribution when the frequency is greater than 1000 Hz. The calculation method of the Mel-frequency is shown in formula (9):

[0097] Mel(f)=2595log 10 (1+f / 700) (9)

[0098] Where f is the frequency of the audio signal, which is obtained by performing a Fourier transform on the audio signal, and the unit is Hz. The Mel-spectrum is obtained according to formula (9), then taking the logarithm and performing a discrete cosine transform (DCT). The 2nd to 13th coefficients are used as MFCC features, and the dimension of MFCC is 12.

[0099] The dimension of the short-time average amplitude is 1, denoted as F, and the calculation method is shown in formula (10):

[0100]

[0101] Where, F s (k) is the short-time average amplitude of the s-th frame of the audio signal at k points.

[0102] The dimension of the short-time threshold crossing rate is 1, denoted as Z, and the calculation method is shown in formula (11):

[0103]

[0104] Where, Z s is the short-time threshold crossing rate of the s-th frame of the audio signal; h is the threshold value; sgn is the sign function, expressed as:

[0105] It should be noted that the short-time threshold crossing rate is easily affected by low-frequency interference, and the low frequency can be filtered through the threshold value.

[0106] The dimension of the short-time power spectrum is 8, denoted as FFT, and the calculation method is shown in formula (12):

[0107]

[0108] Where, FFT s is the short-time power spectrum of the s-th frame of the audio signal; R s (k) is the short-time autocorrelation function.

[0109] In step S130, a first feature matrix is constructed for each dimension of the acoustic features extracted in step S120, and there are a total of 36 first feature matrices. The first feature matrix is denoted as FM, and its construction method is shown in formula (13):

[0110] FM ij = ρ(F ik , F jk ) 1≤i,j≤m (13)

[0111] Among them, m represents the number of samples of the audio signal; ρ(F ik , F jk ) is the Spearman correlation coefficient of the k-dimensional feature vectors of the i-th sample and the j-th sample, and its value range is [-1, 1]. If the value is 0, it means that the feature vectors of the two samples are not correlated. The features are converted into a rank relationship to form two rank number sequences a and b, and the calculation method of the Spearman correlation coefficient is shown in formula (14):

[0112]

[0113] Among them, m is the number of ranks in the rank number sequence; represents the average value of the rank number sequence a; represents the average value of the rank number sequence b.

[0114] In step S150, a second feature matrix is constructed according to the emotional scoring values of the acoustic features, denoted as LM, and its construction method is shown in formula (15):

[0115] LM ij = Euc(L ik , L jk ), 1≤i,j≤m (15)

[0116] Among them, L is the emotional scoring value, and its value range is [1, 9], which is used to characterize the importance of the acoustic feature in emotion recognition. The larger the value, the greater the role played by the acoustic feature in the emotion recognition task; L ik is the emotional scoring value of the k-dimensional feature of the i-th sample; Euc represents the Euclidean distance.

[0117] In some embodiments, as Figure 2 shown, step S160 specifically includes the steps of:

[0118] S210, calculating a first score according to the first feature matrix and the second feature matrix, and calculating a second score according to the first feature matrix;

[0119] S220, selecting target acoustic features from multiple acoustic features according to the first score and the second score.

[0120] In some embodiments, as Figure 3 shown, step S210 specifically includes the steps of:

[0121] S310, calculating a similarity parameter between a first feature matrix and a second feature matrix;

[0122] S320, calculating a first score according to the similarity parameter;

[0123] S330, calculating a dissimilarity parameter according to the first feature matrix;

[0124] S340, calculating a second score according to the dissimilarity parameter.

[0125] In steps S310 and S320, the similarity parameter between the first feature matrix and the second feature matrix is used as the first score. The calculation method of the similarity parameter is as shown in formula (16):

[0126]

[0127] where C1(i) represents the first score of the i-th feature.

[0128] In steps S330 and S340, a dissimilarity parameter is calculated according to the first feature matrix, and the dissimilarity parameter is used as the second score. The calculation method of the dissimilarity parameter is as shown in formula (17):

[0129] [[ID=3,1]]

[0130] where m is the total number of samples of the audio signal; FM ij is the similarity of the i-th feature of the sample with the j-th other sample; C2(i) represents the second score of the i-th feature, which is the sum of the similarities of the i-th feature of this sample with the i-th features of other samples.

[0131] It should be noted that by introducing the similarity parameter, acoustic features related to emotions can be selected.

[0132] In some embodiments, as Figure 4 shown, step S220 specifically includes the steps of:

[0133] S410, obtaining a first preset parameter and a second preset parameter;

[0134] S420, obtaining a third score according to the first score and the first preset parameter;

[0135] S430, obtaining a fourth score according to the second score and the second preset parameter;

[0136] S440, obtaining a target score according to the third score and the fourth score;

[0137] S450. Select target acoustic features from multiple acoustic features according to the target score.

[0138] In step S440, if the first preset parameter is α1 and the second preset parameter is α2, the calculation method of the target score is shown in formula (18):

[0139] C(i) = α1C1(i) + α2C2(i) (18)

[0140] Wherein, the first term represents the third score; the second term represents the fourth score; C(i) is the target score of the i-th feature.

[0141] In step S450, sort the target scores of each feature from largest to smallest, and select the acoustic features with higher rankings as the target acoustic features. The larger the target score, the stronger the correlation between the acoustic feature and the emotion; on the contrary, the weaker the correlation between the acoustic feature and the emotion.

[0142] It should be noted that the embodiments of the present disclosure do not limit the specific values of the first preset parameter being α1 and the second preset parameter being α2. Those skilled in the art can set them according to actual needs. For example, α1 = 0.5, α2 = 0.5.

[0143] In some embodiments, as Figure 5 shown, step S120 specifically includes the steps of:

[0144] S510. Perform frame division processing on the audio signal through overlapping segmentation and a preset window function to obtain multiple frame signals;

[0145] S520. Extract the short-time features of the multiple frame signals, and obtain the acoustic features of the multiple frame signals according to the short-time features.

[0146] In step S510, in order to make the frames transition smoothly and maintain their continuity, the overlapping segmentation and windowing method are used to perform frame division processing on the audio signal. If the frame length is N and the frame shift is T, then the number of frames S = (X - N + T) / T, and the s-th frame x s (n) is expressed as:

[0147] x s (n) = x(s + n) × w(n) n = 0, T, 2T,... (19)

[0148] Wherein, w(n) is the window function. In order to reduce side lobe leakage, the window function can adopt a Hanning window, which is defined as follows:

[0149]

[0150] In step S520, the short-term features include short-term average amplitude, short-term threshold crossing rate, and short-term power spectrum.

[0151] In some embodiments, such as Figure 6 shown, step S170 specifically includes the steps of:

[0152] S610, constructing an optimal feature set according to the target acoustic features;

[0153] S620, constructing a first regression equation and a second regression equation according to the optimal feature set;

[0154] S630, constructing an emotion prediction model according to the first regression equation and the second regression equation.

[0155] In step S610, an optimal feature set is constructed according to the average value of the target acoustic features. For the convenience of calculation, the average value of the target acoustic features is stored in an array, and all the elements in this array are the optimal feature set. The calculation of the average value of the acoustic features is as follows. Taking spectral centroid, spectral flux, and linear prediction coding coefficients as examples for illustration, the final spectral centroid centroid is calculated according to the spectral centroids of multiple framed signals, that is The final spectral flux SF is calculated according to the spectral fluxes of multiple framed signals, that is The linear prediction coding coefficients of the current frame are calculated according to the average value of the linear prediction coding coefficients corresponding to multiple moments in the current frame. According to the average value of the linear prediction coding coefficients corresponding to multiple framed signals, the final 8 linear prediction coding coefficients are calculated. This average value is obtained by dividing the sum of the corresponding linear prediction coding coefficients of multiple frames by the total number of frames of the audio signal. The calculation methods of the average values of other acoustic features are the same as the above-mentioned methods, and will not be elaborated here one by one.

[0156] In steps S620 and S630, if the constructed optimal feature set is {x1, x2,..., x k}, then the first regression equation is Y′ = β0 + β1x1 + β2x2 + … + β k x k + μ, and the second regression equation is Y″ = β0 + β1x1 + β2x2 + … + β k x k + μ. Among them, Y′ is Valence, that is, valence, Y″ is Arousal, that is, activation degree, and 1 ≤ Y1, Y2 ≤ 9. The emotion prediction model is trained according to the audio signal samples marked with emotions, that is, Y′ and Y″ are known. According to the sample optimal feature set {x1, x2,..., x k}, Y′ and Y″ are used to train the model to obtain {β0, β1, …, β k}.

[0157] When performing sentiment prediction according to the trained model, {β0, β1, …, β k} are known. Input the optimal feature sets {x 1i , x 2i , …, x ki} corresponding to multiple samples into the model, and the output of the model is:

[0158] Y i = β0 + β1x 1i + β2x 2i + … + β k x ki + μ i (21)

[0159] That is:

[0160]

[0161] Perform sentiment prediction according to the output value of the model, that is, identify emotions such as excitement, happiness, satisfaction, relaxation, quietness, calmness, sadness, boredom, drowsiness, anxiety, anger, annoyance, etc. according to Y i ′ and Y i ″. Among them, μ i is a random error term, and n is the total number of samples.

[0162] In some embodiments, as Figure 7 shown, the sentiment recognition method includes but is not limited to steps S710 to S770.

[0163] S710, obtain a song;

[0164] S720, perform frame segmentation on the song through overlapping segmentation and a preset window function to obtain multiple frame signals, and extract the acoustic features of the multiple frame signals;

[0165] S730, calculate the Spearman correlation coefficient of the acoustic features, and construct a first feature matrix according to the Spearman correlation coefficient;

[0166] S740, obtain the sentiment score value of the acoustic features, calculate the Euclidean distance of the acoustic features according to the sentiment score value, and construct a second feature matrix according to the Euclidean distance;

[0167] S750, calculate the similarity parameter according to the elements in the first feature matrix and the elements in the second feature matrix, and calculate the dissimilarity parameter according to the elements in the first feature matrix;

[0168] S760, calculate the target score of the acoustic features according to the similarity parameter and the dissimilarity parameter, and select the target acoustic features from the multiple acoustic features according to the target score;

[0169] S770 constructs an emotion prediction model based on target acoustic features and performs emotion recognition according to the emotion prediction model.

[0170] The emotion recognition method proposed in the embodiments of the present disclosure performs frame division processing on an audio signal to obtain a plurality of frame signals, extracts the acoustic features of the plurality of frame signals, reflects the characteristics of emotion changing over time by extracting the time-related acoustic features in the audio signal through frame division, constructs a first feature matrix according to the acoustic features, obtains the emotion scoring value corresponding to the acoustic features, constructs a second feature matrix according to the emotion scoring value, and selects target acoustic features from the plurality of acoustic features according to the first feature matrix and the second feature matrix, making the acoustic features related to emotion, being able to extract the acoustic features related to emotion, that is, target acoustic features, establish an emotion prediction model according to the target acoustic features, and perform emotion recognition according to the emotion prediction model. The principle is simple, the calculation amount is small, and the emotion recognition of the audio signal does not require manual operation, reducing the manual emotion annotation work for the audio.

[0171] The embodiments of the present disclosure also provide an emotion recognition device, as Figure 8 shown, which can implement the above emotion recognition method. The device includes a first acquisition module 810, a first feature extraction module 820, a first feature calculation module 830, a second acquisition module 840, a second feature calculation module 850, a second feature extraction module 860, and an emotion recognition module 870. Among them, the first acquisition module 810 is used to acquire an audio signal; the first feature extraction module 820 is used to perform frame division processing on the audio signal to obtain a plurality of frame signals and extract the acoustic features of the plurality of frame signals; the first feature calculation module 830 is used to construct a first feature matrix according to the acoustic features; the second acquisition module 840 is used to acquire the emotion scoring value corresponding to the acoustic features; the second feature calculation module 850 is used to construct a second feature matrix according to the emotion scoring value; the second feature extraction module 860 is used to select target acoustic features from the plurality of acoustic features according to the first feature matrix and the second feature matrix; the emotion recognition module 870 is used to construct an emotion prediction model according to the target acoustic features and perform emotion recognition according to the emotion prediction model.

[0172] The emotion recognition device of the embodiments of the present disclosure is used to execute the emotion recognition method in the above embodiments, and its specific processing process is the same as that of the emotion recognition method in the above embodiments, and will not be elaborated here one by one.

[0173] The emotion recognition device proposed in the embodiments of the present disclosure reflects the characteristics of emotion changing over time by extracting the time-related acoustic features in the audio signal through frame division, extracts the target acoustic features related to emotion by obtaining the emotion scoring value corresponding to the acoustic features, and improves the accuracy of emotion recognition.

[0174] The embodiments of the present disclosure also provide a computer device, including:

[0175] At least one processor, and,

[0176] A memory communicatively connected to the at least one processor; wherein,

[0177] The memory stores instructions, and the instructions are executed by the at least one processor so that when the at least one processor executes the instructions, the method according to any one of the embodiments of the first aspect of the present application is implemented.

[0178] The computer device includes: a processor, a memory, an input / output interface, a communication interface, and a bus.

[0179] The processor can be implemented in ways such as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure;

[0180] The memory can be implemented in forms such as a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory and are called by the processor to execute the emotion recognition method of the embodiments of the present disclosure;

[0181] The input / output interface is used to implement information input and output;

[0182] The communication interface is used to implement communication interaction between this device and other devices, and can implement communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and

[0183] The bus transmits information between various components of the device (such as the processor, the memory, the input / output interface, and the communication interface);

[0184] Wherein the processor, the memory, the input / output interface, and the communication interface are communicatively connected to each other inside the device through the bus.

[0185] The embodiments of the present disclosure further provide a storage medium, which is a computer-readable storage medium, and the computer-readable storage medium stores computer-executable instructions for causing a computer to execute the emotion recognition method of the embodiments of the present disclosure.

[0186] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory optionally includes a memory remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0187] The embodiments described in the embodiments of the present disclosure are for more clearly illustrating the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are equally applicable to similar technical problems.

[0188] Those skilled in the art can understand that Figures 1 to 7 the technical solutions shown in do not constitute a limitation on the embodiments of the present disclosure, and may include more or fewer steps than those illustrated, or combine certain steps, or different steps.

[0189] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0191] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of this application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0192] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that three relationships may exist. For example, "A and / or B" may mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or its similar expression refers to any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c may mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or plural.

[0193] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the devices or units can be in an electrical, mechanical or other form.

[0194] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] In addition, each functional unit in various embodiments of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0196] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The foregoing storage medium includes: various media that can store programs, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0197] The preferred embodiments of the embodiments of the present disclosure have been described above with reference to the accompanying drawings. This does not limit the scope of the rights of the embodiments of the present disclosure. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present disclosure shall be within the scope of the rights of the embodiments of the present disclosure.

Claims

1. An emotion recognition method, characterized in that, The method includes: Obtaining an audio signal; Performing frame segmentation on the audio signal to obtain a plurality of segmented signals, and extracting acoustic features of the plurality of segmented signals; Construct a first feature matrix for each of the acoustic features, and the construction method of the first feature matrix is: FM ij = ρ(F ik , F jk ) 1 ≤ i, j ≤ m, where the first feature matrix is denoted as FM, m represents the number of samples of the audio signal, and ρ(F ik , F jk ) is the Spearman correlation coefficient of the k-dimensional feature vectors of the i-th sample and the j-th sample; Obtaining an emotional scoring value corresponding to the acoustic feature; Constructing a second feature matrix according to the emotional scoring value; wherein, each element of the second feature matrix is the Euclidean distance between two emotional scoring values; Selecting target acoustic features from the plurality of acoustic features according to the first feature matrix and the second feature matrix; Constructing an emotion prediction model according to the target acoustic features, and performing emotion recognition according to the emotion prediction model; The selecting target acoustic features from the plurality of acoustic features according to the first feature matrix and the second feature matrix includes: Calculating a similarity parameter between the first feature matrix and the second feature matrix; Calculating a first score according to the similarity parameter; Calculating a dissimilarity parameter according to the first feature matrix; Calculating a second score according to the dissimilarity parameter; Selecting target acoustic features from the plurality of acoustic features according to the first score and the second score.

2. The emotional recognition method according to claim 1, wherein The selecting target acoustic features from the plurality of acoustic features according to the first score and the second score includes: Obtaining a first preset parameter and a second preset parameter; Obtaining a third score according to the first score and the first preset parameter; Obtaining a fourth score according to the second score and the second preset parameter; Obtaining a target score according to the third score and the fourth score; Selecting target acoustic features from the plurality of acoustic features according to the target score.

3. The emotional recognition method according to claim 1, characterized in that The performing frame segmentation on the audio signal to obtain a plurality of segmented signals, and extracting acoustic features of the plurality of segmented signals includes: Performing frame segmentation on the audio signal through overlapping segmentation and a preset window function to obtain a plurality of segmented signals; Extracting short-time features of the plurality of segmented signals, and obtaining acoustic features of the plurality of segmented signals according to the short-time features; Wherein, the short-time features include short-time average amplitude, short-time threshold crossing rate, and short-time power spectrum.

4. The emotion recognition method according to claim 1, wherein The constructing an emotion prediction model according to the target acoustic features includes: Constructing an optimal feature set according to the target acoustic features; Constructing a first regression equation and a second regression equation according to the optimal feature set; Constructing an emotion prediction model according to the first regression equation and the second regression equation.

5. The emotional recognition method according to claim 4, characterized in that, The constructing an optimal feature set according to the target acoustic features includes: Obtaining target acoustic features, and storing the target acoustic features in an array; Constructing an optimal feature set according to the elements in the array.

6. An emotion recognition device, characterized in that, The device includes: A first obtaining module, configured to obtain an audio signal; A first feature extraction module, configured to perform frame segmentation on the audio signal to obtain a plurality of segmented signals, and extract acoustic features of the plurality of segmented signals; The first feature calculation module is used to construct a first feature matrix for each of the acoustic features. The construction method of the first feature matrix is: FM ij = ρ(F ik , F jk ) 1 ≤ i, j ≤ m, where the first feature matrix is denoted as FM, m represents the number of samples of the audio signal, and ρ(F ik , F jk ) is the Spearman correlation coefficient of the k-dimensional feature vectors of the i-th sample and the j-th sample; A second obtaining module, configured to obtain an emotional scoring value corresponding to the acoustic feature; A second feature calculation module, configured to construct a second feature matrix according to the emotional scoring value; wherein, each element of the second feature matrix is the Euclidean distance between two emotional scoring values; A second feature extraction module, configured to select target acoustic features from a plurality of the acoustic features according to the first feature matrix and the second feature matrix; An emotion recognition module, configured to construct an emotion prediction model according to the target acoustic features and perform emotion recognition according to the emotion prediction model; The emotion recognition device is further configured to: calculate a similarity parameter between the first feature matrix and the second feature matrix; Calculate a first score according to the similarity parameter; Calculate a dissimilarity parameter according to the first feature matrix; Calculate a second score according to the dissimilarity parameter; Select target acoustic features from a plurality of acoustic features according to the first score and the second score.

7. Computer device, characterized in that, The computer device includes a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor is configured to execute: The method according to any one of claims 1 to 5.

8. A storage medium, the storage medium being a computer-readable storage medium, characterized in that, The computer-readable storage stores a computer program, and when the computer program is executed by a computer, the computer is configured to execute: The method according to any one of claims 1 to 5.

Images