Speech emotion recognition method and device

Abstract

The embodiment of the invention provides a voice emotion recognition method and device, electronic equipment, a readable storage medium and a computer program product, and relates to the technical field of computers. The method comprises the following steps: framing a voice signal to obtain a plurality of audio frames; determining a frame-level amplitude representation value of the audio frame; based on the frame-level amplitude representation values of the plurality of audio frames, amplitude change relative values of adjacent audio frames are determined to construct an inter-frame amplitude feature sequence, and the inter-frame amplitude feature sequence comprises a plurality of amplitude change relative values arranged according to the time sequence of the audio frames; and inputting the audio inter-frame amplitude feature sequence into a sentiment classification model, and recognizing a sentiment classification result of the voice signal. According to the method, the amplitude change relative values of the adjacent audio frames are determined based on the frame-level amplitude representation values of the multiple audio frames, the amplitude change relative values of the adjacent audio frames are dimensionless relative variations, and the method has high robustness and high sensitivity to different recording environments and tiny emotion changes.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and in particular to voice emotion recognition methods and apparatus, electronic devices, readable storage media, and computer program products. Background Technology

[0002] Voice emotion recognition, as a key technology in the field of human-computer interaction, aims to automatically identify the speaker's emotional state by analyzing voice signals, and has wide application value in fields such as intelligent customer service, health monitoring, and in-vehicle systems.

[0003] Speech emotion recognition technology can be mainly divided into two categories: methods based on traditional acoustic features and methods based on deep learning. Methods based on traditional acoustic features typically rely on hand-designed features, such as energy, pitch, and short-duration amplitude, combined with classifiers like support vector machines for emotion classification. Feature design in these methods often depends on expert knowledge, and the extracted features are sensitive to variations in different speakers, recording devices, and ambient noise, resulting in limited generalization ability. Deep learning-based methods convert speech into time-frequency representations such as spectrograms and utilize models like convolutional neural networks (CNNs) for end-to-end learning. While these methods can automatically learn complex features, they are generally computationally complex, rely on large-scale labeled data, and require significant computational resources for model inference, making real-time deployment difficult in low-power scenarios such as embedded devices.

[0004] Therefore, there is an urgent need in this field for a speech emotion recognition technology that combines high environmental robustness with high computational efficiency. Summary of the Invention

[0005] Embodiments of this disclosure provide methods and apparatuses, electronic devices, readable storage media, and computer program products for voice emotion recognition that can at least partially solve the problems described above or other problems in the art.

[0006] According to a first aspect of this disclosure, a speech emotion recognition method is provided, the method comprising: performing frame segmentation processing on a speech signal to obtain multiple audio frames; determining frame-level amplitude representation values ​​of the audio frames; determining relative amplitude changes of adjacent audio frames based on the frame-level amplitude representation values ​​of the multiple audio frames to construct an inter-frame amplitude feature sequence, wherein the inter-frame amplitude feature sequence includes multiple relative amplitude changes arranged in the temporal order of the audio frames; and inputting the inter-frame amplitude feature sequence into an emotion classification model to identify the emotion classification result of the speech signal.

[0007] In some embodiments of this disclosure, the speech signal is framed to obtain multiple audio frames, including: framing the speech signal with a fixed frame length and a fixed frame shift to obtain multiple audio frames.

[0008] In some embodiments of this disclosure, the frame shift is less than the frame length so that there is an overlap between adjacent audio frames.

[0009] In some embodiments of this disclosure, determining the frame-level amplitude representation value of an audio frame includes: determining an amplitude correlation value based on the instantaneous amplitude values ​​of multiple sampling points within the audio frame; and determining the frame-level amplitude representation value based on a statistic of the amplitude correlation value, wherein the statistic is determined by any of the following class values: the average of the amplitude correlation values ​​of multiple sampling points; the weighted average of the amplitude correlation values ​​of multiple sampling points; and the maximum value among the amplitude correlation values ​​of multiple sampling points.

[0010] In some embodiments of this disclosure, the speech signal is processed by framing to obtain multiple audio frames, including: supplementing the multiple audio frames based on the difference between the predetermined number and the number of multiple audio frames in response to the number of multiple audio frames being less than a predetermined number; and dividing the multiple audio frames into multiple groups based on the predetermined number in response to the number of multiple audio frames being greater than the predetermined number, wherein the predetermined number is determined based on the requirement of the emotion classification model for the length of the input inter-frame amplitude feature sequence.

[0011] In some embodiments of this disclosure, in response to the number of multiple audio frames being less than a predetermined number, multiple audio frames are supplemented based on the difference between the predetermined number and the number of multiple audio frames, including: inserting frames with a frame-level amplitude representation value of zero at the end of the multiple audio frames; or repeating multiple audio frames until the number of repeated multiple audio frames equals the predetermined number.

[0012] In some embodiments of this disclosure, in response to the number of multiple audio frames being greater than a predetermined number, the multiple audio frames are divided into multiple groups, each group comprising a predetermined number of audio frames. This includes dividing the multiple audio frames with a fixed length and a fixed shift distance corresponding to the predetermined number to obtain multiple groups, wherein the fixed shift distance is less than the fixed length so that adjacent groups have overlapping portions.

[0013] In some embodiments of this disclosure, inputting inter-frame amplitude feature sequences into an emotion classification model to identify the emotion classification result of a speech signal includes: inputting multiple inter-frame amplitude feature sequences corresponding to multiple groups into the emotion classification model; identifying multiple preliminary emotion classification results corresponding to multiple groups based on the multiple inter-frame amplitude feature sequences corresponding to multiple groups through the emotion classification model; and fusing the multiple preliminary emotion classification results through the emotion classification model to identify the emotion classification result of the speech signal.

[0014] In some embodiments of this disclosure, the relative values ​​of amplitude changes between adjacent audio frames are determined based on the frame-level amplitude representation values ​​of multiple audio frames to construct an inter-frame amplitude feature sequence, including: determining the difference between the frame-level amplitude representation values ​​of adjacent audio frames and the sum between the frame-level amplitude representation values ​​of adjacent audio frames; and determining the relative values ​​of amplitude changes based on the ratio between the difference and the sum.

[0015] In some embodiments of this disclosure, the relative amplitude change values ​​of adjacent audio frames are determined based on the frame-level amplitude representation values ​​of multiple audio frames to construct an inter-frame amplitude feature sequence. The method further includes: determining the relative amplitude change values ​​based on a preset lookup table, using the frame-level amplitude representation values ​​of adjacent audio frames as an index, wherein the lookup table is used to map the relationship between the frame-level amplitude representation values ​​of adjacent audio frames and the relative amplitude change values.

[0016] In some embodiments of this disclosure, the lookup table is indexed by multiple preset feature values, wherein the multiple preset feature values ​​are all integers within a first predetermined integer representation range. The determination of the relative value of amplitude change based on the preset lookup table and indexed by the frame-level amplitude representation values ​​of adjacent audio frames includes: linearly scaling the frame-level amplitude representation values ​​of adjacent audio frames to a first predetermined integer representation range to obtain an integer index; and searching the lookup table to determine the content corresponding to the feature value equal to the integer index as the relative value of amplitude change.

[0017] In some embodiments of this disclosure, the method further includes constructing a lookup table based on the following steps: performing frame segmentation processing on the sample speech signal to obtain multiple sample audio frames; determining the frame-level amplitude representation value of the sample audio frame; linearly scaling the frame-level amplitude representation value of the sample audio frame to a first predetermined integer representation range to generate multiple preset feature values; determining the relative amplitude change value corresponding to the frame-level amplitude representation value of adjacent sample audio frames; quantizing the relative amplitude change value into an integer value within a second predetermined integer representation range; and constructing a lookup table based on the multiple preset feature values ​​and the integer value.

[0018] In some embodiments of this disclosure, the frame-level amplitude representation value of a sample audio frame is linearly scaled to a first predetermined integer representation range to generate multiple preset feature values, including: determining an upper limit of the first predetermined integer representation range based on a bit width parameter; and linearly scaling the frame-level amplitude representation value of the sample audio frame based on a predetermined amplitude reference value and a bit width parameter, wherein the predetermined amplitude reference value is determined based on an upper limit of the amplitude dynamic range of the sample speech signal.

[0019] In some embodiments of this disclosure, determining the relative value of amplitude change corresponding to the frame-level amplitude representation value of adjacent sample audio frames includes: determining the difference between the frame-level amplitude representation values ​​of adjacent sample audio frames and the sum between the frame-level amplitude representation values ​​of adjacent sample audio frames; and determining the relative value of amplitude change based on the ratio between the difference and the sum.

[0020] In some embodiments of this disclosure, a lookup table is constructed based on multiple preset feature values ​​and integer values, including: constructing a lookup table with multiple preset feature values ​​as indexes and integer values ​​as content.

[0021] According to a second aspect of this disclosure, a speech emotion recognition apparatus is provided, the apparatus comprising: a framing unit configured to perform framing processing on a speech signal to obtain multiple frames; an amplitude determination unit configured to determine frame-level amplitude representation values ​​of audio frames; a feature sequence generation unit configured to determine relative amplitude changes of adjacent audio frames based on the frame-level amplitude representation values ​​of multiple audio frames to construct an inter-frame amplitude feature sequence, wherein the inter-frame amplitude feature sequence includes multiple relative amplitude changes arranged in the temporal order of the audio frames; and a recognition unit configured to input the inter-frame amplitude feature sequence of audio frames into an emotion classification model to recognize the emotion classification result of the speech signal.

[0022] In some embodiments of this disclosure, the framing unit is further configured to: framing the speech signal with a fixed frame length and a fixed frame shift to obtain multiple audio frames.

[0023] In some embodiments of this disclosure, the frame shift is less than the frame length so that there is an overlap between adjacent audio frames.

[0024] In some embodiments of this disclosure, the amplitude determination unit is further configured to: determine an amplitude correlation value based on the instantaneous amplitude values ​​of multiple sampling points within an audio frame; and determine a frame-level amplitude representation value based on a statistic of the amplitude correlation value, wherein the statistic is determined by any of the following class values: the average of the amplitude correlation values ​​of multiple sampling points; the weighted average of the amplitude correlation values ​​of multiple sampling points; and the maximum value among the amplitude correlation values ​​of multiple sampling points.

[0025] In some embodiments of this disclosure, the framing unit is further configured to: supplement the multiple audio frames based on the difference between the predetermined number and the number of multiple audio frames in response to the number of multiple audio frames being less than a predetermined number; and divide the multiple audio frames into multiple groups, each group comprising a predetermined number of audio frames, in response to the number of multiple audio frames being greater than the predetermined number, wherein the predetermined number is determined based on the length requirement of the emotion classification model for the input inter-frame amplitude feature sequence.

[0026] In some embodiments of this disclosure, the framing unit is further configured to: insert frames with a frame-level amplitude representation value of zero at the end of multiple audio frames; or repeat multiple audio frames until the number of repeated audio frames equals a predetermined number.

[0027] In some embodiments of this disclosure, the framing unit is further configured to divide multiple audio frames into multiple groups by a fixed length and a fixed shift distance corresponding to a predetermined number, wherein the fixed shift distance is less than the fixed length so that there is an overlap between adjacent groups.

[0028] In some embodiments of this disclosure, the identification unit is further configured to: input multiple inter-frame amplitude feature sequences corresponding to multiple groups respectively into an emotion classification model; identify multiple preliminary emotion classification results corresponding to multiple groups based on the multiple inter-frame amplitude feature sequences corresponding to multiple groups respectively through the emotion classification model; and fuse the multiple preliminary emotion classification results through the emotion classification model to identify the emotion classification result of the speech signal.

[0029] In some embodiments of this disclosure, the feature sequence generation unit is further configured to: determine the difference between frame-level amplitude representation values ​​of adjacent audio frames and the sum between frame-level amplitude representation values ​​of adjacent audio frames; and determine the relative value of amplitude change based on the ratio between the difference and the sum.

[0030] In some embodiments of this disclosure, the feature sequence generation unit is further configured to: determine the relative value of amplitude change based on a preset lookup table and indexed by the frame-level amplitude representation value of adjacent audio frames, wherein the lookup table is used to map the relationship between the frame-level amplitude representation value and the relative value of amplitude change of adjacent audio frames.

[0031] In some embodiments of this disclosure, the lookup table is indexed by multiple preset feature values, wherein the multiple preset feature values ​​are all integers within a first predetermined integer representation range. The feature sequence generation unit is further configured to: linearly scale the frame-level amplitude representation values ​​of adjacent audio frames to the first predetermined integer representation range to obtain an integer index; and search the lookup table to determine the content corresponding to the feature value that is equal to the integer index as the relative value of amplitude change.

[0032] In some embodiments of this disclosure, the feature sequence generation unit is further configured to: perform frame segmentation processing on the sample speech signal to obtain multiple sample audio frames; determine the frame-level amplitude representation value of the sample audio frame; linearly scale the frame-level amplitude representation value of the sample audio frame to a first predetermined integer representation range to generate multiple preset feature values; determine the relative amplitude change value corresponding to the frame-level amplitude representation value of the adjacent sample audio frame; quantize the relative amplitude change value into an integer value within a second predetermined integer representation range; and construct a lookup table based on the multiple preset feature values ​​and the integer value.

[0033] In some embodiments of this disclosure, the feature sequence generation unit is further configured to: determine an upper limit of a first predetermined integer representation range based on a bit width parameter; and linearly scale the frame-level amplitude representation value of the sample audio frame based on a predetermined amplitude reference value and the bit width parameter, wherein the predetermined amplitude reference value is determined based on an upper limit of the amplitude dynamic range of the sample speech signal.

[0034] In some embodiments of this disclosure, the feature sequence generation unit is further configured to: determine the difference between the frame-level amplitude representation values ​​of adjacent sample audio frames and the sum between the frame-level amplitude representation values ​​of adjacent sample audio frames; and determine the relative value of amplitude change based on the ratio between the difference and the sum.

[0035] In some embodiments of this disclosure, the feature sequence generation unit is further configured to: construct a lookup table with multiple preset feature values ​​as indexes and integer values ​​as content.

[0036] According to a third aspect of this disclosure, an electronic device is provided, the electronic device including a processor that can be used to implement the speech emotion recognition method in the first aspect and any possible implementation of the first aspect.

[0037] According to a fourth aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, which, when executed by a processor, implement the speech emotion recognition method of the first aspect and any possible implementation thereof.

[0038] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the speech emotion recognition method in the first aspect and any possible implementation thereof.

[0039] The speech emotion recognition method, apparatus, electronic device, readable storage medium, and computer program product provided by the embodiments of this disclosure determine the relative amplitude changes of adjacent audio frames based on the frame-level amplitude representation values ​​of multiple audio frames. These relative amplitude changes of adjacent audio frames are dimensionless relative changes, thus exhibiting high robustness and sensitivity to different recording environments and subtle emotional changes, thereby demonstrating high adaptability to different recording environments and different speakers. Furthermore, in the speech emotion recognition scheme provided by the embodiments of this disclosure, multiple audio frames are obtained by framing the speech signal; the relative amplitude changes of adjacent audio frames are determined using the frame-level amplitude representation values ​​of the multiple audio frames to construct an inter-frame amplitude feature sequence; by inputting this inter-frame amplitude feature sequence into an emotion classification model, the emotion classification result of the speech signal can be identified. Therefore, the calculation in the speech emotion recognition process is simple, requiring no complex transformations, greatly reducing the demand for computing resources and power consumption, and providing possibilities for real-time processing and embedded deployment.

[0040] It should be understood that the description in this section is not intended to identify key or important features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.

[0041] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0042] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figures 1 to 4 These are flowcharts of a speech emotion recognition method provided according to exemplary embodiments of this disclosure; Figure 5 This is a block diagram of a voice emotion recognition device provided according to exemplary embodiments of the present disclosure; and Figure 6 This is a schematic block diagram of an electronic device provided according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0043] The various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0044] The term “exemplary” as used herein means “serving as an example, implementation method, or illustration.” Any implementation method described herein as “exemplary” is not necessarily to be construed as superior to or better than other implementation methods.

[0045] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, apparatuses, means, elements, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0046] Speech emotion recognition, a key technology in human-computer interaction, aims to automatically identify and analyze the speaker's emotional state, such as happiness, anger, sadness, and calmness, from collected speech signals. Automatically identifying the speaker's emotional state through speech signal analysis has broad application prospects in fields such as intelligent customer service, health monitoring, and in-vehicle systems. Speech emotion recognition technology can be mainly divided into two categories: methods based on traditional acoustic features and methods based on deep learning.

[0047] Traditional acoustic feature-based recognition methods rely on hand-designed acoustic features. For example, these methods extract frame-level or statistical-level acoustic parameters, such as energy, pitch, and short-duration amplitude, from speech signals and then combine them with traditional machine learning classifiers for emotion classification. These methods, to some extent, utilize the dynamic changes in speech signals to capture emotional information.

[0048] Therefore, significant differences in basic volume and average speech rate among speakers due to their physiological structure and pronunciation habits, as well as environmental factors such as the sensitivity of recording equipment, transmission gain, and the distance between the speaker and the microphone, directly affect the accuracy of recognition methods based on traditional acoustic features. These external factors introduce variations unrelated to emotion itself, but they significantly interfere with the absolute value-based feature extraction in traditional acoustic feature-based recognition methods. This leads to a sharp decline in recognition performance when faced with speakers not covered by the training data or new environments. The methods exhibit poor adaptability to speaker and recording environments and insufficient environmental robustness.

[0049] Deep learning-based methods convert speech signals into time-frequency representations, such as spectrograms or Mel spectrograms, and then train them end-to-end into deep neural networks. Deep neural networks can include convolutional neural networks, recurrent neural networks, and others. Deep learning models can automatically learn complex time-frequency features and nonlinear patterns, and typically achieve higher recognition accuracy than traditional methods when given massive amounts of labeled data.

[0050] However, while machine learning-based speech emotion recognition methods demonstrate high accuracy in modeling nonlinear emotion patterns and capturing complex speech features, they rely on computationally complex feature extraction, resulting in high computational demands. Furthermore, deep learning-based solutions, although eliminating manual feature design, suffer from complex model structures and large parameter counts, requiring significant computational resources and storage space for time-frequency transformations and model inference. This high computational complexity makes such speech emotion recognition solutions difficult to deploy efficiently in embedded devices, mobile terminals, or real-time voice interaction systems requiring low latency and high concurrency.

[0051] To at least address the aforementioned problems, embodiments of this disclosure provide a speech emotion recognition method and apparatus, electronic device, readable storage medium, and computer program product. Based on frame-level amplitude representation values ​​of multiple audio frames, it determines the relative amplitude change value of adjacent audio frames. This relative amplitude change value of adjacent audio frames is a dimensionless relative change, thus exhibiting high robustness and high sensitivity to different recording environments and subtle emotional changes, thereby demonstrating extremely high adaptability to different recording environments and different speakers.

[0052] Furthermore, in the speech emotion recognition scheme provided by the embodiments of this disclosure, multiple audio frames are obtained by performing frame segmentation processing on the speech signal; the relative values ​​of amplitude changes between adjacent audio frames are determined by the frame-level amplitude representation values ​​of the multiple audio frames to construct an inter-frame amplitude feature sequence; by inputting the inter-frame amplitude feature sequence into the emotion classification model, the emotion classification result of the speech signal can be identified. Therefore, the calculation in the speech emotion recognition process is simple, without the need for complex transformations, which greatly reduces the demand for computing resources and power consumption, and provides the possibility for real-time processing and embedded deployment.

[0053] Some embodiments of this disclosure provide a speech emotion recognition method. Figure 1 This is a flowchart of a voice emotion recognition method 1000 provided according to an exemplary embodiment of the present disclosure.

[0054] like Figure 1 As shown, the speech emotion recognition method 1000 includes: Step S1: Perform frame segmentation on the speech signal to obtain multiple audio frames.

[0055] Step S2: Determine the frame-level amplitude representation value of the audio frame.

[0056] Step S3: Based on the frame-level amplitude representation values ​​of multiple audio frames, determine the relative values ​​of amplitude changes between adjacent audio frames to construct an inter-frame amplitude feature sequence, wherein the inter-frame amplitude feature sequence includes multiple relative values ​​of amplitude changes arranged in the temporal order of the audio frames.

[0057] Step S4: Input the amplitude feature sequence between audio frames into the emotion classification model to identify the emotion classification result of the speech signal.

[0058] Speech signals are a typical non-stationary signal, but their characteristics can be considered relatively stable in a short time of milliseconds. Therefore, the speech emotion recognition scheme provided by the embodiments of this disclosure can perform short-time frame processing on the input continuous speech signal, thereby cutting the continuous speech signal into a series of discrete signal units suitable for analysis. The discrete signal unit is the audio frame, and the frame length of the audio frame is at the millisecond level, thus it is a stable audio signal suitable for analysis.

[0059] The frame-level amplitude representation value of an audio frame can be understood as a single numerical value representing the overall amplitude or energy level of a single audio frame. By determining the frame-level amplitude representation value of an audio frame, a complex audio frame including multiple sampling points can be summarized into a scalar feature that can be used for subsequent inter-frame relationship analysis.

[0060] Based on the frame-level amplitude representation values ​​of multiple audio frames, the relative amplitude changes of adjacent audio frames are determined to construct an inter-frame amplitude feature sequence. The relative amplitude changes of adjacent audio frames can be understood as feature vectors used for inter-frame relationship analysis; these are dimensionless relative changes. Alternatively, based on the frame-level amplitude representation values ​​of multiple audio frames, feature vectors that characterize the relative changes in amplitude or energy levels of adjacent audio frames can be determined. By traversing all adjacent frame pairs, a set of relative amplitude changes arranged temporally across multiple audio frames can be obtained; this set can be understood as an inter-frame amplitude feature sequence.

[0061] Emotion classification models can be trained to learn the mapping relationship from inter-frame amplitude feature sequences to specific emotion categories, which may include happiness, anger, sadness, calmness, etc.

[0062] Therefore, the speech emotion recognition method provided by the embodiments of this disclosure determines the relative value of amplitude change between adjacent audio frames based on the frame-level amplitude representation values ​​of multiple audio frames. This relative value of amplitude change between adjacent audio frames is a dimensionless relative change. From a psychoacoustic perspective, the human ear's perception of changes in sound intensity follows the Weber-Fechner law, meaning that the perceived quantity is proportional to the relative change in the stimulus, rather than the absolute change. Therefore, the speech emotion recognition scheme based on this dimensionless relative change can exhibit high robustness and high sensitivity to different recording environments and subtle emotional changes, thus demonstrating extremely high adaptability to different recording environments and different speakers.

[0063] Furthermore, the process of obtaining the inter-frame amplitude feature sequence using the above method is very simple, completely avoiding complex operations such as Fast Fourier Transform, Mel filter bank filtering, or calculation of higher-order statistics. This computationally lightweight characteristic enables the speech emotion recognition scheme provided by the embodiments of this disclosure to run efficiently on embedded devices with limited computing resources, mobile terminals, or servers requiring high-concurrency processing, providing a solid technical foundation for real-time, low-power speech emotion recognition applications.

[0064] Step S1

[0065] In some embodiments of this disclosure, step S1, which performs framing processing on the speech signal to obtain multiple audio frames, may include: framing the speech signal with a fixed frame length and a fixed frame shift to obtain multiple audio frames. The frame length can be understood as the duration of each short-time analysis window used to segment the speech signal. Exemplarily, the frame length can be a fixed value in the millisecond range, such as 25 milliseconds. Depending on the sampling rate, the frame length can be directly converted to the number of sampling points contained in each audio frame. The frame shift can be understood as the time interval between the start points of two adjacent short-time analysis windows in the framing process. Exemplarily, the frame shift can be a fixed value in the millisecond range, such as 25 milliseconds.

[0066] By using fixed frame lengths and fixed frame shifts for frame segmentation, a uniform audio frame sequence can be generated. This regular data structure is the input format desired by most sentiment classification models, greatly simplifying the model design and training process.

[0067] Optionally, the frame shift can be less than the frame length to allow overlap between adjacent audio frames. Since the changing patterns of speech emotion are continuous and gradual, key acoustic cues characterizing emotional transitions may appear precisely at the boundaries between two frames. Therefore, by introducing overlap between adjacent audio frames, it is ensured that the speech signal at any given time point is covered by at least one complete audio frame. This reduces the risk of losing emotional features due to frame boundary effects, allowing the generated inter-frame amplitude feature sequence to more completely and continuously depict the dynamic process of emotional change.

[0068] For example, step S1, which performs frame segmentation on the speech signal to obtain multiple audio frames, may include sampling operations and frame segmentation. In the sampling operation, a 16kHz sampling rate and a 16-bit mono format can be used to sample a 10-second segment of user speech, obtaining 160,000 sampling points. In the frame segmentation, a fixed frame length of 25ms and a fixed frame shift of 10ms can be used to analyze the speech signal.

[0069] In this embodiment, based on the sampling rate and frame length, the number of sampling points within an audio frame is determined to be 400. For a 10-second speech signal, with a frame shift of 10ms, the total number of frames is 1000. Since the frame length is greater than the frame shift, there is a 15ms overlap between adjacent audio frames, which ensures a smooth transition of speech signal features.

[0070] Optionally, in some embodiments of this disclosure, step S1, which performs frame segmentation processing on the speech signal to obtain multiple audio frames, may further include: supplementing the multiple audio frames based on the difference between the predetermined number and the number of multiple audio frames in response to the number of multiple audio frames being less than a predetermined number; and dividing the multiple audio frames into multiple groups based on the predetermined number in response to the number of multiple audio frames being greater than the predetermined number, wherein the predetermined number is determined based on the requirement of the emotion classification model for the length of the input inter-frame amplitude feature sequence.

[0071] Specifically, in real-world speech emotion recognition applications, the duration of user-input speech signals is not fixed. For example, in intelligent customer service interactions, a user's speech may be as short as less than a second, such as a brief "yes" or "no" response; or it may be as long as tens of seconds, such as describing a specific problem. However, most emotion classification models designed for optimal performance, especially deep learning-based models, typically require a fixed-length feature sequence in their input layer.

[0072] To address the mismatch between variable natural speech input and the model's fixed input requirements, a predetermined number of audio frames can be determined based on the length requirements of the inter-frame amplitude feature sequence of the input from the emotion classification model. Based on this predetermined number, it can be analyzed whether the number of multiple audio frames obtained after frame segmentation processing is sufficient to ensure that the audio inter-frame amplitude feature sequence generated from them meets the model's fixed input requirements.

[0073] In response to the number of multiple audio frames being less than a predetermined number, the multiple audio frames are supplemented based on the difference between the predetermined number and the number of multiple audio frames; and in response to the number of multiple audio frames being greater than a predetermined number, the multiple audio frames are divided into multiple groups based on the predetermined number.

[0074] In some implementations, after dividing the multiple audio frames into multiple groups based on the predetermined number in response to the number of multiple audio frames being greater than a predetermined number, each group may include a predetermined number of audio frames.

[0075] In other embodiments, after dividing the multiple audio frames into multiple groups based on a predetermined number in response to the number of multiple audio frames exceeding a predetermined number, the number of audio frames included in the multiple groups may not be equal. For example, the number of audio frames in at least one of the multiple groups may be less than the predetermined number. In this embodiment, the group with fewer than the predetermined number of audio frames (hereinafter referred to as the first group) may be supplemented. For example, frames with a frame-level amplitude representation value of zero may be added to the end of the multiple audio frames in the first group; or, the multiple audio frames in the first group may be repeated until the number of repeated multiple audio frames in the first group reaches or exceeds the predetermined number, and multiple audio frames equal to the predetermined number may be extracted from the entire audio frames in the expanded first group from front to back for subsequent speech emotion recognition.

[0076] In this way, the first group with fewer than the predetermined number of audio frames, after the above supplementation, can be used together with the remaining groups with the predetermined number of audio frames in the multiple groups for speech emotion recognition. Alternatively, the groups with fewer than the predetermined number of audio frames formed after grouping can be deleted, and only the groups with the predetermined number of audio frames in the multiple groups can be used for speech emotion recognition.

[0077] This implementation can seamlessly process all voice signals, from very short commands to lengthy narratives, so that the voice emotion recognition scheme provided by the present disclosure is no longer limited to regular-length voice samples in the laboratory, but can be directly applied to complex and varied voice input scenarios in the real world, greatly enhancing the practical value and commercial potential of the technology.

[0078] Optionally, in response to the number of multiple audio frames being less than a predetermined number, frames with a frame-level amplitude representation value of zero can be added to the end of the multiple audio frames based on the difference between the predetermined number and the number of multiple audio frames. Alternatively, in response to the number of multiple audio frames being less than a predetermined number, multiple audio frames can be repeated based on the difference between the predetermined number and the number of multiple audio frames until the number of repeated audio frames equals the predetermined number.

[0079] For example, in response to the number of multiple audio frames being less than a predetermined number, the process of repeating multiple audio frames based on the difference between the predetermined number and the number of multiple audio frames until the number of repeated audio frames equals the predetermined number can be implemented by: copying all the obtained audio frames as the original sequence and concatenating the copied sequence after the original sequence to expand the original sequence. If the total number of expanded frames is still less than the predetermined number, the above copying and concatenation operation is repeated until the total number of expanded frames reaches or exceeds the predetermined number. If the total number of expanded frames exceeds the predetermined number, multiple audio frames equal to the predetermined number can be extracted from the total number of expanded audio frames from front to back for subsequent speech emotion recognition.

[0080] Alternatively, in response to the number of multiple audio frames being less than a predetermined number, multiple audio frames can be repeated based on the difference between the predetermined number and the number of multiple audio frames until the number of repeated audio frames equals the predetermined number. This can be implemented by: using all the obtained audio frames as the original sequence, selecting multiple audio frames arranged sequentially in the original sequence (hereinafter referred to as repeating units) for repetition, and then concatenating the copied sequence after the original sequence to expand the original sequence. If the total number of expanded frames is still less than the predetermined number, the above copying and concatenation operation is repeated until the total number of expanded frames reaches or exceeds the predetermined number. If the total number of expanded frames exceeds the predetermined number, multiple audio frames equal to the predetermined number can be extracted from the entire expanded audio frame sequence from front to back for subsequent speech emotion recognition. Optionally, the number of repeating units can be determined according to actual needs. For example, the number of repeating units can be calculated based on the difference between the predetermined number and the number of multiple audio frames obtained after frame processing, or the number of repeating units can be set to a fixed value.

[0081] By performing a "supplement" operation on multiple audio frames after frame segmentation, it is ensured that the amplitude feature sequence between audio frames in the input model contains complete contextual information, thus avoiding model misjudgment due to insufficient data.

[0082] "Based on the difference between a predetermined number and the number of multiple audio frames, repeat multiple audio frames until the number of repeated audio frames equals the predetermined number" can be understood as a strategy of repeating the entire speech. This strategy can maintain the periodic pattern of the original emotion to the maximum extent without introducing invalid information, thereby improving the confidence and accuracy of short speech emotion classification.

[0083] In some embodiments of this disclosure, in response to the number of multiple audio frames being greater than a predetermined number, dividing the multiple audio frames into multiple groups may include: dividing the multiple audio frames with a fixed length and a fixed shift distance corresponding to the predetermined number to obtain multiple groups, wherein the fixed shift distance is less than the fixed length so that there is an overlap between adjacent groups.

[0084] By performing a "segmentation" operation on multiple audio frames after frame-segmentation, long speech can be decomposed into multiple segments that match the model input, ensuring that the emotion classification model can perform detailed analysis on each local segment. This prevents the multiple mixed emotions or emotional evolution processes that may exist in long speech from being averaged out, allowing the system to capture richer and more nuanced emotional dynamics. In addition, the overlapping parts between adjacent groups ensure a smooth transition of speech signal features.

[0085] Step S2

[0086] In some embodiments of this disclosure, step S2, determining the frame-level amplitude representation value of the audio frame, may include: determining an amplitude correlation value based on the instantaneous amplitude values ​​of multiple sampling points within the audio frame; and determining the frame-level amplitude representation value based on a statistic of the amplitude correlation value, wherein the statistic is determined by any of the following class values: the average of the amplitude correlation values ​​of multiple sampling points; the weighted average of the amplitude correlation values ​​of multiple sampling points; and the maximum value among the amplitude correlation values ​​of multiple sampling points.

[0087] A frame-level amplitude representation can be understood as a single scalar representing the overall signal amplitude or energy intensity within a single audio frame. The calculation of this value forms the basis for subsequent analysis of relative changes between frames, and its determination process may include: obtaining the amplitude correlation value for each sampling point; and calculating representative frame-level statistics based on these amplitude correlation values.

[0088] The amplitude correlation value of a sampling point can be understood as a numerical value derived from the original sampling point data and used to characterize the instantaneous intensity of the signal. Optionally, the amplitude correlation value of a sampling point can be the instantaneous amplitude of the sampling point. Optionally, the amplitude correlation value of a sampling point can also be the absolute value of the instantaneous amplitude of the sampling point. Optionally, the amplitude correlation value of a sampling point can also be the square of the instantaneous amplitude of the sampling point. In other words, the amplitude correlation value of a sampling point can be either the instantaneous amplitude of the sampling point or a non-negative value obtained by mathematical transformation of the instantaneous amplitude value.

[0089] As shown in formula (1), by way of example, the amplitude correlation value of the sampling point can be defined as the instantaneous amplitude of the sampling point, where [ [] is the instantaneous amplitude value of the k-th sample point within the t-th audio frame. The t-th audio frame may include N sample points. The frame-level amplitude representation value can be obtained by calculating the average of the absolute values ​​of the instantaneous amplitude values ​​of all sample points within the audio frame. .

[0090] (1)

[0091] Optionally, the statistic is not limited to the above implementation method that determines it by the sum of the instantaneous amplitude values ​​of all sampling points within the audio frame, but can also be determined by any of the following class values: the average value of the amplitude correlation values ​​of multiple sampling points; the weighted average value of the amplitude correlation values ​​of multiple sampling points; and the maximum value among the amplitude correlation values ​​of multiple sampling points.

[0092] For example, if the audio signal is framed in 25ms frames, at a sampling rate of 16kHz, each audio segment can contain 400 sampling points N. Optionally, the amplitude correlation value of the sampling point can also be the absolute value of the instantaneous amplitude of the sampling point.

[0093] Therefore, the absolute value of the instantaneous amplitude at 400 sampling points can be calculated, where the absolute value of the instantaneous amplitude at the k-th sampling point can be expressed as: Optionally, a statistical measure of the amplitude correlation value of the audio frame can be used to determine the frame-level amplitude representation value. This statistical measure is the average of the absolute values ​​of 400 sampling points. Through this calculation, the complex waveform representing these 400 sampling points is summarized into a single numerical value. It stably reflects the average amplitude level of speech within this 25ms time period.

[0094] Therefore, the embodiments of this disclosure compress an audio frame that may contain hundreds of sampling points into a representative scalar during the process of determining the frame-level amplitude representation value of an audio frame, greatly reducing the amount of data in subsequent processing. In addition, using statistical measures to obtain the frame-level amplitude representation value can effectively smooth out interference caused by instantaneous noise or small fluctuations within the frame, and the obtained frame-level amplitude representation value is more representative and stable than any isolated sampling point.

[0095] Step S3

[0096] Figure 2 This is a flowchart of a voice emotion recognition method 1000 provided according to an exemplary embodiment of the present disclosure.

[0097] like Figure 2 As shown, in some embodiments of this disclosure, step S3, based on the frame-level amplitude representation values ​​of multiple audio frames, determines the relative value of amplitude change between adjacent audio frames to construct an inter-frame amplitude feature sequence. This may include step 3-1: determining the difference between the frame-level amplitude representation values ​​of adjacent audio frames and the sum between the frame-level amplitude representation values ​​of adjacent audio frames; and determining the relative value of amplitude change based on the ratio between the difference and the sum.

[0098] As shown in formula (2), the frame-level amplitude representation value of the t-th audio frame can be represented as: The frame-level amplitude representation value of the (t-1)th audio frame can be represented as: The relative values ​​of amplitude changes between adjacent audio frames can be characterized as In some embodiments of this disclosure, the relative values ​​of amplitude changes between adjacent audio frames are... The difference between the frame-level amplitude representation value of the t-th audio frame and the frame-level amplitude representation value of the (t-1)-th audio frame can be obtained from this value. And the sum of the frame-level amplitude representation value of the t-th audio frame and the frame-level amplitude representation value of the (t-1)-th audio frame. The ratio between them is used to represent the ratio.

[0099]

[0100] In this embodiment, the relative value of the amplitude change of adjacent audio frames is divided by the amplitude of adjacent audio frames, which cancels out the overall energy difference of the speech signal. Therefore, it is not sensitive to factors such as volume changes, microphone sensitivity differences, and recording distance, thereby improving the model's generalization ability under different recording conditions.

[0101] The relative value of the amplitude change between adjacent audio frames is a dimensionless relative change, not an absolute change. From a psychoacoustic perspective, the human ear's perception of changes in sound intensity follows the Weber-Fechner law, which states that the perceived quantity is proportional to the relative change of the stimulus, not the absolute change. Therefore, a speech emotion recognition scheme based on this dimensionless relative change can exhibit high robustness and sensitivity to different recording environments and subtle emotional changes, thus demonstrating extremely high adaptability to different recording environments and different speakers.

[0102] Furthermore, the process of obtaining the inter-frame amplitude feature sequence using the above method is very simple, completely avoiding complex operations such as Fast Fourier Transform, Mel filter bank filtering, or calculation of higher-order statistics. This computationally lightweight characteristic enables the speech emotion recognition scheme provided by the embodiments of this disclosure to run efficiently on embedded devices with limited computing resources, mobile terminals, or servers requiring high-concurrency processing, providing a solid technical foundation for real-time, low-power speech emotion recognition applications.

[0103] Figure 3 This is a flowchart of a voice emotion recognition method 1000 provided according to an exemplary embodiment of the present disclosure.

[0104] like Figure 3 As shown, in some other embodiments of this disclosure, step S3, based on the frame-level amplitude representation values ​​of multiple audio frames, determines the relative amplitude change values ​​of adjacent audio frames to construct an inter-frame amplitude feature sequence, and may include step 3-2: based on a preset lookup table, using the frame-level amplitude representation values ​​of adjacent audio frames as an index, the relative amplitude change values ​​are determined, wherein the lookup table is used to map the relationship between the frame-level amplitude representation values ​​of adjacent audio frames and the relative amplitude change values.

[0105] In this implementation, the feature calculation process performed during the actual runtime of the voice emotion recognition scheme is transformed into a one-time pre-calculation during the initialization phase, and completed only through an index lookup operation during runtime, completely eliminating the computational bottleneck in the feature extraction path. This enables the voice emotion recognition scheme to process voice signals with extremely high efficiency, providing technical support for achieving real-time, high-concurrency voice emotion recognition on low-power computing platforms such as embedded devices and mobile terminals.

[0106] Furthermore, lookup operations on lookup tables are essentially one or more memory accesses, which are much faster than the speed at which the processor performs arithmetic and logical operations. This design, which transforms the computational burden into a memory access burden, greatly increases the number of voice frames that can be processed per unit of time, thereby significantly reducing the overall system latency.

[0107] Optionally, step S-2, based on a preset lookup table and indexed by the frame-level amplitude representation values ​​of adjacent audio frames, further includes: linearly scaling the frame-level amplitude representation values ​​of adjacent audio frames to a first predetermined integer representation range to obtain an integer index; and searching the lookup table to determine the content corresponding to the feature value equal to the integer index as the relative value of the amplitude change. In this embodiment, the lookup table is indexed by multiple preset feature values, wherein all preset feature values ​​can be integers within the first predetermined integer representation range.

[0108] Before searching the lookup table, the frame-level amplitude representation values ​​of adjacent audio frames are also linearly scaled to a first predetermined integer representation range. This ensures that both the input and output to the lookup table are integers, and the entire feature extraction process is completed within the integer domain. This characteristic makes the scheme extremely suitable for implementation and optimization in hardware such as application-specific integrated circuits (ASICs) or FPGAs. It enables feature extraction with extremely low power consumption and extremely high throughput, greatly expanding the application scope of this invention in various edge computing and edge AI (Artificial Intelligence) scenarios.

[0109] Figure 4 This is a flowchart of a voice emotion recognition method 1000 provided according to an exemplary embodiment of the present disclosure.

[0110] Specifically, such as Figure 4 As shown, the speech emotion recognition method 1000 may further include constructing a lookup table, which may include: Step S01: Perform frame segmentation on the sample speech signal to obtain multiple sample audio frames.

[0111] Step S02: Determine the frame-level amplitude representation value of the sample audio frame.

[0112] Step S03: Linearly scale the frame-level amplitude representation value of the sample audio frame to a first predetermined integer representation range to generate multiple preset feature values.

[0113] Step S04: Determine the relative value of amplitude change corresponding to the frame-level amplitude representation value of the adjacent sample audio frame.

[0114] Step S05: Quantize the relative value of the amplitude change into an integer value within a second predetermined integer representation range.

[0115] Step S06: Construct a lookup table based on multiple preset feature values ​​and integer values.

[0116] Optionally, the lookup table can be determined before the actual execution of the speech emotion recognition scheme, or in other words, the lookup table can be determined during the initialization phase of the actual execution of the speech emotion recognition scheme. By pre-compiling and determining the lookup table once, subsequent speech emotion recognition can be based on this lookup table. In each subsequent run of the speech emotion recognition scheme, it can be completed solely through index lookup operations, completely eliminating the computational bottleneck on the feature extraction path.

[0117] It should be noted that the relevant content of the implementation methods described above for "performing frame segmentation processing on the speech signal to obtain multiple audio frames", "determining the frame-level amplitude representation value of the audio frame", and "determining the difference between the frame-level amplitude representation values ​​of adjacent audio frames and the sum between the frame-level amplitude representation values ​​of adjacent audio frames; and determining the relative value of amplitude change based on the ratio between the difference and the sum" can be fully or partially applied to the implementation methods described here for "performing frame segmentation processing on the sample speech signal, determining the frame-level amplitude representation value of the sample audio frame, and determining the relative value of amplitude change corresponding to the frame-level amplitude representation value of adjacent sample audio frames". Therefore, related or similar content will not be repeated.

[0118] The following describes the specific process of generating the lookup table using only one embodiment. Specifically, after obtaining the frame-level amplitude representation value shown in formula (1)... Then, it can be linearly scaled according to formula (3) to obtain the frame-level amplitude representation value. Linearly scaled to a first predetermined integer representation range, to obtain preset feature values. .

[0119] (3)

[0120] Alternatively, the upper limit of the first predetermined integer representation range can be determined based on the bit width parameter n. ; and based on a predetermined amplitude reference value And the bit width parameter n, linearly scales the frame-level amplitude representation value of the sample audio frame, where a predetermined amplitude reference value is used. The upper limit of the dynamic range of amplitude based on the sample speech signal is determined.

[0121] For example, a predetermined amplitude reference value The maximum absolute value of the 16-bit PCM (Pulse-Code Modulation) data of the sample speech signal is 32767. The bit width parameter n can take values ​​of 8, 10, or 12, etc. The bit width parameter n can be understood as a precision parameter, which can be a fixed value and does not need to be changed due to changes in the specific speech emotion recognition scenario.

[0122] After determining the preset feature values Then, based on formula (2), the difference between the frame-level amplitude representation value of the t-th audio frame and the frame-level amplitude representation value of the (t-1)-th audio frame can be used. And the sum of the frame-level amplitude representation value of the t-th audio frame and the frame-level amplitude representation value of the (t-1)-th audio frame. The ratio between them determines the relative value of the amplitude change between adjacent audio frames of the sample. .

[0123] Alternatively, formula (4) can also be based on the difference between the t-th preset eigenvalue and the (t-1)-th preset eigenvalue. And the sum between the t-th preset feature value and the (t-1)-th preset feature value. Determine the relative values ​​of amplitude changes between adjacent audio frames of the sample. .

[0124]

[0125] Referring to formulas (5) and (6), the relative value of the amplitude change can be... Quantized into integer values ​​within a second predetermined integer representation range Formula (5) represents the relative values ​​of amplitude changes between adjacent audio frames of a sample. Translate to The range. As shown in formula (6), in order to continue to use the original data precision to represent the feature values, the feature values ​​within the finite feature range [0,2] can be mapped to [0,2]. n Within the numerical space of ] . The round() operation is used for rounding.

[0126] (6)

[0127] In this way, a lookup table is constructed based on multiple preset feature values ​​and integer values, where the lookup table is indexed by the preset feature values ​​and contains integer values. The size of the lookup table can be [size missing]. × In the subsequent process of determining the relative value of the amplitude change, it is only necessary to know the t-th preset feature value a[t] and the (t-1)-th preset feature value a[t-1] in the integer index. The feature can be obtained by looking up the address, saving the calculation process.

[0128] Step S4

[0129] In some embodiments of this disclosure, the sentiment classification model may include traditional machine learning models and deep learning models. The traditional machine learning models may include support vector machines (SVM), K-nearest neighbors (KNN), etc.; the deep learning models may include convolutional neural networks (CNN), recurrent neural networks (RNN) and their variants such as long short-term memory networks (LSTM), and Transformer models, etc.

[0130] Traditional machine learning models have relatively low computational resource requirements, enabling rapid deployment and stable classification performance in small- to medium-scale scenarios with limited labeled data. Deep learning models can automatically learn deeper temporal patterns and complex nonlinear relationships from feature sequences, typically achieving higher recognition accuracy when large-scale labeled data is available. It should be noted that the appropriate sentiment classification model can be selected based on the specific application scenario, and this disclosure does not impose any limitations on it.

[0131] Optionally, during the training of the emotion classification model, data augmentation can be performed on the sample speech data to improve the model's generalization ability. For example, background noise can be added, the speech rate can be changed, and different recording environments can be simulated to improve the model's environmental robustness.

[0132] In addition, before training, the inter-frame amplitude feature sequences extracted from all sample speech samples need to be processed to the same length by padding or truncation to meet the requirement of fixed input for the model.

[0133] Optionally, the voice emotion recognition scheme provided by the embodiments of this disclosure can recognize basic emotions such as happiness, anger, sadness, fear, surprise, and disgust. It can also recognize more business-oriented emotions such as calmness, satisfaction (e.g., happiness), frustration (e.g., sadness), anger, and urgency in specific scenarios. Users only need to input raw voice, and the voice emotion recognition scheme provided by the embodiments of this disclosure can output structured, quantifiable, and understandable emotional information, greatly improving the intelligence level of human-computer interaction.

[0134] Optionally, in an implementation where the number of multiple audio frames is greater than a predetermined number and the multiple audio frames are divided into multiple groups, multiple inter-frame amplitude feature sequences corresponding to each of the multiple groups can be input into the emotion classification model; through the emotion classification model, based on the multiple inter-frame amplitude feature sequences corresponding to each of the multiple groups, multiple preliminary emotion classification results corresponding to each of the multiple groups can be identified; and through the emotion classification model, the multiple preliminary emotion classification results can be fused to identify the emotion classification result of the speech signal.

[0135] As an alternative, the fusion method can employ at least one of the following: majority voting and weighted averaging. The majority voting method can be understood as selecting the sentiment category that appears most frequently as the final result; the weighted averaging method can be understood as weighting the predicted probabilities of each group and taking the category corresponding to the highest probability.

[0136] In this implementation, by grouping and fusing long speech data, a fine characterization of emotional dynamics, effective suppression of local interference, and flexible adaptation to model limitations are achieved, thereby comprehensively improving the accuracy, robustness, and practicality of the speech emotion recognition system in real-world scenarios.

[0137] Therefore, the speech emotion recognition method provided by the embodiments of this disclosure determines the relative value of amplitude change between adjacent audio frames based on the frame-level amplitude representation values ​​of multiple audio frames. This relative value of amplitude change between adjacent audio frames is a dimensionless relative change. From a psychoacoustic perspective, the human ear's perception of changes in sound intensity follows the Weber-Fechner law, meaning that the perceived quantity is proportional to the relative change in the stimulus, rather than the absolute change. Therefore, the speech emotion recognition scheme based on this dimensionless relative change can exhibit high robustness and high sensitivity to different recording environments and subtle emotional changes, thus demonstrating extremely high adaptability to different recording environments and different speakers.

[0138] Furthermore, the process of obtaining the inter-frame amplitude feature sequence using the above method is very simple, completely avoiding complex operations such as Fast Fourier Transform, Mel filter bank filtering, or calculation of higher-order statistics. This computationally lightweight characteristic enables the speech emotion recognition scheme provided by the embodiments of this disclosure to run efficiently on embedded devices with limited computing resources, mobile terminals, or servers requiring high-concurrency processing, providing a solid technical foundation for real-time, low-power speech emotion recognition applications.

[0139] Figure 5 This is a block diagram of a voice emotion recognition device provided according to an exemplary embodiment of the present disclosure.

[0140] like Figure 5As shown, some embodiments of this disclosure provide a speech emotion recognition device 2000. The speech emotion recognition device 2000 may include a framing unit 100, an amplitude determination unit 200, a feature sequence generation unit 300, and a recognition unit 400. The framing unit 100 is configured to perform framing processing on the speech signal to obtain multiple frames; the amplitude determination unit 200 is configured to determine the frame-level amplitude representation value of the audio frames; the feature sequence generation unit 300 is configured to determine the relative amplitude change values ​​of adjacent audio frames based on the frame-level amplitude representation values ​​of the multiple audio frames to construct an inter-frame amplitude feature sequence, wherein the inter-frame amplitude feature sequence includes multiple relative amplitude change values ​​arranged in the temporal order of the audio frames; and the recognition unit 400 is configured to input the inter-frame amplitude feature sequence into an emotion classification model to identify the emotion classification result of the speech signal.

[0141] In some embodiments of this disclosure, the framing unit 100 is further configured to frame the speech signal with a fixed frame length and a fixed frame shift, resulting in multiple audio frames. The frame length can be understood as the duration of each short-time analysis window used to segment the speech signal. Exemplarily, the frame length can be a fixed value in the millisecond range, such as 25 milliseconds. Depending on the sampling rate, the frame length can be directly converted to the number of sampling points contained in each audio frame. The frame shift can be understood as the time interval between the start points of two adjacent short-time analysis windows in the framing process. Exemplarily, the frame shift can be a fixed value in the millisecond range, such as 25 milliseconds.

[0142] By using fixed frame lengths and fixed frame shifts for frame segmentation, a uniform audio frame sequence can be generated. This regular data structure is the input format desired by most sentiment classification models, greatly simplifying the model design and training process.

[0143] Optionally, the frame shift can be less than the frame length to allow overlap between adjacent audio frames. Since the changing patterns of speech emotion are continuous and gradual, key acoustic cues characterizing emotional transitions may appear precisely at the boundaries between two frames. Therefore, by introducing overlap between adjacent audio frames, it is ensured that the speech signal at any given time point is covered by at least one complete audio frame. This reduces the risk of losing emotional features due to frame boundary effects, allowing the generated inter-frame amplitude feature sequence to more completely and continuously depict the dynamic process of emotional change.

[0144] Furthermore, in some embodiments of this disclosure, the framing unit 100 is also configured to: supplement multiple audio frames based on the difference between the predetermined number and the number of multiple audio frames in response to the number of multiple audio frames being less than a predetermined number; and divide multiple audio frames into multiple groups based on the predetermined number in response to the number of multiple audio frames being greater than the predetermined number, wherein the predetermined number is determined based on the requirement of the emotion classification model for the length of the input inter-frame amplitude feature sequence.

[0145] In real-world applications of voice emotion recognition, the duration of user-input voice signals is not fixed. For example, in intelligent customer service interactions, a user's speech may be as short as less than a second, such as a brief "yes" or "no," or it may be as long as tens of seconds, such as describing a specific problem. However, most emotion classification models designed for optimal performance, especially deep learning-based models, typically require a fixed-length feature sequence in their input layer.

[0146] To address the mismatch between variable natural speech input and the model's fixed input requirements, a predetermined number of audio frames can be determined based on the length requirements of the inter-frame amplitude feature sequence of the input from the emotion classification model. Based on this predetermined number, it can be analyzed whether the number of multiple audio frames obtained after frame segmentation processing is sufficient to ensure that the audio inter-frame amplitude feature sequence generated from them meets the model's fixed input requirements.

[0147] In response to the number of multiple audio frames being less than a predetermined number, the multiple audio frames are supplemented based on the difference between the predetermined number and the number of multiple audio frames; and in response to the number of multiple audio frames being greater than a predetermined number, the multiple audio frames are divided into multiple groups based on the predetermined number.

[0148] In some implementations, after dividing the multiple audio frames into multiple groups based on the predetermined number in response to the number of multiple audio frames being greater than a predetermined number, each group may include a predetermined number of audio frames.

[0149] In other embodiments, after dividing the multiple audio frames into multiple groups based on a predetermined number in response to the number of multiple audio frames exceeding a predetermined number, the number of audio frames included in the multiple groups may not be equal. For example, the number of audio frames in at least one of the multiple groups may be less than the predetermined number. In this embodiment, the group with fewer than the predetermined number of audio frames (hereinafter referred to as the first group) may be supplemented. For example, frames with a frame-level amplitude representation value of zero may be added to the end of the multiple audio frames in the first group; or, the multiple audio frames in the first group may be repeated until the number of repeated multiple audio frames in the first group reaches or exceeds the predetermined number, and multiple audio frames equal to the predetermined number may be extracted from the entire audio frames in the expanded first group from front to back for subsequent speech emotion recognition.

[0150] In this way, the first group with fewer than the predetermined number of audio frames, after the above supplementation, can be used together with the remaining groups with the predetermined number of audio frames in the multiple groups for speech emotion recognition. Alternatively, the groups with fewer than the predetermined number of audio frames formed after grouping can be deleted, and only the groups with the predetermined number of audio frames in the multiple groups can be used for speech emotion recognition.

[0151] This implementation can seamlessly process all voice signals, from very short commands to lengthy narratives, so that the voice emotion recognition scheme provided by the present disclosure is no longer limited to regular-length voice samples in the laboratory, but can be directly applied to complex and varied voice input scenarios in the real world, greatly enhancing the practical value and commercial potential of the technology.

[0152] Optionally, in response to the number of multiple audio frames being less than a predetermined number, frames with a frame-level amplitude representation value of zero can be padded to the end of the multiple audio frames based on the difference between the predetermined number and the number of multiple audio frames. Alternatively, in response to the number of multiple audio frames being less than a predetermined number, multiple audio frames can be repeated based on the difference between the predetermined number and the number of multiple audio frames until the number of repeated audio frames equals the predetermined number. By performing a "padded" operation on the multiple audio frames after frame segmentation, it is ensured that the inter-frame amplitude feature sequence of the input model contains complete contextual information, avoiding model misjudgment due to insufficient data.

[0153] For example, in response to the number of multiple audio frames being less than a predetermined number, the process of repeating multiple audio frames based on the difference between the predetermined number and the number of multiple audio frames until the number of repeated audio frames equals the predetermined number can be implemented by: copying all the obtained audio frames as the original sequence and concatenating the copied sequence after the original sequence to expand the original sequence. If the total number of expanded frames is still less than the predetermined number, the above copying and concatenation operation is repeated until the total number of expanded frames reaches or exceeds the predetermined number. If the total number of expanded frames exceeds the predetermined number, multiple audio frames equal to the predetermined number can be extracted from the total number of expanded audio frames from front to back for subsequent speech emotion recognition.

[0154] Alternatively, in response to the number of multiple audio frames being less than a predetermined number, multiple audio frames can be repeated based on the difference between the predetermined number and the number of multiple audio frames until the number of repeated audio frames equals the predetermined number. This can be implemented by: using all the obtained audio frames as the original sequence, selecting multiple audio frames arranged sequentially in the original sequence (hereinafter referred to as repeating units) for repetition, and then concatenating the copied sequence after the original sequence to expand the original sequence. If the total number of expanded frames is still less than the predetermined number, the above copying and concatenation operation is repeated until the total number of expanded frames reaches or exceeds the predetermined number. If the total number of expanded frames exceeds the predetermined number, multiple audio frames equal to the predetermined number can be extracted from the entire expanded audio frame sequence from front to back for subsequent speech emotion recognition. Optionally, the number of repeating units can be determined according to actual needs. For example, the number of repeating units can be calculated based on the difference between the predetermined number and the number of multiple audio frames obtained after frame processing, or the number of repeating units can be set to a fixed value.

[0155] "Based on the difference between a predetermined number and the number of multiple audio frames, repeat multiple audio frames until the number of repeated audio frames equals the predetermined number" can be understood as a strategy of repeating the entire speech. This strategy can maintain the periodic pattern of the original emotion to the maximum extent without introducing invalid information, thereby improving the confidence and accuracy of short speech emotion classification.

[0156] In some embodiments of this disclosure, in response to the number of multiple audio frames being greater than a predetermined number, dividing the multiple audio frames into multiple groups may include: dividing the multiple audio frames with a fixed length and a fixed shift distance corresponding to the predetermined number to obtain multiple groups, wherein the fixed shift distance is less than the fixed length so that there is an overlap between adjacent groups.

[0157] By performing a "segmentation" operation on multiple audio frames after frame-segmentation, long speech can be decomposed into multiple segments that match the model input, ensuring that the emotion classification model can perform detailed analysis on each local segment. This prevents the multiple mixed emotions or emotional evolution processes that may exist in long speech from being averaged out, allowing the system to capture richer and more nuanced emotional dynamics. In addition, the overlapping parts between adjacent groups ensure a smooth transition of speech signal features.

[0158] In some embodiments of this disclosure, the amplitude determination unit 200 is further configured to: determine an amplitude correlation value based on the instantaneous amplitude values ​​of multiple sampling points within an audio frame; and determine a frame-level amplitude representation value based on a statistic of the amplitude correlation value, wherein the statistic is determined by any of the following class values: the average of the amplitude correlation values ​​of multiple sampling points; the weighted average of the amplitude correlation values ​​of multiple sampling points; and the maximum value among the amplitude correlation values ​​of multiple sampling points.

[0159] A frame-level amplitude representation can be understood as a single scalar representing the overall signal amplitude or energy intensity within a single audio frame. The calculation of this value forms the basis for subsequent analysis of relative changes between frames, and its determination process may include: obtaining the amplitude correlation value for each sampling point; and calculating representative frame-level statistics based on these amplitude correlation values.

[0160] The amplitude correlation value of a sampling point can be understood as a numerical value derived from the original sampling point data and used to characterize the instantaneous intensity of the signal. Optionally, the amplitude correlation value of a sampling point can be the instantaneous amplitude of the sampling point. Optionally, the amplitude correlation value of a sampling point can also be the absolute value of the instantaneous amplitude of the sampling point. Optionally, the amplitude correlation value of a sampling point can also be the square of the instantaneous amplitude of the sampling point. In other words, the amplitude correlation value of a sampling point can be either the instantaneous amplitude of the sampling point or a non-negative value obtained by mathematical transformation of the instantaneous amplitude value.

[0161] Optionally, the statistic is not limited to the above implementation method that determines it by the sum of the instantaneous amplitude values ​​of all sampling points within the audio frame, but can also be determined by any of the following class values: the average value of the amplitude correlation values ​​of multiple sampling points; the weighted average value of the amplitude correlation values ​​of multiple sampling points; and the maximum value among the amplitude correlation values ​​of multiple sampling points.

[0162] The embodiments of this disclosure compress an audio frame, which may contain hundreds of sampling points, into a representative scalar during the determination of the frame-level amplitude representation value of an audio frame, greatly reducing the amount of data in subsequent processing. Furthermore, using statistical measures to obtain the frame-level amplitude representation value effectively smooths out interference caused by transient noise or minute fluctuations within the frame, resulting in a frame-level amplitude representation value that is more representative and stable than any isolated sampling point.

[0163] Alternatively, in some embodiments of this disclosure, the feature sequence generation unit 300 is further configured to: determine the difference between frame-level amplitude representation values ​​of adjacent audio frames and the sum between frame-level amplitude representation values ​​of adjacent audio frames; and determine the relative value of amplitude change based on the ratio between the difference and the sum.

[0164] In this embodiment, the relative value of the amplitude change of adjacent audio frames is divided by the amplitude of adjacent audio frames, which cancels out the overall energy difference of the speech signal. Therefore, it is not sensitive to factors such as volume changes, microphone sensitivity differences, and recording distance, thereby improving the model's generalization ability under different recording conditions.

[0165] The relative value of the amplitude change between adjacent audio frames is a dimensionless relative change, not an absolute change. From a psychoacoustic perspective, the human ear's perception of changes in sound intensity follows the Weber-Fechner law, which states that the perceived quantity is proportional to the relative change of the stimulus, not the absolute change. Therefore, a speech emotion recognition scheme based on this dimensionless relative change can exhibit high robustness and sensitivity to different recording environments and subtle emotional changes, thus demonstrating extremely high adaptability to different recording environments and different speakers.

[0166] Furthermore, the process of obtaining the inter-frame amplitude feature sequence described above is very simple, completely avoiding complex operations such as Fast Fourier Transform, Mel filter bank filtering, or calculation of higher-order statistics. This computationally lightweight characteristic enables the speech emotion recognition scheme provided by the embodiments of this disclosure to run efficiently on embedded devices with limited computing resources, mobile terminals, or servers requiring high-concurrency processing, providing a solid technical foundation for real-time, low-power speech emotion recognition applications.

[0167] As an alternative, in some embodiments of this disclosure, the feature sequence generation unit 300 is further configured to: determine the relative value of amplitude change based on a preset lookup table, using the frame-level amplitude representation value of adjacent audio frames as an index, wherein the lookup table is used to map the relationship between the frame-level amplitude representation value of adjacent audio frames and the relative value of amplitude change.

[0168] In this implementation, the feature calculation process performed during the actual runtime of the voice emotion recognition scheme is transformed into a one-time pre-calculation during the initialization phase, and completed only through an index lookup operation during runtime, completely eliminating the computational bottleneck in the feature extraction path. This enables the voice emotion recognition scheme to process voice signals with extremely high efficiency, providing technical support for achieving real-time, high-concurrency voice emotion recognition on low-power computing platforms such as embedded devices and mobile terminals.

[0169] Furthermore, lookup operations on lookup tables are essentially one or more memory accesses, which are much faster than the speed at which the processor performs arithmetic and logical operations. This design, which transforms the computational burden into a memory access burden, greatly increases the number of voice frames that can be processed per unit of time, thereby significantly reducing the overall system latency.

[0170] Optionally, the feature sequence generation unit 300 is further configured to: linearly scale the frame-level amplitude representation values ​​of adjacent audio frames to a first predetermined integer representation range to obtain an integer index; and search a lookup table to determine the content corresponding to the feature value equal to the integer index as the relative value of amplitude change. In this embodiment, the lookup table is indexed by multiple preset feature values, wherein all preset feature values ​​can be integers within the first predetermined integer representation range.

[0171] Before searching the lookup table, the frame-level amplitude representation values ​​of adjacent audio frames are linearly scaled to a first predetermined integer representation range. This ensures that both the input and output to the lookup table are integers, and the entire feature extraction process is completed within the integer domain. This characteristic makes the scheme extremely suitable for implementation and optimization in hardware such as application-specific integrated circuits (ASICs) or FPGAs. It enables feature extraction with extremely low power consumption and extremely high throughput, greatly expanding the application scope of this invention in various edge computing and edge AI scenarios.

[0172] Furthermore, the feature sequence generation unit 300 is also configured to: construct a lookup table, which may include: performing frame segmentation processing on the sample speech signal to obtain multiple sample audio frames; determining the frame-level amplitude representation value of the sample audio frame; linearly scaling the frame-level amplitude representation value of the sample audio frame to a first predetermined integer representation range to generate multiple preset feature values; determining the relative amplitude change value corresponding to the frame-level amplitude representation value of adjacent sample audio frames; quantizing the relative amplitude change value into an integer value within a second predetermined integer representation range; and constructing a lookup table based on the multiple preset feature values ​​and the integer value.

[0173] Optionally, the lookup table can be determined before the actual execution of the speech emotion recognition scheme, or in other words, the lookup table can be determined during the initialization phase of the actual execution of the speech emotion recognition scheme. By pre-compiling and determining the lookup table once, subsequent speech emotion recognition can be based on this lookup table. In each subsequent run of the speech emotion recognition scheme, it can be completed solely through index lookup operations, completely eliminating the computational bottleneck on the feature extraction path.

[0174] Furthermore, the feature sequence generation unit 300 is configured to: determine an upper limit of a first predetermined integer representation range based on a bit width parameter; and linearly scale the frame-level amplitude representation value of the sample audio frame based on a predetermined amplitude reference value and a bit width parameter, wherein the predetermined amplitude reference value is determined based on an upper limit of the amplitude dynamic range of the sample speech signal.

[0175] Optionally, the feature sequence generation unit 300 is further configured to: determine the difference between the frame-level amplitude representation values ​​of adjacent sample audio frames and the sum between the frame-level amplitude representation values ​​of adjacent sample audio frames; and determine the relative value of amplitude change based on the ratio between the difference and the sum.

[0176] Optionally, the feature sequence generation unit 300 is further configured to construct a lookup table using multiple preset feature values ​​as indexes and integer values ​​as content. In this way, the lookup table is constructed based on multiple preset feature values ​​and integer values, where the lookup table uses multiple preset feature values ​​as indexes and integer values ​​as content. In the subsequent process of determining the relative value of the magnitude change, only two adjacent preset feature values ​​in the integer index are known; the feature can be obtained by looking up the address in the table, eliminating the need for calculation.

[0177] In some embodiments of this disclosure, the sentiment classification model may include traditional machine learning models and deep learning models, wherein traditional machine learning models may include SVM, KNN, etc.; and deep learning models may include CNN, RNN, LSTM, and Transformer, etc.

[0178] Traditional machine learning models have relatively low computational resource requirements, enabling rapid deployment and stable classification performance in small- to medium-scale scenarios with limited labeled data. Deep learning models can automatically learn deeper temporal patterns and complex nonlinear relationships from feature sequences, typically achieving higher recognition accuracy when large-scale labeled data is available. It should be noted that the appropriate sentiment classification model can be selected based on the specific application scenario, and this disclosure does not impose any limitations on it.

[0179] Optionally, during the training of the emotion classification model, data augmentation can be performed on the sample speech data to improve the model's generalization ability. For example, background noise can be added, the speech rate can be changed, and different recording environments can be simulated to improve the model's environmental robustness.

[0180] In addition, before training, the inter-frame amplitude feature sequences extracted from all sample speech samples need to be processed to the same length by padding or truncation to meet the requirement of fixed input for the model.

[0181] Optionally, the voice emotion recognition scheme provided by the embodiments of this disclosure can recognize basic emotions such as happiness, anger, sadness, fear, surprise, and disgust. It can also recognize more business-oriented emotions such as calmness, satisfaction (e.g., happiness), frustration (e.g., sadness), anger, and urgency in specific scenarios. Users only need to input raw voice, and the voice emotion recognition scheme provided by the embodiments of this disclosure can output structured, quantifiable, and understandable emotional information, greatly improving the intelligence level of human-computer interaction.

[0182] Optionally, in an implementation where the number of multiple audio frames is greater than a predetermined number and the multiple audio frames are divided into multiple groups, multiple inter-frame amplitude feature sequences corresponding to each of the multiple groups can be input into the emotion classification model; through the emotion classification model, based on the multiple inter-frame amplitude feature sequences corresponding to each of the multiple groups, multiple preliminary emotion classification results corresponding to each of the multiple groups can be identified; and through the emotion classification model, the multiple preliminary emotion classification results can be fused to identify the emotion classification result of the speech signal.

[0183] As an alternative, the fusion method can employ at least one of the following: majority voting and weighted averaging. The majority voting method can be understood as selecting the sentiment category that appears most frequently as the final result; the weighted averaging method can be understood as weighting the predicted probabilities of each group and taking the category corresponding to the highest probability.

[0184] In this implementation, by grouping and fusing long speech data, a fine characterization of emotional dynamics, effective suppression of local interference, and flexible adaptation to model limitations are achieved, thereby comprehensively improving the accuracy, robustness, and practicality of the speech emotion recognition system in real-world scenarios.

[0185] Therefore, the speech emotion recognition device provided by the embodiments of this disclosure determines the relative value of amplitude change between adjacent audio frames based on the frame-level amplitude representation values ​​of multiple audio frames. This relative value of amplitude change between adjacent audio frames is a dimensionless relative change. From a psychoacoustic perspective, the human ear's perception of changes in sound intensity follows the Weber-Fechner law, meaning that the perceived quantity is proportional to the relative change in the stimulus, rather than the absolute change. Therefore, the speech emotion recognition scheme based on this dimensionless relative change can exhibit high robustness and sensitivity to different recording environments and subtle emotional changes, thus possessing extremely high adaptability to different recording environments and different speakers.

[0186] Furthermore, the process of obtaining the inter-frame amplitude feature sequence described above is very simple, completely avoiding complex operations such as Fast Fourier Transform, Mel filter bank filtering, or calculation of higher-order statistics. This computationally lightweight characteristic enables the speech emotion recognition scheme provided by the embodiments of this disclosure to run efficiently on embedded devices with limited computing resources, mobile terminals, or servers requiring high-concurrency processing, providing a solid technical foundation for real-time, low-power speech emotion recognition applications.

[0187] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0188] Figure 6 A schematic block diagram of an example electronic device 3000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobility methods, such as personal digital processing, cellular phones, smartphones, wearable devices, and other similar computing methods. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0189] like Figure 6 As shown, device 3000 includes a computing unit 301, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 302 or a computer program loaded from storage unit 308 into random access memory (RAM) 303. The RAM 303 may also store various programs and data required for the operation of device 3000. The computing unit 301, ROM 302, and RAM 303 are interconnected via bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0190] Multiple components in device 3000 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of monitors, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows device 3000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0191] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various means and processes described above, such as the speech emotion recognition method. For example, in some embodiments, the speech emotion recognition method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on device 3000 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of the speech emotion recognition method described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the speech emotion recognition method by any other suitable means (e.g., by means of firmware).

[0192] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input method, and at least one output method, and transferring data and instructions to the storage system, the at least one input method, and the at least one output method.

[0193] Program code for implementing the apparatus of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing method, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0194] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, method, or apparatus. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, methods, or apparatuses, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0195] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display method for showing information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing method (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of methods can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0196] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0197] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0198] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, apparatuses, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0199] Various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technological improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0200] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0201] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A speech emotion recognition method, comprising: The speech signal is segmented into frames to obtain multiple audio frames; Determine the frame-level amplitude representation value of the audio frame; Based on the frame-level amplitude representation values ​​of multiple audio frames, the relative values ​​of amplitude changes of adjacent audio frames are determined to construct an inter-frame amplitude feature sequence, wherein the inter-frame amplitude feature sequence includes multiple relative values ​​of amplitude changes arranged in the temporal order of the audio frames. as well as The amplitude feature sequence between audio frames is input into the emotion classification model to identify the emotion classification result of the speech signal.

2. The method according to claim 1, wherein, The process of segmenting the speech signal into frames to obtain multiple audio frames includes: The speech signal is divided into frames with a fixed frame length and a fixed frame shift to obtain multiple audio frames.

3. The method according to claim 2, wherein, The frame shift is less than the frame length so that there is an overlap between the adjacent audio frames.

4. The method according to claim 1, wherein, The determination of the frame-level amplitude representation value of the audio frame includes: An amplitude correlation value is determined based on the instantaneous amplitude values ​​of multiple sampling points within the audio frame; and Based on the statistics of the amplitude correlation values, the frame-level amplitude representation value is determined. The statistic is determined by any of the following class values: The average of the amplitude correlation values ​​of the multiple sampling points; The weighted average of the amplitude correlation values ​​of the multiple sampling points; and The maximum value among the amplitude correlation values ​​of the plurality of sampling points.

5. The method according to claim 1, wherein, The process of segmenting the speech signal into frames to obtain multiple audio frames includes: In response to the fact that the number of the plurality of audio frames is less than a predetermined number, the plurality of audio frames are supplemented based on the difference between the predetermined number and the number of the plurality of audio frames; and In response to the fact that the number of multiple audio frames exceeds a predetermined number, the multiple audio frames are divided into multiple groups based on the predetermined number. The predetermined quantity is determined based on the length requirement of the inter-frame amplitude feature sequence of the input by the emotion classification model.

6. The method according to claim 5, wherein, In response to the fact that the number of multiple audio frames is less than a predetermined number, the multiple audio frames are supplemented based on the difference between the predetermined number and the number of multiple audio frames, including: At the end of the plurality of audio frames, insert frames with a frame-level amplitude representation value of zero; or Repeat the audio frames multiple times until the number of repeated audio frames equals the predetermined number.

7. The method according to claim 5, wherein, In response to the number of multiple audio frames exceeding a predetermined number, the multiple audio frames are divided into multiple groups, each group comprising the predetermined number of audio frames, including: The audio frames are divided into multiple groups by using a fixed length and a fixed shift distance corresponding to the predetermined number. Wherein, the fixed displacement distance is less than the fixed length, so that there is an overlap between adjacent groups.

8. The method according to claim 7, wherein, The step of inputting the inter-frame amplitude feature sequence into the emotion classification model to identify the emotion classification result of the speech signal includes: The inter-frame amplitude feature sequences corresponding to the multiple groups are input into the sentiment classification model; Using the sentiment classification model, based on the inter-frame amplitude feature sequences corresponding to the multiple groups, multiple preliminary sentiment classification results corresponding to the multiple groups are identified; and The emotion classification model is used to fuse multiple preliminary emotion classification results to identify the emotion classification result of the speech signal.

9. The method according to claim 1, wherein, The step of determining the relative amplitude changes of adjacent audio frames based on the frame-level amplitude representation values ​​of multiple audio frames to construct an inter-frame amplitude feature sequence includes: Determine the difference between the frame-level amplitude representation values ​​of the adjacent audio frames and the sum of the frame-level amplitude representation values ​​of the adjacent audio frames; and The relative value of the magnitude change is determined based on the ratio between the difference and the sum.

10. The method according to claim 1, wherein, The step of determining the relative amplitude changes of adjacent audio frames based on the frame-level amplitude representation values ​​of multiple audio frames to construct an inter-frame amplitude feature sequence further includes: Based on a preset lookup table, the relative value of the amplitude change is determined using the frame-level amplitude representation value of the adjacent audio frames as an index. The lookup table is used to map the relationship between the frame-level amplitude representation value and the relative value of amplitude change of the adjacent audio frames.

11. The method according to claim 10, wherein, The lookup table is indexed by multiple preset feature values, wherein each preset feature value is an integer within a first predetermined integer range. The step of determining the relative value of the amplitude change based on a preset lookup table, using the frame-level amplitude representation value of the adjacent audio frames as an index, includes: The frame-level amplitude representation values ​​of the adjacent audio frames are linearly scaled to the first predetermined integer representation range to obtain an integer index; and The lookup table is searched, and the content corresponding to the feature value that is equal to the integer index is determined as the relative value of the amplitude change.

12. The method according to claim 1, wherein, The method further includes constructing a lookup table based on the following steps: The sample speech signal is segmented into frames to obtain multiple sample audio frames; Determine the frame-level amplitude representation value of the sample audio frame; The frame-level amplitude representation value of the sample audio frame is linearly scaled to a first predetermined integer representation range to generate multiple preset feature values; Determine the relative value of the amplitude change corresponding to the frame-level amplitude representation value of the adjacent sample audio frames; The relative value of the amplitude change is quantized into an integer value within a second predetermined integer representation range; and The lookup table is constructed based on multiple preset feature values ​​and integer values.

13. The method according to claim 12, wherein, The linear scaling of the frame-level amplitude representation value of the sample audio frame to a first predetermined integer representation range generates multiple preset feature values, including: The upper limit of the first predetermined integer representation range is determined based on the bit width parameter; and Based on a predetermined amplitude reference value and the bit width parameter, the frame-level amplitude representation value of the sample audio frame is linearly scaled. The predetermined amplitude reference value is determined based on the upper limit of the dynamic range of the amplitude of the sample speech signal.

14. The method according to claim 12, wherein, Determining the relative value of the amplitude change corresponding to the frame-level amplitude representation value of adjacent sample audio frames includes: Determine the difference between the frame-level amplitude representation values ​​of the adjacent sample audio frames and the sum of the frame-level amplitude representation values ​​of the adjacent sample audio frames; and The relative value of the magnitude change is determined based on the ratio between the difference and the sum.

15. The method according to claim 12, wherein, The construction of the lookup table based on multiple preset feature values ​​and the integer value includes: The lookup table is constructed using multiple preset feature values ​​as indexes and integer values ​​as content.

16. A voice emotion recognition device, comprising: The framing unit is configured to process the speech signal into frames to obtain multiple frames; An amplitude determination unit is configured to determine the frame-level amplitude representation value of the audio frame; A feature sequence generation unit is configured to determine relative values ​​of amplitude changes between adjacent audio frames based on the frame-level amplitude representation values ​​of multiple audio frames, in order to construct an inter-frame amplitude feature sequence, wherein the inter-frame amplitude feature sequence includes multiple relative values ​​of amplitude changes arranged in the temporal order of the audio frames. as well as The recognition unit is configured to input the inter-frame amplitude feature sequence of the audio into the emotion classification model and identify the emotion classification result of the speech signal.

17. An electronic device comprising: A processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the speech emotion recognition method as described in any one of claims 1 to 15.

18. A non-volatile computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the speech emotion recognition method according to any one of claims 1 to 15.

19. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the voice emotion recognition method according to any one of claims 1 to 15.

Images