Speech emotion classification method and device based on capsule network and attention mechanism

Abstract

The application relates to the technical field of data processing, and discloses a speech emotion classification method and device based on a capsule network and an attention mechanism, which comprises the following steps: processing a speech signal to obtain an acoustic feature sequence; inputting the acoustic feature sequence into a bidirectional long short-term memory network to respectively extract forward and backward time sequence dependent features, and obtaining a fusion time sequence feature sequence according to the features; inputting the fusion time sequence feature sequence into a self-attention mechanism layer to obtain a focused feature sequence; inputting the focused feature sequence into a capsule network to extract local spatial features through a feature extraction layer in the capsule network; converting the local spatial features into a plurality of primary capsule vectors through a capsule encoding layer in the capsule network; transmitting the plurality of primary capsule vectors to a category capsule layer through a dynamic routing mechanism in the capsule network to obtain output vectors of each category capsule; and determining the emotion category of the speech signal according to the lengths of the output vectors. The application can be applied to the speech emotion classification scene in the field of financial science and technology.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and apparatus for speech emotion classification based on capsule networks and attention mechanisms, as well as a storage medium and computer equipment. Background Technology

[0002] Speech emotion recognition technology aims to perceive and identify a speaker's emotional state by analyzing the acoustic features of speech signals, and is an important research direction in the fields of artificial intelligence and affective computing. In recent years, with the rapid development of deep learning technology and the increasing prevalence of voice interaction scenarios, speech emotion recognition has shown broad application prospects in fields such as fintech. In the fintech field, speech emotion recognition is widely used in intelligent customer service and call center systems of financial institutions such as banks and insurance companies. For example, by analyzing customer call voices in real time to identify emotional states, it improves the efficiency of customer service systems and customer satisfaction; insurance customer service systems extract voiceprint features to calculate emotional polarity values, achieving accurate quantification of customer emotions and service urgency. Therefore, speech emotion recognition has become a key technological bridge connecting the upgrade of fintech services, possessing significant research value and application demand.

[0003] Despite the immense application potential of speech emotion recognition technology in the aforementioned fields, existing technologies still face numerous unresolved issues. First, traditional speech emotion recognition methods primarily rely on manually designed acoustic features (such as fundamental frequency, energy, and Mel-frequency cepstral coefficients) combined with shallow classifiers. These methods struggle to fully extract the deep emotional information contained within speech signals, resulting in limited accuracy. Second, while existing deep learning methods based on convolutional neural networks (CNNs) or recurrent neural networks (RNNs) have made some progress, the pooling operation in traditional CNNs loses the relative positional relationships between features, and standard RNNs struggle to effectively capture long-range temporal dependencies. More critically, existing models exhibit significant performance degradation and insufficient robustness when handling complex real-world environments (such as background noise, different speaking styles, and individual differences). Furthermore, existing models generally suffer from low discriminative power of emotional features, insensitivity to subtle emotional changes, and instability in multi-class emotion classification tasks, failing to meet the demands for high-precision and robust speech emotion classification in real-world business scenarios. Summary of the Invention

[0004] In view of this, this application provides a speech emotion classification method and device, storage medium, and computer equipment based on capsule networks and attention mechanisms. By constructing a three-level feature extraction architecture consisting of a bidirectional long short-term memory network, a self-attention mechanism, and a one-dimensional capsule network cascaded in sequence, it solves the technical problems of difficulty in effectively capturing long-distance temporal dependencies of speech signals, loss of feature spatial location information caused by traditional convolutional neural network pooling operations, and low discrimination of emotion features due to insufficient ability of existing models to focus on key emotion frames.

[0005] According to one aspect of this application, a speech emotion classification method based on capsule networks and attention mechanisms is provided, comprising: Acquire the speech signal to be classified, perform frame segmentation on the speech signal, extract the acoustic features of each frame of speech, and concatenate the acoustic features of each frame in the feature dimension to form an acoustic feature sequence. The acoustic feature sequence is input into a bidirectional long short-term memory network. Forward temporal dependency features and backward temporal dependency features are extracted through the forward long short-term memory unit and the backward long short-term memory unit, respectively. The extracted results are then concatenated to obtain a fused temporal feature sequence. The fused temporal feature sequence is input into the self-attention mechanism layer, and the correlation of each fused temporal feature vector in the fused temporal feature sequence is calculated by the self-attention mechanism layer. Based on the correlation calculation result, a focused feature sequence is constructed. The focused feature sequence is input into the capsule network. The feature extraction layer of the capsule network performs a one-dimensional convolution operation on the focused feature sequence along the time dimension of the speech signal to extract local spatial features. The capsule encoding layer of the capsule network converts the local spatial features into multiple primary capsule vectors. The dynamic routing mechanism of the capsule network passes the multiple primary capsule vectors to the category capsule layer of the capsule network to obtain the output vector of each category capsule. The emotion category of the speech signal is determined based on the magnitude of the output vector of each category of capsule.

[0006] According to another aspect of this application, a speech emotion classification device based on capsule networks and attention mechanisms is provided, comprising: The feature extraction module is used to acquire the speech signal to be classified, perform frame-by-frame processing on the speech signal, extract the acoustic features of each frame of speech, and concatenate the acoustic features corresponding to each frame in the feature dimension to form an acoustic feature sequence. The first processing module is used to input the acoustic feature sequence into a bidirectional long short-term memory network, extract forward temporal dependency features and backward temporal dependency features through the forward long short-term memory unit and the backward long short-term memory unit, respectively, and concatenate the extraction results to obtain a fused temporal feature sequence. The second processing module is used to input the fused temporal feature sequence into the self-attention mechanism layer, perform correlation calculation on each fused temporal feature vector in the fused temporal feature sequence through the self-attention mechanism layer, and construct a focused feature sequence based on the correlation calculation result; The third processing module is used to input the focused feature sequence into the capsule network, and to perform a one-dimensional convolution operation on the focused feature sequence along the time dimension of the speech signal through the feature extraction layer of the capsule network to extract local spatial features; to convert the local spatial features into multiple primary capsule vectors through the capsule encoding layer of the capsule network; and to pass the multiple primary capsule vectors to the category capsule layer of the capsule network through the dynamic routing mechanism of the capsule network to obtain the output vector of each category capsule. The classification module is used to determine the emotion category of the speech signal based on the magnitude of the output vector of each category capsule.

[0007] According to another aspect of this application, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described speech emotion classification method based on capsule networks and attention mechanisms.

[0008] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described speech emotion classification method based on capsule networks and attention mechanisms.

[0009] Using the above technical solutions, this application provides a speech emotion classification method, device, storage medium, and computer equipment based on capsule networks and attention mechanisms. First, it directly uses the frame-by-frame concatenated acoustic feature sequence as network input, and automatically learns deep emotional representations through hierarchical abstraction of multi-layer networks, avoiding the subjectivity and incompleteness of manual feature design. Second, it introduces a bidirectional long short-term memory network, which captures and merges forward and backward contextual information through two long short-term memory units, so that the output features of each frame can integrate global temporal information, effectively solving the gradient vanishing problem in long sequences. It adopts an improved one-dimensional capsule network, using a dynamic routing mechanism to replace pooling operations, and retains the spatial positional relationship and high-level hierarchical structure between speech features in a vectorized capsule representation, enabling the model to capture the complete mapping from local acoustic features to the overall emotion category. A self-attention mechanism layer is introduced between temporal feature extraction and the capsule network, which automatically focuses on key speech segments rich in emotional information by calculating inter-frame correlation, suppressing the interference of background noise and neutral expressions, making the model more sensitive to subtle emotional changes. The embodiments of this application significantly improve the accuracy and robustness of speech emotion classification by constructing a three-level cascaded architecture of "bidirectional long short-term memory network - self-attention mechanism - one-dimensional capsule network".

[0010] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 The diagram illustrates a flowchart of a speech emotion classification method based on capsule networks and attention mechanisms provided in an embodiment of this application. Figure 2 A flowchart illustrating another speech emotion classification method based on capsule networks and attention mechanisms provided in an embodiment of this application is shown. Figure 3 A schematic diagram of a capsule network structure provided in an embodiment of this application is shown; Figure 4 This paper illustrates a schematic diagram of a speech emotion classification device based on capsule networks and attention mechanisms provided in an embodiment of this application. Figure 5 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0012] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0013] This embodiment provides a speech emotion classification method based on capsule networks and attention mechanisms, such as Figure 1 As shown, the method includes: Step 101: Obtain the speech signal to be classified, perform frame segmentation on the speech signal, extract the acoustic features of each frame of speech, and splice the acoustic features corresponding to each frame in the feature dimension to form an acoustic feature sequence.

[0014] Step 102: Input the acoustic feature sequence into a bidirectional long short-term memory network, extract forward temporal dependency features and backward temporal dependency features through the forward long short-term memory unit and the backward long short-term memory unit, respectively, and concatenate the extraction results to obtain a fused temporal feature sequence.

[0015] Step 103: Input the fused temporal feature sequence into the self-attention mechanism layer, and perform correlation calculation on each fused temporal feature vector in the fused temporal feature sequence through the self-attention mechanism layer. Based on the correlation calculation result, construct the focused feature sequence.

[0016] Step 104: Input the focused feature sequence into the capsule network. The feature extraction layer of the capsule network performs a one-dimensional convolution operation on the focused feature sequence along the time dimension of the speech signal to extract local spatial features. The capsule encoding layer of the capsule network converts the local spatial features into multiple primary capsule vectors. The dynamic routing mechanism of the capsule network passes the multiple primary capsule vectors to the category capsule layer of the capsule network to obtain the output vector of each category capsule.

[0017] Step 105: Determine the emotion category of the speech signal based on the magnitude of the output vector of each category of capsule.

[0018] This application provides a speech emotion classification method based on capsule networks and attention mechanisms. First, the acquired speech signal to be classified is preprocessed. Since the original speech signal is a continuous waveform in the time domain, it cannot be directly input into a neural network for processing. Therefore, the speech signal can be segmented into several short frames, and multi-dimensional acoustic parameters are extracted from each frame, such as Mel-frequency cepstral coefficients, fundamental frequency, short-time energy, and zero-crossing rate. These parameters characterize the acoustic properties of speech from different dimensions such as frequency and energy. The multiple acoustic parameters extracted from each frame are horizontally concatenated along the feature dimension to form the acoustic features of that frame. The acoustic features of all frames are arranged in chronological order to form an acoustic feature sequence. This preprocessing process transforms the original speech waveform into a computer-processable numerical matrix, providing a normalized input format for subsequent deep networks.

[0019] Next, the acoustic feature sequence is fed into a bidirectional Long Short-Term Memory (LSTM) network for temporal dependency feature extraction. The LSM network addresses the vanishing gradient problem that standard recurrent neural networks (RNNs) often encounter when processing long sequences by introducing gating mechanisms (forget gate, input gate, and output gate), enabling it to capture long-term dependencies. To further enhance the utilization of contextual information, this embodiment employs a bidirectional structure. The forward LSM unit processes the sequence sequentially along the time axis, extracting forward temporal dependency features; the reverse LSM unit processes the sequence in reverse order along the time axis, extracting backward temporal dependency features. The forward and backward temporal dependency features are then concatenated to obtain a fused temporal feature sequence. This bidirectional processing mechanism allows the output features of each frame to simultaneously incorporate contextual information, avoiding the limitation of unidirectional networks that can only utilize historical information and cannot perceive future context, thus providing a more complete contextual representation for sentiment classification.

[0020] Building upon this, the fused temporal feature sequence is input into a self-attention mechanism layer for feature focusing and enhancement. The core idea of ​​the self-attention mechanism is to calculate the correlation weights between different positions in the sequence, enabling the model to automatically identify and focus on key frames that contribute more to emotion judgment. In speech signals, not every frame has the same amount of emotional information; often only a few key segments (such as words with drastic tone changes or specific syllables) carry the strongest emotional expression, while the rest may be neutral expressions or background noise. The self-attention mechanism automatically learns this difference in importance through a data-driven approach, allowing the model to allocate more attention to frames rich in emotional information while suppressing interference from irrelevant frames, thereby enhancing the model's sensitivity and discriminative ability to subtle emotional changes.

[0021] Subsequently, the focused feature sequence is input into a capsule network for spatial hierarchical feature extraction and sentiment classification. Traditional capsule networks are used to process two-dimensional image data; this embodiment improves upon this by using a one-dimensional model to adapt to the temporal characteristics of speech signals. Specifically, firstly, a feature extraction layer performs a one-dimensional convolution operation along the time dimension on the focused feature sequence to extract local spatial features. Then, a capsule encoding layer converts these local spatial features into multiple primary capsule vectors. Each capsule vector encodes the local spatial structure information of the speech within a certain time window in vector form; the direction of the vector represents entity attributes, and the magnitude of the vector represents the probability of entity existence. Next, a dynamic routing mechanism passes the primary capsule vectors to the category capsule layer: dynamic routing replaces the pooling operation in traditional convolutional neural networks, dynamically adjusting the coupling coefficient between the capsule encoding layer and the category capsule layer through iteration, enabling the capsule encoding layer to select the optimal path to pass information to the category capsule layer. After multiple iterations of dynamic routing, the category capsule layer outputs multiple vectors, each corresponding to a sentiment category, with the magnitude of the vector directly representing the probability of that sentiment category. Finally, the category with the largest magnitude is selected as the final sentiment classification result. For example, in the context of intelligent customer service outbound calls in the fintech field, the system can use this method to determine in real time whether the customer is satisfied, complaining, or hesitant, thereby triggering differentiated service strategies.

[0022] By applying the technical solution of this embodiment, firstly, the acoustic feature sequence spliced ​​from frames is directly used as the network input. Through the hierarchical abstraction of multi-layer networks, deep emotional representations are automatically learned, avoiding the subjectivity and incompleteness of manual feature design. Secondly, a bidirectional long short-term memory network is introduced. By capturing and splicing forward and backward contextual information through two long short-term memory units, the output features of each frame can integrate global temporal information, effectively solving the gradient vanishing problem in long sequences. An improved one-dimensional capsule network is adopted, using a dynamic routing mechanism to replace pooling operations. The vectorized capsule representation preserves the spatial positional relationship and high-level hierarchical structure between speech features, enabling the model to capture the complete mapping from local acoustic features to the overall emotional category. A self-attention mechanism layer is introduced between temporal feature extraction and the capsule network. By calculating inter-frame correlation, it automatically focuses on key speech segments rich in emotional information, suppressing the interference of background noise and neutral expressions, making the model more sensitive to subtle emotional changes. The embodiments of this application significantly improve the accuracy and robustness of speech emotion classification by constructing a three-level cascaded architecture of "bidirectional long short-term memory network - self-attention mechanism - one-dimensional capsule network".

[0023] Optionally, in this embodiment, step 102 includes: inputting the acoustic feature sequence forward along the time axis to a forward long short-term memory unit, and calculating the forward hidden state of each frame through the forward long short-term memory unit; inputting the acoustic feature sequence backward along the time axis to a reverse long short-term memory unit, and calculating the reverse hidden state of each frame through the reverse long short-term memory unit; concatenating the forward hidden state and the reverse hidden state of the same frame in the feature dimension to obtain the corresponding fused temporal feature vector, and constructing the fused temporal feature sequence based on the fused temporal feature vectors corresponding to each frame.

[0024] In this embodiment, the acoustic feature sequence is first input into the forward long short-term memory unit along the time axis. The forward long short-term memory unit processes each frame sequentially from the first frame of the speech. After processing the entire acoustic feature sequence, a forward hidden state is generated for each frame, which contains all historical information from the start of the speech to the current frame.

[0025] Simultaneously, the acoustic feature sequence is input backward along the time axis into the reverse long short-term memory (LSM). The reverse LSM processes each frame forward, starting from the last frame of the speech. Its internal gating mechanism and computational logic are identical to the forward LSM, but the processing direction is reversed. Therefore, the reverse hidden state generated in the reverse LSM for each frame contains all the "future" information from the end of the speech to the current frame—that is, the context of all speech frames after the current frame. The rationale for this bidirectional processing strategy lies in the fact that in actual speech emotion expression, the emotional state at a given moment is often influenced by both preceding and following speech content. For example, a sudden rise in pitch at the end of a smooth speech may require combining subsequent supplementary statements to accurately determine whether it indicates anger; this information cannot be obtained by observing the forward sequence alone. Through the parallel processing of the forward and reverse LSMs, two independent hidden states containing past and future information can be generated for each frame. Here, the structures of the forward and reverse LSMs can refer to existing structures and are not limited here.

[0026] After obtaining the forward and backward hidden states for the same frame, they are concatenated along the feature dimension. Since the hidden states output by the forward and backward Long Short-Term Memory (LSTM) units each have the same dimension (e.g., 64 dimensions each), the dimension of the fused temporal feature vector obtained by concatenating them end-to-end is doubled (e.g., 128 dimensions). This concatenation operation is not a simple numerical addition or averaging, but rather it preserves the two complementary information extracted from the forward and backward directions side-by-side in the feature space, so that each frame's features after fusion are simultaneously rich in the contextual information of all speech frames before and after that frame. The fused temporal feature vectors of all frames are arranged in their original temporal order, together forming a fused temporal feature sequence, which serves as the input to the subsequent self-attention mechanism layer.

[0027] In a specific embodiment, in the insurance intelligent outbound call scenario in the fintech field, the customer appears calm in the first half of the call but suddenly becomes agitated at the end. The bidirectional long short-term memory network can enable the subsequent agitation information to be fused into the fusion temporal feature vector of the calm frame in advance, thereby making the emotional representation of the entire fusion temporal feature sequence more continuous and accurate. This ensures that a correct prediction of the caller's overall emotional state can be established in the early stages of the conversation, providing a reliable basis for subsequent precise intervention.

[0028] This application's embodiments effectively solve the modeling challenge of long-distance temporal dependencies in speech signals through a bidirectional processing and splicing strategy using a bidirectional long short-term memory network. Compared to the limitation of unidirectional LSTM, which can only utilize historical information and cannot perceive future context, the bidirectional long short-term memory network, through parallel computation of both forward and backward channels, enables the output features of each frame to simultaneously integrate global temporal information, effectively improving the accuracy of subsequent speech signal emotion classification.

[0029] In this embodiment of the application, step 103 may optionally include: multiplying the fusion temporal feature vector of each frame in the fusion temporal feature sequence by a first learnable weight matrix, a second learnable weight matrix, and a third learnable weight matrix, respectively, to map the corresponding query vector, key vector, and value vector; for each frame, calculating the relevance score between the query vector of the current frame and the key vectors of other frames, and normalizing each relevance score to obtain the attention weight of the current frame relative to other frames, multiplying each attention weight by the corresponding value vector of other frames and summing the results to obtain the focus feature vector of the current frame; and constructing the focus feature sequence based on the focus feature vectors of each frame.

[0030] In this embodiment, firstly, the fused temporal feature vector of each frame in the fused temporal feature sequence is multiplied by three learnable weight matrices to obtain a query vector, a key vector, and a value vector. In the self-attention mechanism, the query vector represents the query of "what information do we want to query" in the current frame, the key vector represents the identifier of "what information can each frame provide," and the value vector represents the actual content to be extracted carried in each frame. The three weight matrices are learnable parameters that are continuously optimized and updated through backpropagation during the training of the self-attention mechanism layer model. Through this triple mapping, the fused temporal feature vector of each frame is simultaneously transformed into three different representation forms, each serving a different purpose in subsequent relevance calculations.

[0031] After obtaining the query vector, key vector, and value vector, the relevance score between the query vector and the key vectors of each frame is calculated. This calculation can be performed by taking the dot product of the query vector and the key vector. The larger the dot product value of the two vectors in the feature space, the higher their directional similarity, which means that the current frame is more closely related to that frame in terms of emotional expression features. For example, in a speech, if the current frame is an emotional outburst, its dot product value with the key vectors of other emotionally agitated frames will be higher, while its dot product value with neutral expression frames will be lower. However, the raw scores obtained from these dot products are not uniform in numerical range and cannot be directly used as weights. Therefore, all relevance scores corresponding to each frame are further normalized, for example, by using the Softmax function to convert a set of scores into a probability distribution with a sum of 1. The attention weight obtained after normalization represents the proportion of "attention" that should be allocated to other frames when analyzing the current frame. The larger the weight, the more important that frame is for understanding the emotional state of the current frame.

[0032] After obtaining the attention weights of the current frame relative to each other frame, each attention weight is multiplied by the value vectors of the corresponding other frames and then summed to obtain the focused feature vector of the current frame. Through this mechanism, the value vectors of frames highly correlated with the current frame's emotional expression (with large attention weights) dominate the summation result, while irrelevant or low-correlation frames (with small attention weights) are effectively suppressed. Since the sum of the normalized attention weights is 1, the weighted summation result is essentially a weighted average of all frame value vectors. Its effect is equivalent to redistributing the information fusion ratio based on the relevance of each frame to the current frame, making the output features of the current frame more focused on the contextual information related to its emotional expression. After performing the above calculations sequentially on all frames, each frame obtains its own focused feature vector, which, arranged in the original chronological order, constitutes the focused feature sequence.

[0033] In a specific embodiment, within the context of intelligent outbound insurance calls in the fintech field, customers may remain calm and rational in the first half of the call, only revealing a complaining tone in the last few seconds. The self-attention mechanism can accurately locate this brief, crucial emotional segment, ensuring that the overall emotional judgment is not diluted by the presence of numerous neutral frames. The final output focused feature sequence not only preserves the temporal structure of the original speech but also enhances key emotional information and filters out irrelevant information at the feature level, providing high-quality input for the subsequent refined classification by the capsule network.

[0034] This application's embodiments utilize a self-attention mechanism to adaptively focus on key emotional frames within an acoustic feature sequence, significantly enhancing the expressive power and discriminative power of emotional features. Compared to traditional methods that treat all frames equally, the self-attention mechanism automatically identifies and highlights emotionally rich speech segments. This dynamic focusing capability makes the model more sensitive to subtle emotional changes in speech signals and more adaptable to complex acoustic environments. Furthermore, the computational method of the self-attention mechanism allows direct association between any two frames without temporal distance limitations, thus compensating for the potential shortcomings of recurrent neural networks in modeling extremely long-distance dependencies. This enables the model to simultaneously capture local emotional details and global emotional trends.

[0035] Optionally, in this embodiment, the feature extraction layer extracts local spatial features based on the following operations: for each preset one-dimensional convolutional kernel, the one-dimensional convolutional kernel slides along the time dimension of the speech signal on the focused feature sequence. After each slide, according to the local window corresponding to the one-dimensional convolutional kernel, the focused feature vectors contained in the local window are weighted, summed, and biased, and then activated by a nonlinear activation function to obtain a scalar value. Based on the scalar value corresponding to each slide, the one-dimensional feature vector of the one-dimensional convolutional kernel is output. The one-dimensional feature vectors output by each one-dimensional convolutional kernel are stacked to obtain two-dimensional local spatial features.

[0036] In this embodiment, the focused feature sequence is first used as input to the feature extraction layer. The dimension of the focused feature sequence is the number of frames multiplied by the dimension of the focused feature vector, where the focused feature vector of each frame has been weighted and recombined by the self-attention mechanism, making the emotional information more prominent. Since speech signals have local continuity in the temporal dimension—for example, acoustic features between adjacent frames are often highly correlated, such as a syllable or word typically spanning multiple consecutive frames—this combination of features within a local neighborhood contains rich spatial structural information. However, while the self-attention mechanism can model global inter-frame relationships, it does not explicitly extract such local spatial patterns within the temporal neighborhood. Therefore, this embodiment uses one-dimensional convolution operations in the feature extraction layer of the capsule network to capture this local spatial correlation. The difference between one-dimensional convolution and the two-dimensional convolution commonly used in image processing is that its convolution kernel only slides in the time axis direction, while the feature dimension direction is fully covered. This adapts to the one-dimensional temporal structure of speech data, which is "time-axis and feature-channel."

[0037] For each pre-defined one-dimensional convolutional kernel, it can be slid along the temporal dimension of the speech signal on the focused feature sequence. This one-dimensional convolutional kernel is essentially a learnable weight matrix, its size covering all channels of the focused feature vector and the local window range on the time axis (e.g., the window covers several consecutive frames). At each sliding position, the local window currently covered by the one-dimensional convolutional kernel contains the focused feature vectors of all frames within that time interval. The weight parameters of the one-dimensional convolutional kernel are multiplied element-wise with the focused feature vectors at the corresponding positions within the local window, summed, and then a learnable bias term is added. Finally, after activation by a non-linear activation function (such as ReLU), a scalar value is output. This calculation process is mathematically equivalent to the one-dimensional convolutional kernel performing a linear transformation plus a non-linear mapping on the features in the local temporal neighborhood. The magnitude of the output value reflects the degree of matching between the feature pattern within the local window and the pattern learned by the convolutional kernel; the higher the matching degree, the larger the output value. After a one-dimensional convolutional kernel completes its sliding along the entire time axis, it outputs a scalar value for each slide. All scalar values ​​are arranged in chronological order, which constitutes the one-dimensional feature vector corresponding to the one-dimensional convolutional kernel.

[0038] The feature extraction layer can contain multiple pre-defined one-dimensional convolutional kernels, each with independent learnable weight parameters, thus learning different local feature patterns. For example, different one-dimensional convolutional kernels may be sensitive to different frequency variation patterns, energy abrupt changes of different durations, or different prosodic fluctuations in speech. After sliding, each one-dimensional convolutional kernel outputs a one-dimensional feature vector, and these one-dimensional feature vectors have the same length in the feature dimension (equal to the number of slides on the time axis). Stacking the one-dimensional feature vectors output by all the one-dimensional convolutional kernels in the channel dimension forms a two-dimensional local spatial feature. Thus, the original single-channel temporal data is mapped to a multi-channel temporal feature map, where the richness of the channel dimension provides more diverse spatial feature inputs for subsequent capsule coding layers.

[0039] This application's embodiments effectively extract local spatial structure information from speech signals through one-dimensional convolution operations, laying the foundation for subsequent high-order feature abstraction in capsule networks. Compared to self-attention mechanisms or recurrent neural networks that rely solely on global context modeling, one-dimensional convolution can explicitly capture local dependencies between adjacent frames. These local structures are often key clues for determining emotion categories. Simultaneously, the parallel use of multiple convolutional kernels allows the model to perceive local acoustic patterns from multiple angles and scales, greatly enriching the diversity of feature representation.

[0040] Optionally, in this embodiment, step 104, "converting the local spatial features into multiple primary capsule vectors through the capsule coding layer of the capsule network," includes: inputting the two-dimensional local spatial features into the capsule coding layer; the capsule coding layer contains multiple capsule units, each capsule unit corresponding to a one-dimensional filter; for each capsule unit, the one-dimensional filter corresponding to the capsule unit slides along the time dimension of the speech signal on the two-dimensional local spatial features; after each slide, according to the local window corresponding to the one-dimensional filter, the feature values ​​within the local window are weighted, summed, and biased, and then activated by a nonlinear activation function to obtain a scalar value; based on the scalar value corresponding to each slide, the primary capsule vector of the capsule unit is output; the primary capsule vectors output by each capsule unit are used as the multiple primary capsule vectors.

[0041] In this embodiment, the two-dimensional local spatial features output from the previous feature extraction layer are first input into the capsule encoding layer. The two dimensions of the two-dimensional local spatial features correspond to the time axis and the convolutional kernel channel axis, respectively, forming a multi-channel temporal feature map. The channel dimension records the response intensity of different convolutional kernels at different time positions, containing rich local acoustic pattern information. The capsule encoding layer acts as a bridge connecting the lower-level feature extraction and higher-level sentiment classification. Its core task is to convert the two-dimensional temporal feature map into a vector representation, providing input for the subsequent dynamic routing mechanism. This layer contains multiple capsule units, each essentially an independent feature extractor responsible for extracting specific types of spatial patterns from the two-dimensional local spatial features and outputting them as vectors. It is important to note that there is a fundamental difference between capsule units and traditional convolutional neurons. Traditional convolutional neurons output a scalar value, which only represents the activation intensity of a certain feature; while capsule units output a vector. Different dimensions of the vector can encode different attribute information of the entity, such as the amplitude of pitch change, the duration of energy mutation, and the direction of the shift of the spectral centroid. This vectorized representation allows a single capsule to carry much richer entity description information than scalar neurons.

[0042] Each capsule unit is equipped with an independent one-dimensional filter that slides along the time dimension of the speech signal across two-dimensional local spatial features. Similar to the one-dimensional convolutional kernels in the feature extraction layer, the one-dimensional filter moves only along the time axis, covering the feature values ​​of all channels within the current time window at each sliding position. The time window corresponding to each sliding position contains the local spatial response values ​​of each channel within that time period, reflecting the acoustic pattern information detected by different convolutional kernels in that temporal neighborhood. Then, the learnable weight parameters of the one-dimensional filter are multiplied element-wise with the feature values ​​at each position within the current local window, summed, and a learnable bias term is added. This is then mapped through a non-linear activation function (such as ReLU) to finally output a scalar value. Each capsule unit learns its own filter parameters and independently determines how to spatially integrate the multi-channel features within its local window, thereby capturing the local patterns most relevant to its function.

[0043] As the one-dimensional filter slides along the time axis, each capsule unit generates a scalar output at each sliding position. The scalar values ​​corresponding to these sliding positions are arranged chronologically to form the primary capsule vector output by that capsule unit. It's important to note that the length of the primary capsule vector is equal to the number of times the filter slides along the time axis, and each component of the vector corresponds to the filter's response intensity at a specific time position. By arranging the response values ​​at each time position into a vector form, this primary capsule vector actually encodes the intensity and distribution of a specific type of spatial pattern across the entire speech time axis. For example, a one-dimensional filter in a capsule unit might be sensitive to patterns with sudden increases in high-frequency energy. In this case, the component value at the actual time position where the high-frequency increase occurs in the speech would be larger, while the component values ​​at other positions would be smaller. After all capsule units have independently completed this process, each capsule unit outputs its corresponding primary capsule vector.

[0044] In a specific embodiment, in the insurance intelligent outbound call scenario in the fintech field, different capsule units can encode different acoustic behavior patterns such as "sudden increase in speech rate", "rising tone", "energy burst", and "abnormal pause". Based on this, it is possible to distinguish between two completely different negative emotions, such as "intense outburst of anger" and "dejected and melancholic pause", effectively avoiding misjudgment caused by feature confusion and providing more comprehensive evidence support for the comprehensive judgment of subsequent dynamic routing.

[0045] This embodiment of the application, through the vectorization transformation of the capsule coding layer described above, can decompose two-dimensional local spatial features into multiple independent primary capsule vectors encoding different spatial patterns. Each capsule unit acts as an independent feature channel, outputting its detection result for a certain type of local spatial pattern in speech in vector form. Compared to the traditional method of directly flattening the convolution output into a scalar sequence, the capsule coding layer preserves the independence between each feature pattern, avoiding the mixing and interference of information between different patterns. At the same time, the primary capsule vector output by each capsule unit allows a single array element to carry multi-dimensional information of attributes (such as the occurrence time, intensity, and spectral characteristics of the pattern), providing a richer feature description for subsequent dynamic routing.

[0046] In this embodiment of the application, optionally, step 104, "passing the multiple primary capsule vectors to the category capsule layer of the capsule network through the dynamic routing mechanism of the capsule network to obtain the output vector of each category capsule," includes: initializing the feature parameters between the capsule encoding layer and the category capsule layer; converting the current feature parameters into coupling coefficients for each primary capsule vector to be passed to each category capsule through the Softmax function; each primary capsule vector corresponds to a coupling coefficient for each category capsule; for each category capsule, according to the coupling coefficients corresponding to each primary capsule vector, performing a weighted summation on the prediction vectors obtained after mapping each primary capsule vector through a weight matrix to obtain the pre-activation vector of the category capsule; performing a compression transformation on the pre-activation vectors of each category capsule through the Squash activation function to obtain the output vector of each category capsule; updating the feature parameters according to the consistency between the output vector of each category capsule and the prediction vector of each primary capsule vector, and returning to the step of converting the current feature parameters into coupling coefficients through the Softmax function, until a preset convergence condition is met to obtain the iteratively updated output vector of each category capsule.

[0047] In this embodiment, the feature parameters between the capsule encoding layer and the category capsule layer are first initialized. In the dynamic routing mechanism, the feature parameters belong to the communication channels connecting the primary capsule vectors and the category capsules. Each feature parameter corresponds to a specific primary capsule vector-category capsule connection pair, and its value determines the basic weight of the connection in subsequent information transmission. Before routing begins, all feature parameters can be initialized to zero, meaning that in the initial state, all possible connection paths have the exact same basic weight, and there is no prior preference for any path.

[0048] Subsequently, the Softmax function is used to convert the current feature parameters into coupling coefficients that are passed from each primary capsule vector to each category capsule. The Softmax function transforms a set of arbitrary real-valued vectors into a probability distribution, where all coupling coefficients are non-negative and sum to 1. This transformation gives the coupling coefficients a clear probabilistic interpretation: the sum of the coupling coefficients between each primary capsule vector and all category capsules is 1, indicating that the primary capsule vector proportionally allocates all its "voting power" to each category capsule. The larger the coupling coefficient, the more likely the primary capsule vector is to pass information to the corresponding category capsule. Since there is an independent coupling coefficient between each pair of primary capsule vectors and category capsules, all coupling coefficients form a matrix, with rows corresponding to primary capsule vectors and columns corresponding to category capsules. In the first iteration, since all feature parameters are zero, all coupling coefficients are equal after the Softmax transformation, and each primary capsule vector equally distributes its voting power to all category capsules. As the iteration progresses, the coupling coefficients are continuously adjusted based on the consistency feedback of feature transmission, gradually converging to a differentiated distribution that effectively distinguishes each category.

[0049] After obtaining the coupling coefficients, for each category capsule, based on the coupling coefficients corresponding to the primary capsule vectors, the predicted vectors obtained after mapping each primary capsule vector through the weight matrix are weighted and summed to obtain the pre-activation vector for that category capsule. Specifically, each primary capsule vector is first multiplied by a learnable weight matrix. The dimension of this matrix determines the dimension of the predicted vector, and it is usually consistent with the output dimension of the category capsule. It is important to note that the learnable weight matrix here is a set of mathematical transformation tables whose values ​​are continuously adjusted through error feedback during training. Its function is to transform the low-level features (primary capsule vectors) into a form that the high-level categories (emotion categories) can understand, and different transformation standards are used for different emotion categories. This mapping process ensures that primary capsule vectors of different dimensions are uniformly projected into the vector space of the category capsules, so that subsequent vector operations are performed within the same geometric framework. Each pair of primary capsule vectors and category capsules has an independent weight matrix. Therefore, the same primary capsule vector can produce different predicted vectors when projected onto different category capsules. For example, the prediction submitted to the "angry" category capsule may be completely different from the prediction submitted to the "happy" category capsule. After mapping, the predicted vectors of each primary capsule are weighted and summed according to their corresponding coupling coefficients. The resulting pre-activation vector is a vector with dimensions equal to the vector dimension of the category capsule (e.g., 16-dimensional), representing the combined voting result of all primary capsule vectors for that category capsule. Since the coupling coefficients are independently normalized for each category capsule, a weighted summation is performed for each category capsule, resulting in a total of weighted summation operations equal to the number of emotion categories. Each operation outputs a pre-activation vector. For example, if five emotions need to be identified, five pre-activation vectors are obtained, corresponding to the categories of anger, happiness, sadness, calmness, and surprise, respectively.

[0050] After obtaining the pre-activation vectors of each category capsule, the Squash activation function is used to compress and transform them, yielding the output vectors for each category capsule. The mathematical property of the Squash function is that it compresses the magnitude of the input vector to between 0 and 1, while preserving the direction of the input vector. The magnitude of the compressed vector can be directly interpreted as the probability of a certain category existing; the closer the magnitude is to 1, the higher the probability of that category existing; the closer the magnitude is to 0, the lower the probability of that category existing. Simultaneously, the direction of the vector encodes the specific attribute information of that category. For example, in speech emotion recognition, the direction of the anger category capsule might encode a combination of attributes such as extremely fast speech rate, extremely high energy, and a high spectral density, while the direction of the happiness category capsule might encode a combination of attributes such as moderate speech rate, slightly high energy, and a smooth spectral density. After Squash activation, the output vectors of each category capsule possess clear probabilistic semantics, allowing the system to directly make emotion classification decisions based on the vector magnitude without the need for additional classification layers or post-processing steps.

[0051] After calculating the output vector for this round, the feature parameters are updated based on the consistency between the output vectors of each category capsule and the predicted vectors of each primary capsule vector, and the process returns to the Softmax transformation step to enter the next iteration. The mathematical measure of consistency can be the dot product of two vectors. The larger the dot product value, the closer the two vectors are in direction and magnitude, meaning the more closely the predicted vector of the primary capsule vector matches the actual output of the category capsule. If the predicted vector of a primary capsule vector is highly consistent with the output vector of the category capsule, it indicates that the primary capsule vector's judgment of the current category capsule is "accurate," and the feature parameters corresponding to this connection pair can be increased to obtain a larger coupling coefficient in the next iteration, giving the primary capsule vector greater voting influence. Conversely, if the two directions are contradictory, the feature parameters will be decreased, and the influence of the primary capsule vector will be weakened. The entire iterative process continues until a preset convergence condition is met, such as reaching a preset number of iterations or the change in feature parameters being less than a certain threshold. After the iteration is completed, the output vectors of each category of capsules are output after multiple rounds of optimization. These vectors integrate the weighted opinions of all primary capsule vectors, and the weight allocation has been repeatedly adjusted through iteration to ensure the optimality of information transmission.

[0052] In a specific embodiment, in the insurance intelligent outbound call quality inspection scenario in the fintech field, multiple acoustic anomaly patterns such as "sudden increase in volume," "faster speech rate," and "high-frequency energy surge" may occur simultaneously in customer calls. The dynamic routing mechanism can automatically identify the pattern combination most relevant to the current call emotion category in multiple iterations. For example, it can increase the coupling coefficient of the two primary capsule vectors "sudden increase in volume" and "faster speech rate" while suppressing the voting weight of irrelevant primary capsule vectors such as "background silence," thereby accurately judging the customer's anger.

[0053] This application's embodiments, through the aforementioned dynamic routing mechanism, achieve hierarchical information aggregation from multiple primary capsule vectors to global sentiment decisions, effectively replacing the pooling operation in traditional convolutional neural networks. Compared to pooling's "retaining local maxima and discarding the rest" strategy, dynamic routing, through a multi-round iterative "voting-feedback-adjustment" mechanism, ensures that information from all primary capsule vectors participates in high-level decisions with appropriate weights, effectively improving the accuracy of subsequent speech sentiment classification.

[0054] In this embodiment of the application, step 105 may optionally include: calculating the magnitude of the output vector of each category capsule; the magnitude of the output vector represents the probability that the speech signal belongs to the emotion category of the corresponding category capsule layer; determining the maximum magnitude from each magnitude, and taking the emotion category corresponding to the maximum magnitude as the emotion category of the speech signal.

[0055] In this embodiment, after obtaining the output vectors of each category capsule, the magnitude of the output vector of each category capsule is first calculated. In the capsule network, each category capsule outputs a vector rather than a scalar. The magnitude of the vector is obtained by calculating the square root of the sum of the squares of the vector's components, resulting in a non-negative real number. The magnitude of the output vector of each category capsule directly represents the probability that the speech signal belongs to the corresponding emotion category of that category capsule. During training, the model continuously adjusts the network parameters through supervised learning, so that when the input speech belongs to a certain emotion category, the magnitude of the output vector of the corresponding category capsule approaches 1, while the magnitude of the output vector of other category capsules approaches 0. Under this probabilistic representation, the outputs of all category capsules have the same numerical range and the same semantic interpretation (i.e., the magnitude is the probability), without the need for additional probability normalization using Softmax as in traditional classifiers. For example, if the magnitude of the anger category capsule is 0.85 and the magnitude of the happiness category capsule is 0.12, it indicates that the emotional evidence for the speech is strong in the anger dimension, but weak in the happiness dimension. The modulus of multiple categories of capsules forms a probability distribution vector, which comprehensively reflects the degree of tendency of the speech signal in various emotions.

[0056] This application's embodiments achieve end-to-end sentiment classification decisions through the aforementioned classification decision mechanism based on the output vector magnitude, offering significant advantages such as simplicity, efficiency, and interpretability. Compared to traditional classification methods that require a Softmax layer to convert feature vectors into probability distributions, the capsule network's magnitude decision is performed directly within the vector space, avoiding potential information loss due to additional nonlinear transformations. The magnitude decision mechanism in this application's embodiments, with its intuitive probabilistic semantics and extremely low computational cost, enables the entire sentiment classification process, from feature extraction to decision output, to form a complete logical closed loop, providing an efficient and reliable classification scheme for real-time speech sentiment analysis scenarios.

[0057] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, another speech emotion classification method based on capsule networks and attention mechanisms is provided, such as... Figure 2As shown, the processing flow of this method includes: the speech signal is first segmented and acoustic features are extracted to form a structured acoustic feature sequence, which serves as the input to subsequent networks. This acoustic feature sequence then undergoes three core processing stages: a bidirectional long short-term memory network captures the temporal dependencies of the speech context through forward and backward channels, outputting a fused temporal feature sequence; a self-attention mechanism layer models the relevance of the fused temporal feature sequence, focusing on keyframes rich in emotional information, and outputting a focused feature sequence; a capsule network performs one-dimensional convolution on the focused feature sequence to extract local spatial features, which are then converted into multiple primary capsule vectors by a capsule encoding layer, and then passed to the category capsule layer through a dynamic routing mechanism, outputting category capsule vectors. Finally, the magnitude of the output vectors of each category capsule is calculated, and the category with the largest magnitude is determined as the emotional category of the speech signal. The entire process achieves end-to-end automatic mapping from raw speech to emotional labels.

[0058] In another specific embodiment, such as Figure 3 The diagram illustrates the structure of a capsule network provided in an embodiment of this application. The processing flow of this capsule network is as follows: An acoustic feature sequence with input features of 444×1 is first passed through a feature extraction layer (containing...) Figure 3 The input consists of a one-dimensional convolutional layer (Conv1), which uses 64 one-dimensional convolutional kernels to extract local features from the input and outputs local spatial features after dimensionality transformation. Next, the input enters a capsule encoding layer, which uses 32 one-dimensional filters, each outputting an 8-dimensional primary capsule vector. This means each capsule unit encodes local spatial features as an 8-dimensional primary capsule vector. Then, a dynamic routing mechanism passes the 32 primary capsule vectors to the category capsule layer (LabelCapsule). This layer contains n (corresponding to n sentiment categories) 16-dimensional capsules (meaning each category capsule outputs a 16-dimensional vector). Each category capsule is assigned an independent weight matrix W. ij (N×16 dimension) It receives the predicted vectors from the primary capsule vectors and performs weighted aggregation. Finally, it calculates the magnitude of the output vector for each category of capsules, i.e. Figure 3 In , to n The category with the largest value (i.e. the largest modulus) is output as the sentiment classification result.

[0059] Furthermore, as Figure 1 In terms of specific implementation, this application provides a speech emotion classification device based on capsule networks and attention mechanisms, such as... Figure 4 As shown, the device includes: The feature extraction module is used to acquire the speech signal to be classified, perform frame-by-frame processing on the speech signal, extract the acoustic features of each frame of speech, and concatenate the acoustic features corresponding to each frame in the feature dimension to form an acoustic feature sequence. The first processing module is used to input the acoustic feature sequence into a bidirectional long short-term memory network, extract forward temporal dependency features and backward temporal dependency features through the forward long short-term memory unit and the backward long short-term memory unit, respectively, and concatenate the extraction results to obtain a fused temporal feature sequence. The second processing module is used to input the fused temporal feature sequence into the self-attention mechanism layer, perform correlation calculation on each fused temporal feature vector in the fused temporal feature sequence through the self-attention mechanism layer, and construct a focused feature sequence based on the correlation calculation result; The third processing module is used to input the focused feature sequence into the capsule network, and to perform a one-dimensional convolution operation on the focused feature sequence along the time dimension of the speech signal through the feature extraction layer of the capsule network to extract local spatial features; to convert the local spatial features into multiple primary capsule vectors through the capsule encoding layer of the capsule network; and to pass the multiple primary capsule vectors to the category capsule layer of the capsule network through the dynamic routing mechanism of the capsule network to obtain the output vector of each category capsule. The classification module is used to determine the emotion category of the speech signal based on the magnitude of the output vector of each category capsule.

[0060] Optionally, the first processing module is configured to: The acoustic feature sequence is input into the forward long short-term memory unit along the time axis, and the forward hidden state of each frame is calculated through the forward long short-term memory unit; The acoustic feature sequence is input in reverse along the time axis to the inverse long short-term memory unit, and the inverse hidden state of each frame is calculated through the inverse long short-term memory unit; The forward and reverse hidden states of the same frame are concatenated along the feature dimension to form the corresponding fusion temporal feature vector. The fusion temporal feature sequence is constructed based on the fusion temporal feature vectors corresponding to each frame.

[0061] Optionally, the second processing module is used for: The fused temporal feature vector of each frame in the fused temporal feature sequence is multiplied by the first learnable weight matrix, the second learnable weight matrix, and the third learnable weight matrix, respectively, to obtain the corresponding query vector, key vector, and value vector. For each frame, calculate the relevance score between the query vector of the current frame and the key vectors of other frames, normalize each relevance score, obtain the attention weight of the current frame relative to other frames, multiply each attention weight by the value vector of the corresponding other frame and sum them to obtain the focus feature vector of the current frame. The focus feature sequence is constructed based on the focus feature vectors of each frame.

[0062] Optionally, the feature extraction layer extracts local spatial features based on the following operation: For each preset one-dimensional convolutional kernel, the one-dimensional convolutional kernel slides along the time dimension of the speech signal on the focused feature sequence. After each slide, according to the local window corresponding to the one-dimensional convolutional kernel, the focused feature vector contained in the local window is weighted, summed, and biased. After activation by a nonlinear activation function, a scalar value is obtained. According to the scalar value corresponding to each slide, the one-dimensional feature vector of the one-dimensional convolutional kernel is output. Stacking the one-dimensional feature vectors output by each one-dimensional convolution kernel yields two-dimensional local spatial features.

[0063] Optionally, the third processing module is used for: The two-dimensional local spatial features are input into the capsule coding layer; the capsule coding layer contains multiple capsule units, and each capsule unit corresponds to a one-dimensional filter. For each capsule unit, the one-dimensional filter corresponding to the capsule unit slides along the time dimension of the speech signal on the two-dimensional local spatial features. After each slide, according to the local window corresponding to the one-dimensional filter, the feature values ​​in the local window are weighted, summed, and biased. After activation by a nonlinear activation function, a scalar value is obtained. According to the scalar value after each slide, the primary capsule vector of the capsule unit is output. The primary capsule vectors output by each capsule unit are used as the plurality of primary capsule vectors.

[0064] Optionally, the third processing module is used for: Initialize the feature parameters between the capsule coding layer and the category capsule layer; The Softmax function converts the current feature parameters into coupling coefficients for each primary capsule vector, which are then passed to each category capsule. Each primary capsule vector corresponds to a coupling coefficient for each category capsule. For each type of capsule, based on the coupling coefficient corresponding to each primary capsule vector, the predicted vectors obtained after mapping each primary capsule vector through the weight matrix are weighted and summed to obtain the pre-activation vector of the type of capsule. The pre-activation vectors of each type of capsule are compressed and transformed using the Squash activation function to obtain the output vectors of each type of capsule; Based on the consistency between the output vector of each category capsule and the predicted vector of each primary capsule, the feature parameters are updated, and the process returns to the step of converting the current feature parameters into coupling coefficients using the Softmax function, until the preset convergence condition is met, and the iteratively updated output vectors of each category capsule are obtained.

[0065] Optionally, the classification module is used for: Calculate the magnitude of the output vector for each category of capsule; the magnitude of the output vector represents the probability that the speech signal belongs to the emotion category of the corresponding category capsule layer; The largest module length is determined from all module lengths, and the emotion category corresponding to the largest module length is taken as the emotion category of the speech signal.

[0066] It should be noted that other corresponding descriptions of the functional units involved in the speech emotion classification device based on capsule networks and attention mechanisms provided in this application embodiment can be found in the following references. Figures 1 to 3 The corresponding descriptions in the method will not be repeated here.

[0067] This application also provides a computer device, which may specifically be a personal computer, a server, a network device, etc. Figure 5 As shown, the computer device includes a bus, a processor, memory, and a communication interface, and may also include an input / output interface and a display device. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores location information. The network interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0068] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0069] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, having stored thereon a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0070] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0071] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0072] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0073] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0074] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A speech emotion classification method based on capsule networks and attention mechanisms, characterized in that, include: Acquire the speech signal to be classified, perform frame segmentation on the speech signal, extract the acoustic features of each frame of speech, and concatenate the acoustic features corresponding to each frame in the feature dimension to form an acoustic feature sequence. The acoustic feature sequence is input into a bidirectional long short-term memory network. Forward temporal dependency features and backward temporal dependency features are extracted through the forward long short-term memory unit and the backward long short-term memory unit, respectively. The extracted results are then concatenated to obtain a fused temporal feature sequence. The fused temporal feature sequence is input into the self-attention mechanism layer, and the correlation of each fused temporal feature vector in the fused temporal feature sequence is calculated by the self-attention mechanism layer. Based on the correlation calculation result, a focused feature sequence is constructed. The focused feature sequence is input into the capsule network. The feature extraction layer of the capsule network performs a one-dimensional convolution operation on the focused feature sequence along the time dimension of the speech signal to extract local spatial features. The capsule encoding layer of the capsule network converts the local spatial features into multiple primary capsule vectors. The dynamic routing mechanism of the capsule network passes the multiple primary capsule vectors to the category capsule layer of the capsule network to obtain the output vector of each category capsule. The emotion category of the speech signal is determined based on the magnitude of the output vector of each category of capsule.

2. The method according to claim 1, characterized in that, The acoustic feature sequence is input into a bidirectional long short-term memory (LSTM) network. Forward and backward temporal dependency features are extracted through the forward and backward LSM units, respectively. The extracted results are then concatenated to obtain a fused temporal feature sequence, including: The acoustic feature sequence is input into the forward long short-term memory unit along the time axis, and the forward hidden state of each frame is calculated through the forward long short-term memory unit; The acoustic feature sequence is input in reverse along the time axis to the inverse long short-term memory unit, and the inverse hidden state of each frame is calculated through the inverse long short-term memory unit; The forward and reverse hidden states of the same frame are concatenated along the feature dimension to form the corresponding fusion temporal feature vector. The fusion temporal feature sequence is constructed based on the fusion temporal feature vectors corresponding to each frame.

3. The method according to claim 2, characterized in that, The step involves inputting the fused temporal feature sequence into a self-attention mechanism layer, calculating the correlation between each fused temporal feature vector in the fused temporal feature sequence through the self-attention mechanism layer, and constructing a focused feature sequence based on the correlation calculation results, including: The fused temporal feature vector of each frame in the fused temporal feature sequence is multiplied by the first learnable weight matrix, the second learnable weight matrix, and the third learnable weight matrix, respectively, to obtain the corresponding query vector, key vector, and value vector. For each frame, calculate the relevance score between the query vector of the current frame and the key vectors of other frames, normalize each relevance score, obtain the attention weight of the current frame relative to other frames, multiply each attention weight by the value vector of the corresponding other frame and sum them to obtain the focus feature vector of the current frame. The focus feature sequence is constructed based on the focus feature vectors of each frame.

4. The method according to claim 3, characterized in that, The feature extraction layer extracts local spatial features based on the following operations: For each preset one-dimensional convolutional kernel, the one-dimensional convolutional kernel slides along the time dimension of the speech signal on the focused feature sequence. After each slide, according to the local window corresponding to the one-dimensional convolutional kernel, the focused feature vector contained in the local window is weighted, summed, and biased. After activation by a nonlinear activation function, a scalar value is obtained. According to the scalar value corresponding to each slide, the one-dimensional feature vector of the one-dimensional convolutional kernel is output. Stacking the one-dimensional feature vectors output by each one-dimensional convolution kernel yields two-dimensional local spatial features.

5. The method according to claim 1, characterized in that, The process of converting the local spatial features into multiple primary capsule vectors through the capsule encoding layer of the capsule network includes: The two-dimensional local spatial features are input into the capsule coding layer; the capsule coding layer contains multiple capsule units, and each capsule unit corresponds to a one-dimensional filter. For each capsule unit, the one-dimensional filter corresponding to the capsule unit slides along the time dimension of the speech signal on the two-dimensional local spatial features. After each slide, according to the local window corresponding to the one-dimensional filter, the feature values ​​in the local window are weighted, summed, and biased. After activation by a nonlinear activation function, a scalar value is obtained. According to the scalar value after each slide, the primary capsule vector of the capsule unit is output. The primary capsule vectors output by each capsule unit are used as the plurality of primary capsule vectors.

6. The method according to claim 1, characterized in that, The process of passing the multiple primary capsule vectors to the category capsule layer of the capsule network through the dynamic routing mechanism of the capsule network to obtain the output vector of each category capsule includes: Initialize the feature parameters between the capsule coding layer and the category capsule layer; The Softmax function converts the current feature parameters into coupling coefficients for each primary capsule vector, which are then passed to each category capsule. Each primary capsule vector corresponds to a coupling coefficient for each category capsule. For each type of capsule, based on the coupling coefficient corresponding to each primary capsule vector, the predicted vectors obtained after mapping each primary capsule vector through the weight matrix are weighted and summed to obtain the pre-activation vector of the type of capsule. The pre-activation vectors of each type of capsule are compressed and transformed using the Squash activation function to obtain the output vectors of each type of capsule; Based on the consistency between the output vector of each category capsule and the predicted vector of each primary capsule, the feature parameters are updated, and the process returns to the step of converting the current feature parameters into coupling coefficients using the Softmax function, until the preset convergence condition is met, and the iteratively updated output vectors of each category capsule are obtained.

7. The method according to claim 1, characterized in that, Determining the emotion category of the speech signal based on the magnitude of the output vector of each category of capsule includes: Calculate the magnitude of the output vector for each category of capsule; the magnitude of the output vector represents the probability that the speech signal belongs to the emotion category of the corresponding category capsule layer; The largest module length is determined from all module lengths, and the emotion category corresponding to the largest module length is taken as the emotion category of the speech signal.

8. A speech emotion classification device based on capsule networks and attention mechanisms, characterized in that, include: The feature extraction module is used to acquire the speech signal to be classified, perform frame-by-frame processing on the speech signal, extract the acoustic features of each frame of speech, and concatenate the acoustic features corresponding to each frame in the feature dimension to form an acoustic feature sequence. The first processing module is used to input the acoustic feature sequence into a bidirectional long short-term memory network, extract forward temporal dependency features and backward temporal dependency features through the forward long short-term memory unit and the backward long short-term memory unit, respectively, and concatenate the extraction results to obtain a fused temporal feature sequence. The second processing module is used to input the fused temporal feature sequence into the self-attention mechanism layer, perform correlation calculation on each fused temporal feature vector in the fused temporal feature sequence through the self-attention mechanism layer, and construct a focused feature sequence based on the correlation calculation result; The third processing module is used to input the focused feature sequence into the capsule network, and to perform a one-dimensional convolution operation on the focused feature sequence along the time dimension of the speech signal through the feature extraction layer of the capsule network to extract local spatial features; to convert the local spatial features into multiple primary capsule vectors through the capsule encoding layer of the capsule network; and to pass the multiple primary capsule vectors to the category capsule layer of the capsule network through the dynamic routing mechanism of the capsule network to obtain the output vector of each category capsule. The classification module is used to determine the emotion category of the speech signal based on the magnitude of the output vector of each category capsule.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

10. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

Images