Lightweight few-shot audio emotion recognition method and system for edge deployment

Abstract

This invention discloses a lightweight, few-shot audio emotion recognition method and system for edge deployment, belonging to the field of audio signal processing technology. The method includes: acquiring and preprocessing multi-emotion audio data; extracting multi-layer features using a parameter-frozen self-supervised pre-trained model, and weighting and fusing them using learnable inter-layer weight parameters to generate a high-dimensional acoustic feature sequence; constructing a dual-stream complementary feature encoder composed of static and dynamic branches to extract long-term global prosodic features and short-term local prosodic features respectively, and concatenating and fusing them; employing an attention-based statistical pooling mechanism to calculate the weighted mean and weighted standard deviation to generate a fixed-length emotion embedding vector; and comparing the embedding vector with the Euclidean distance of each category prototype in the support set based on a few-shot learning task to output the emotion classification result. This invention achieves efficient and real-time emotion recognition at the edge, possessing the advantages of high accuracy and low computational overhead under conditions of data scarcity.

Description

Technical Field

[0001] This invention belongs to the field of audio signal processing and artificial intelligence technology, specifically relating to a lightweight, few-sample audio emotion recognition method and system for edge deployment. Background Technology

[0002] With the breakthroughs achieved by deep learning (DL) technology in the field of audio signal processing, it has been widely applied in personalized human-computer interaction scenarios. As a key application, speech emotion recognition (SER) technology automatically determines the speaker's emotional state by analyzing the acoustic features in audio, showing great potential in patient emotion monitoring, rehabilitation effect assessment, and empowering intelligent interactive devices.

[0003] However, in rehabilitation scenarios where sample acquisition is difficult and resources are limited, existing technologies still face a series of fundamental challenges in achieving low-latency, accurate, and personalized emotion recognition. First, training data is extremely scarce, leading to severely insufficient model generalization ability. Not only is it difficult to collect and accurately label emotional audio data for specific pathological features or specific users on a large scale, but it also involves strict privacy protection issues. Traditional deep learning heavily relies on massive amounts of data for training, making it prone to overfitting under small sample conditions and unable to learn robust, generalized acoustic emotion representations, resulting in a sharp drop in recognition accuracy on the test set. Second, the huge differences in acoustic features between individuals make it difficult to personalize the model. Different users have significant physiological and habitual differences in pronunciation mechanisms, speech rates, fundamental frequency ranges, and pronunciation clarity, resulting in severe feature distribution shifts. General emotion recognition models struggle to effectively adapt to these individual differences, exhibiting significant performance fluctuations or even failure when facing new users, failing to meet the requirements for consistency and accuracy in emotion monitoring in rehabilitation scenarios. Third, the excessive complexity of the model architecture leads to high deployment costs at the edge. While existing mainstream methods utilize pre-trained models to extract general acoustic features, their backends are often cascaded with heavy encoders based on self-attention mechanisms. Since the computational complexity of self-attention increases quadratically with sequence length, it leads to severe parameter redundancy and significant inference latency, making it difficult to deploy on portable rehabilitation terminals or embedded devices with limited computing power.

[0004] In summary, existing technologies have the following problems: First, training data is extremely scarce in small sample scenarios, and traditional high-parameter models are prone to overfitting during training, resulting in limited generalization performance; Second, there are significant differences in acoustic features among different individuals, and general models are unable to effectively cope with severe feature distribution shifts, resulting in poor cross-user adaptability; Third, existing methods mostly use heavy self-attention encoders, resulting in excessively high computational complexity, making it difficult to achieve low-latency inference on resource-constrained edge devices.

[0005] While existing research attempts to address some of these issues through data augmentation, transfer learning, or model lightweighting, these approaches often fall short of achieving synergistic optimization. Therefore, the field urgently needs a new technological solution that can simultaneously solve the challenges of data scarcity and personalized adaptation, while significantly reducing computational complexity for efficient edge deployment. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a lightweight few-sample audio emotion recognition method and system for edge deployment.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a lightweight few-sample audio emotion recognition method for edge deployment, comprising the following steps:

[0008] S1. Collect raw audio segments from multiple subjects under different emotional states, and label each raw audio segment with an emotional category to form an emotionally labeled audio dataset.

[0009] S2. Preprocess the original audio segment, including: resampling, and removing, cyclically filling or segmenting the resampled audio segment to obtain an audio segment of fixed length.

[0010] S3. Input the fixed-length audio segment into the self-supervised pre-trained model with frozen parameters, extract the output of the multi-layer Transformer hidden layer, and obtain the multi-layer feature representation of the audio; based on different inter-layer weight parameters, perform weighted summation on the multi-layer feature representation to generate a high-dimensional acoustic feature sequence.

[0011] S4. Input the high-dimensional acoustic feature sequence into the static and dynamic branches of the dual-stream complementary feature encoder, and output the long-term global prosodic features and short-term local prosodic features of the audio segment respectively. Then, splice and fuse them to obtain the complementary enhanced acoustic feature representation.

[0012] S5. The complementary enhanced acoustic feature sequences are aggregated using an attention statistical pooling mechanism. Invalid pauses and background noise are suppressed by attention weights. The weighted mean and weighted standard deviation along the time dimension are calculated to generate a fixed-length emotion embedding vector.

[0013] S6. Construct a few-shot learning task based on the sentiment-annotated audio dataset collected in step S1. Divide the dataset into a support set and a query set. Use the fixed-length sentiment embedding vector as the sample feature representation. Calculate the mean of each sentiment sample in the support set as the prototype center. Calculate the predicted probability of each sentiment category based on the Euclidean distance between the query set samples and each prototype center. Take the sentiment category with the highest predicted probability as the sentiment classification result.

[0014] S7. Using the negative log-likelihood loss between the sentiment classification result and the true label as the optimization objective, the inter-layer weight parameters, the two-stream complementary feature encoder parameters, and the attention statistical pooling mechanism parameters are updated through the backpropagation algorithm until the loss is less than the preset value.

[0015] Further, step S1 includes: using a microphone that supports a 48kHz sampling rate to collect original audio segments of multiple subjects in emotional states of happiness, sadness, anger, neutrality, and surprise, wherein the frequency response range of the microphone is 20Hz to 20kHz.

[0016] Further, step S2 includes:

[0017] Endpoint detection algorithm is used to remove silent segments. The resampled signal is divided into frames using sliding window technology. The short-time energy of each frame is calculated. Frames with short-time energy below a preset dynamic threshold are marked as silent frames and removed. Non-silent frames are spliced ​​together to form a clean signal.

[0018] The system sets a target duration. For audio segments that exceed the target duration, a random trimming strategy is used to extract audio segments that meet the target duration. For audio segments that are shorter than the target duration, the system performs loop filling to obtain audio segments that meet the target duration.

[0019] Further, in step S3, the weighted summation of the multi-layer feature representation includes: initializing a set of learnable weight parameters, which correspond to the output features of the L+1 hidden layers from layer 0 to layer L; normalizing the weights using a normalized exponential function to obtain the fusion coefficients of each layer, and then weighted summing the features of each layer accordingly to generate the fused acoustic feature sequence.

[0020] Further, in step S4, the dual-stream complementary feature encoder includes:

[0021] The static branch is composed of an exponentially dilated temporal convolutional network, which contains multiple cascaded temporal convolutional blocks. Each temporal convolutional block is composed of a dilated convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence. The dilation rate of each temporal convolutional block increases exponentially with the network depth. The high-dimensional acoustic feature sequence is sequentially input into each temporal convolutional block. Each temporal convolutional block gradually expands its receptive field through dilation convolution to capture long-term global prosodic patterns in the audio signal with a span greater than a preset value. Finally, the last temporal convolutional block outputs the static features.

[0022] The dynamic branching introduces a low-rank supernetwork mechanism, specifically:

[0023] First, global average pooling is performed on the input feature sequence to obtain a global context vector representing the overall style of the current speech segment. A lightweight hypernetwork is then used to predict and generate mixing coefficients and channel scaling factors using the global context vector as input. Subsequently, the input features are deep convolved using the reconstructed dynamic convolution kernel, and the channel scaling factor is applied to scale the convolution result element by element to obtain dynamic features.

[0024] During feature fusion, the static features output by the static branch and the dynamic features output by the dynamic branch are concatenated along the channel dimension. The concatenated features are then processed by a convolutional layer with a kernel size of 1 to perform inter-channel information exchange and dimensionality reduction, resulting in the final dual-stream complementary feature sequence.

[0025] Further, step S5 includes:

[0026] The input feature sequence is processed by an attention network containing convolutional layers, hyperbolic tangent activation functions, and normalized exponential function normalization layers. For the feature vector at each time step, its attention score is calculated, and the attention scores of all time steps are converted into attention weights through the normalized exponential function. The larger the value of the attention weight, the higher the importance of the speech segment at that time in emotion recognition. Conversely, for background noise or invalid pause segments, the network automatically assigns lower weights to achieve soft noise suppression.

[0027] Based on the calculated attention weights, the feature sequences are weighted and aggregated in the time dimension, and the weighted mean and weighted standard deviation are calculated respectively. The weighted mean and weighted standard deviation are then concatenated in the feature channel dimension to construct the final fixed-length sentiment embedding vector.

[0028] Furthermore, the specific construction process of the few-shot learning task in step S6 is as follows:

[0029] S61. Define an N-way K-shot few-shot classification task; for each training or testing scenario, randomly select N sentiment categories from the sentiment-annotated audio dataset collected in S1, and for each selected category, select K labeled samples to form a support set to construct the feature prototypes of each sentiment category; at the same time, select Q unlabeled samples to construct a query set to evaluate the model performance and calculate the loss.

[0030] S62. For the selected nth sentiment category (n=1, ..., N), calculate the prototype center of the category using K samples from the support set. The prototype center is defined as the arithmetic mean of the embedding vectors of all samples in the support set.

[0031] Furthermore, the specific process of the task-based training method in step S7 is as follows:

[0032] S71. In each round of the task, a test sample is taken from the query set. First, the Euclidean distance between its embedding vector and all N sentiment prototypes is calculated. The smaller the distance, the higher the similarity between the sample and the sentiment prototype. Then, the Softmax function is used to convert the distance into the posterior probability of belonging to each sentiment category.

[0033] S72. During the training phase, the average negative log-likelihood loss of all samples in the query set is used as the objective function. The inter-layer fusion weights in step S3, the parameters of the dual-stream complementary feature encoder in step S4, and the parameters of the attention network of the attention statistical pooling mechanism in step S5 are updated end-to-end through the backpropagation algorithm. When the decrease of the average negative log-likelihood loss on the validation set is less than the preset threshold in a consecutive preset training round (10 times), the model is determined to have converged and the training is terminated.

[0034] This invention also provides a lightweight few-shot audio emotion recognition system for edge deployment, used to implement the method, comprising:

[0035] The audio data acquisition module integrates a microphone sensor interface, is configured with a data sampling rate of 48kHz, and inputs mono audio waveform signals. It is used to acquire raw audio segments from multiple subjects in different emotional states.

[0036] The signal preprocessing module is used to resample the original audio segment and remove, cyclically fill or segment the resampled audio segment to obtain an audio segment of fixed length.

[0037] An online inference module is used to input the fixed-length audio segment into the ensemble model for online inference; wherein, the ensemble model is composed of the self-supervised pre-trained model, the two-stream complementary feature encoder, and the attention statistical pooling mechanism.

[0038] The results output module is used to generate online inference results in real time at time intervals, which serve as sentiment prediction results.

[0039] The modules communicate with each other using a standardized data interface protocol.

[0040] Furthermore, the system is built on a deep learning framework based on Python.

[0041] The beneficial effects of this invention are as follows: First, the computational requirements for training and deployment are significantly reduced. By adopting an architecture of a distilled backbone network with frozen parameters plus a lightweight adapter, the computational requirements for model training and actual deployment are significantly reduced, providing an efficient foundation for applications in resource-constrained environments. Second, the feature representation is more refined and has high-precision personalized adaptation capabilities. By constructing a dual-stream complementary architecture, this invention can simultaneously capture global intonation and local details. In particular, the low-rank supernetwork mechanism overcomes the defect of traditional dynamic convolution being prone to overfitting under small sample conditions by compressing the parameter search space, and can adaptively adjust parameters according to the patient's specific pronunciation habits. Third, it effectively solves the pain point of data scarcity and enhances cross-user generalization ability. Through the use of a few-sample learning strategy, the model can quickly build a stable emotional prototype in the metric space with only a small number of labeled samples. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the overall process in a specific embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of audio preprocessing in a specific embodiment of the present invention;

[0044] Figure 3 This is a structural diagram of the overall model in a specific embodiment of the present invention;

[0045] Figure 4 This is a structural diagram of a dynamic branch in a specific embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of the attention statistical pooling mechanism in a specific embodiment of the present invention;

[0047] Figure 6 This is a schematic diagram of a small sample learning method in a specific embodiment of the present invention;

[0048] Figure 7 This is a multi-model comparison Pareto front plot in a specific embodiment of the present invention;

[0049] Figure 8 This is a visualization of the nonlinear dimensionality reduction features of Hypershift in a specific embodiment of the present invention;

[0050] Figure 9 This is a cross-subject 5-shot confusion matrix diagram of Hypershift in a specific embodiment of the present invention;

[0051] Figure 10 This is a cross-subject 10-shot confusion matrix diagram of Hypershift in a specific embodiment of the present invention;

[0052] Figure 11 This is a cross-subject 5-shot thermogram of Hypershift in a specific embodiment of the present invention;

[0053] Figure 12 This is a cross-subject 10-shot thermogram of Hypershift in a specific embodiment of the present invention;

[0054] Figure 13 This is a data flow architecture diagram of an audio emotion assessment system in a specific embodiment of the present invention. Detailed Implementation

[0055] The following embodiments of the invention, in conjunction with the accompanying drawings, provide detailed descriptions of the implementation scheme, data processing, model construction and training, and performance verification, further clarifying the application areas, design concepts, and technical solutions of the invention.

[0056] This embodiment relates to a lightweight, few-shot audio emotion recognition method for edge deployment, such as... Figure 1 As shown, it includes the following steps:

[0057] Step S1: Through audio acquisition experiments, using a microphone supporting a 48kHz sampling rate, original audio clips were collected from multiple subjects under emotional states of happiness, sadness, anger, neutrality, and surprise. It was ensured that subjects read standard text aloud in a uniform context to guarantee that the emotional characteristics of the data were clearly identifiable and the content consistent. Neutrality is defined as an emotional state without a clearly positive or negative bias.

[0058] In this embodiment, to reduce potential interference factors during audio signal acquisition, five healthy subjects were recruited during the experimental preparation phase. All participants had no history of unclear speech or oral-related diseases. To ensure consistent experimental conditions, all subjects were required to read the same standard sentences aloud to eliminate the impact of different speech content on emotion recognition.

[0059] The audio acquisition device is a smartphone or tablet that supports audio input at a sampling rate of 48 kHz. Its built-in microelectromechanical system (MEMS) microphone has omnidirectional pickup characteristics and a frequency response range of 20 Hz to 20 kHz, covering the audible sound range. During the acquisition process, the presence of ambient background noise (such as fan noise or distant human voices) is permitted; this noise will be preserved as background noise to improve the model's robustness and generalization ability in real-world noisy environments.

[0060] To construct a basic emotion recognition model for few-shot learning scenarios, audio data was collected from each subject for five typical emotion categories: happiness, sadness, anger, neutrality, and surprise. Fifty audio samples were collected for each emotion category, totaling 250 raw audio data to form the experimental dataset.

[0061] Step S2: Resample the acquired raw audio data, use the endpoint detection algorithm to remove silent segments, use loop filling for audio that is shorter than the target duration, and use a random cropping strategy to divide long audio into audio segments of fixed length.

[0062] In this embodiment, combined with Figure 2 The audio preprocessing flow shown first checks the format of the input audio, taking into account the diversity of acquisition terminals. If the input is in a compressed format such as MP3, AAC, or M4A, it is decompressed and transcoded into a unified lossless WAV format using an audio decoding engine. Then, the WAV data is read and the number of channels is checked; if it is multi-channel, it is mixed into mono using average amplitude. Finally, to match the back-end model, a polyphase filter is applied to uniformly resample the sampling rate to 16kHz, resulting in a standardized digital signal sequence. .

[0063] Since recordings often contain invalid operation tones or long silences at the beginning and end, a short-time energy-based endpoint detection algorithm is used for cleaning. The resampled signal is divided into frames with a frame length of 20ms (320 sampling points) and a frame shift of 10ms (160 sampling points). The short-time energy of each frame is calculated. And set dynamic thresholds (in The maximum short-time energy of the audio signal is 30dB (a commonly used empirical value). Frames below this threshold are marked as background silence and discarded. The remaining valid audio frames are spliced ​​together in time sequence to form a clean signal.

[0064] To adapt to the model input, a target duration is set. For continuous long audio segments, a random pruning strategy is employed during both the training and testing phases, randomly selecting 3-second segments from the effective speech range to enhance the model's adaptability to different emotional segments. For extremely short voice commands, a cyclic padding strategy is used, self-copying and splicing the signal until it exceeds 3 seconds before truncating it to maintain the continuity of the spectral texture. Due to varying gain settings across different devices, the recording volume varies significantly. Finally, peak normalization is performed on the signal. , To prevent extremely small constants with a denominator of zero, the amplitudes of all samples are uniformly scaled to the range of [-1.0, 1.0] to eliminate bias caused by the recording gain of the device.

[0065] Step S3: Use a parameter-frozen self-supervised pre-trained model as a feature extractor. Input the preprocessed audio segment into the parameter-frozen self-supervised pre-trained model to extract the output of multiple Transformer hidden layers, thereby obtaining multi-layer feature representations of the audio.

[0066] In this embodiment, as Figure 3 As shown in the overall model structure diagram, to meet the stringent requirements of edge terminals for low latency and low power consumption, a lightweight distillation model, DistilWavLM, is used as the acoustic feature extractor. All parameters of this lightweight backbone network are set to an untrainable state, serving only as an efficient feature extractor, further reducing training memory usage.

[0067] The normalized audio segment from step S2 is input into a lightweight backbone network. This lightweight backbone network consists of a front-end CNN encoding layer and a back-end stacked... Layered Transformer structure composition (in this embodiment) (The value is 6). Let the first... The output features of the layer are Its dimensions are Among them, batch size The time step is 32. The feature dimension is 300. It is 768.

[0068] Step S4: Based on the self-supervised pre-trained model with the frozen parameters, introduce learnable inter-layer weight parameters, perform weighted summation on the multi-layer features, and generate a high-dimensional acoustic feature sequence.

[0069] In this embodiment, the different layers of the lightweight backbone network in step S3 have different features: shallow layers emphasize signal details, while deep layers emphasize sentiment and semantics. To compensate for the subtle information loss that may result from model compression, this embodiment introduces an adaptive inter-layer fusion mechanism. First, a set of learnable weight parameters is initialized. ,correspond Layer features. Among them, The number of Transformer layers in the distilled lightweight backbone network (in this embodiment, Distil-Wavlm has a total of 6 Transformer layers, plus the 0th embedding layer, therefore...) The number of feature layers to be fused is Output features of each layer The dimension is B is the batch size (32 in this embodiment), N is the number of time steps (300 in this embodiment, corresponding to 3 seconds of audio, 16kHz sampling rate, and 10ms frame shift), and D=768 is the output dimension of the distillation target model. For the first Learnable weight parameters corresponding to the layer ( Secondly, the fusion coefficients are obtained by normalizing the weights using the Softmax function. And perform weighted summation on the features of each layer. As shown in equations (1) and (2) below:

[0070] ;

[0071] .

[0072] in, satisfy and In this way, the model can dynamically select the most critical feature layer combinations for the current emotion recognition task.

[0073] Since the output dimension D=768 of the distillation model does not match the backend encoder, a projection layer is set. A one-dimensional convolutional layer with a kernel size of 1 is used to fuse the features. The channel dimension from Mapped to a target dimension of 256. The final dimension is... Compact acoustic feature sequence Where B=32, N=300 are the time steps, and 256 is the input dimension of the back-end encoder, which serves as the input to the subsequent dual-stream complementary encoder.

[0074] Step S5: Construct a dual-stream complementary feature encoder, inputting the high-dimensional acoustic feature sequence into the static branch and the dynamic branch respectively. The static branch is used to extract long-term global prosodic features of the audio segment, while the dynamic branch is used to extract short-term local emotional cues of the audio segment. The outputs of the two branches are concatenated and fused to obtain a complementary and enhanced acoustic feature representation.

[0075] In this embodiment, the fused acoustic feature sequence output from step S4 is input into a dual-stream complementary feature encoder. The static branch uses a temporal convolutional network with exponentially growing dilation rate, while the dynamic branch uses a low-rank supernetwork to predict the displacement basis mixing coefficients and reconstruct a sample-adaptive dynamic convolutional kernel. Long-term global prosodic features and short-term local emotional cues of the audio are extracted separately, and the outputs of the two branches are concatenated and fused.

[0076] The static branch consists of an exponentially dilated temporal convolutional network designed to capture long-span prosodic patterns in audio signals, such as the fluctuations in intonation. Its parameters are fixed after training and remain unchanged during inference to provide stable baseline features. In this embodiment, the static branch consists of three cascaded temporal convolutional blocks, each containing dilated convolutional layers, batch normalization layers, and ReLU activation functions. The number of output channels in each layer remains at 256, consistent with the input dimension. To expand the receptive field without significantly increasing the number of parameters, the dilation rate of each convolutional kernel increases exponentially with network depth. Let the layer index be... , No. The expansion rate of the layer is ,follow The growth pattern is as follows (i.e., the first layer d=1, the second layer d=2, the third layer d=4). For the first layer in the input sequence... time step ,in, , Given the length of the input sequence, its output is... Defined by the following formula (3):

[0077] ;

[0078] in, In this embodiment, K=9, which is the kernel size. The learnable convolutional kernel weights at the k-th position have the same dimension as the input features. Once the training is completed, it remains unchanged; For the input sequence with index The feature vectors are zero-padded when the index exceeds the range [1,N]. Let be the output feature at time step t, with dimension 1. By stacking, this branch can effectively cover hundreds of milliseconds of context, extracting long-term global prosodic features. .

[0079] Dynamic branching introduces a low-rank supernetwork mechanism, the structure of which is as follows: Figure 4 As shown, the aim is to generate sample-adaptive convolutional kernels in real time based on the specific content of the input speech, in order to keenly capture brief local emotional shifts (such as a momentary sob or exclamation). First, the input feature sequence... Perform global average pooling to obtain a global context vector representing the overall style of the current speech segment. As shown in equation (4):

[0080] .

[0081] Using lightweight hypernetworks As input, predict and generate mixing coefficients. and channel scaling factor As shown in equation (5):

[0082] ;

[0083] in, This is a lightweight network consisting of two fully connected layers: the first layer has an input dimension C=256 and an output dimension of 64, followed by a ReLU activation function; the second layer has an input dimension of 64 and an output dimension of 64. The hypernetting output is split according to the following rules: (Previous...) Reshape the dimension into a mixing coefficient matrix Then C=256 dimensions are used as the channel scaling factor. The number of parameters was reduced from approximately 589,000 in direct mapping to approximately 164,000, a reduction of about 72%, effectively suppressing the risk of overfitting in small sample training.

[0084] In this embodiment, the number of displacement bases Set to 8, share displacement basis It is generated using random initialization.

[0085] Using the mixing coefficient For a set of learnable shared displacement bases By performing linear weighted combination, a dynamic convolution kernel for the current channel is reconstructed. As shown in equation (6):

[0086] ;

[0087] in, For the number of displacement bases, Mixing coefficient matrix The element in the c-th row and j-th column, For shared displacement basis The j-th basis vector in the vector is used, and all channels share the same set of displacement bases. Indicates the first Dynamic convolution kernels for each channel.

[0088] Finally, the reconstructed dynamic convolution kernel is used to perform a depthwise convolution operation on the input features (denoted as ). Then, a scaling factor is applied for element-wise multiplication to obtain the dynamic features. As shown in equation (7):

[0089] .

[0090] in, This represents a depthwise convolution operation, where each channel uses its own reconstructed convolution kernel independently. Perform one-dimensional convolution, without summing or interaction between channels; This indicates element-wise multiplication. Broadcasting is performed along the channel dimension, i.e., all time steps of the c-th channel are multiplied by 1. ; The input feature sequence.

[0091] Feature fusion combines static features With dynamic characteristics The system performs concatenation along the channel dimension and then performs inter-channel information exchange through a convolutional dimensionality reduction operation with a kernel size of 1, resulting in the final dual-stream complementary feature sequence. As shown in equation (8):

[0092] .

[0093] in, This is for channel splicing operations. It is a one-dimensional convolutional layer with a kernel size of 1, 512 input channels, and 256 output channels, realizing dimensionality reduction and information interaction between channels.

[0094] Step S6: The complementary enhanced acoustic feature sequences are aggregated using an attention-based statistical pooling mechanism. Attention weights are used to suppress invalid pauses and background noise. The weighted mean and weighted standard deviation along the time dimension are calculated to generate a fixed-length sentiment embedding vector.

[0095] In this embodiment, the following is adopted: Figure 5 The attention-based statistical pooling mechanism shown aggregates the two-stream complementary feature sequences output from step S5. This mechanism achieves noise suppression and sentiment volatility modeling through adaptive weight allocation. The specific implementation process is as follows:

[0096] Although the dual-stream complementary feature sequence output in step S5 incorporates spatiotemporal information, it still retains the temporal dimension and is considered variable-length sequence data. To transform the temporal features into fixed-length vectors suitable for classification and automatically focus on segments of speech containing strong emotions (such as stress or crying sounds), this embodiment introduces a temporal attention mechanism. First, the input feature sequence is processed through an attention network. In this embodiment, the attention network consists of a one-dimensional convolutional layer with a kernel size of 1, a Tanh activation function, and a Softmax normalization layer. Let the input feature sequence be... , , Attention networks map features to attention dimensions. Calculate the first Attention score at each time step and normalized weights As shown in equations (9) and (10) below:

[0097] ;

[0098] ;

[0099] in, and , The total number of parameters is approximately 33,000. The larger the value, the more important the speech segment at that moment is in emotion recognition; conversely, for background noise or invalid pause segments, the network will automatically assign lower weights, thereby achieving soft noise suppression.

[0100] Based on the calculated attention weights, the weighted mean (first-order statistic) and weighted standard deviation (second-order statistic) of the feature sequences over the time dimension are calculated respectively. Weighted mean The average emotional tone used to characterize speech is calculated using the following formula (11):

[0101] .

[0102] Weighted standard deviation Standard deviation is used to explicitly model the temporal volatility of speech signals. Compared to traditional pooling that only uses the mean, standard deviation can effectively capture the dynamic range of emotional expression (e.g., calm speech has smaller fluctuations, while angry or excited speech has larger fluctuations), thereby enhancing the model's ability to discriminate emotional arousal. The calculation formula is shown in equation (12) below:

[0103] ;

[0104] in, To prevent unstable minimum values ​​in numerical calculations.

[0105] The calculated weighted mean vector with weighted standard deviation vector The feature channels are concatenated to generate the final fixed-length sentiment embedding vector. As shown in equation (13):

[0106] .

[0107] If the input feature dimension is C (256 in step S5), then the final embedded vector dimension after concatenation is 2C (i.e., 512). This vector simultaneously contains the central tendency and dispersion of sentiment, providing a complete feature representation for subsequent prototype classification.

[0108] Step S7: Construct a few-shot learning task based on the sentiment-annotated audio dataset collected in Step S1, and divide the samples in the dataset into a support set and a query set; use the fixed-length sentiment embedding vector as the sample feature representation, calculate the mean of each type of sentiment sample in the support set as the prototype center, and perform sentiment classification by Euclidean distance between the query set samples and each prototype center.

[0109] In this embodiment, an N-way K-shot few-shot classification task is defined. Specifically, N=5 (corresponding to five emotion categories: happy, sad, angry, neutral, and surprised). For each emotion category, K=5 labeled samples are randomly selected to construct a support set, used to build the feature prototype for that category; simultaneously, Q=5 samples are randomly selected from each emotion category to form a query set, used to evaluate model performance and calculate loss. Thus, the support set contains N×K=25 samples, and the query set contains N×Q=25 samples. All samples are mapped to 2C-dimensional emotion embedding vectors through the feature extraction process described in steps S1 to S6.

[0110] like Figure 6 As shown, for the selected nth sentiment category (n=1,...,N), the prototype center of this category is calculated using K samples from the support set. The prototype center is defined as the arithmetic mean of the embedding vectors of all samples in the support set. The embedding vector of the kth sample belonging to the nth category in the support set is denoted as... Then the emotional prototype of the nth class The calculation formula is shown in equation (14) below:

[0111] .

[0112] For any test sample in the query set First, calculate its embedding vector. With all N emotional archetypes The Euclidean distance between the two samples. The smaller the distance, the higher the similarity between the sample and the emotional archetype. The distance formula is shown in equation (15) below:

[0113] .

[0114] Subsequently, the Euclidean distance is converted into posterior probabilities belonging to each sentiment category using the Softmax function. To convert the distance metric into a similarity metric, the negative of the distance is used as the input to the exponent. The probability that sample x belongs to category n. As shown in equation (16):

[0115] ;

[0116] The denominator is the sum of the distance exponents between sample x and all N prototypes, used to normalize the probability distribution. The category with the highest probability is taken as the final sentiment classification prediction result.

[0117] Step S8: The self-supervised pre-trained model, the two-stream complementary feature encoder, and the attention statistical pooling mechanism together constitute an ensemble model. A task-based training mechanism is adopted. In the few-shot learning task, the predicted probability is calculated based on the Euclidean distance. The negative log-likelihood loss between the predicted result and the true label is used as the optimization objective. The model parameters are updated through the backpropagation algorithm until the model converges and the test is completed.

[0118] In this embodiment, the self-supervised pre-trained model serves as a front-end feature extractor, used to capture basic temporal or spatial representations from the original input signal. The dual-stream complementary feature encoder is connected to the output of the self-supervised pre-trained model and contains two parallel feature processing streams, which perform differential encoding on features of different modalities or scales to achieve complementary feature enhancement. After the dual-stream encoder, the attention statistical pooling mechanism is introduced, which performs weighted aggregation on the feature maps output by the dual streams. The entire assembly forms the final ensemble model.

[0119] During the training phase, the negative log-likelihood loss (NLLLoss) is used as the objective function. Using the posterior probability calculated in step S7, the loss function is defined as shown in equation (17):

[0120] ;

[0121] in, To query the total number of samples in the set (in this embodiment) ), For query samples The true label.

[0122] The system updates the inter-layer fusion weights in step S4, the dual-stream encoder parameters in step S5, and the attention network parameters in step S6 in an end-to-end manner using the backpropagation algorithm. The training hyperparameters are configured as follows: the maximum number of training epochs is set to 30; 200 small sample learning tasks described in step S7 are randomly sampled from the dataset in each epoch; an early stopping mechanism is adopted, using the validation set loss as the monitoring metric; if the validation loss does not show a significant decrease for 10 consecutive epochs, training is terminated early to suppress overfitting; the optimizer is Adam, with an initial learning rate of 0.0001 and a weight decay coefficient of 0.0001.

[0123] To verify the effectiveness and superiority of the method proposed in this embodiment, relevant experimental analyses were conducted, as follows:

[0124] First, regarding the comprehensive evaluation of model performance and complexity. For example... Figure 7The multi-model comparison Pareto front plot shown demonstrates that the proposed method achieves superior sentiment classification accuracy while maintaining low parameter count and computational overhead, placing it on the Pareto optimal boundary. This fully validates the deployment advantages of this method in scenarios where edge devices have limited computing power.

[0125] Secondly, regarding feature representation capabilities. For example... Figure 8 As shown in the nonlinear low-dimensional feature visualization diagram, after the low-rank supernetwork dual-stream encoder and attention pooling processing of the present invention, the emotion features of different categories exhibit intra-class compactness and inter-class separation characteristics in the feature space, proving the effectiveness of the present invention in constructing the emotion prototype space under small sample conditions.

[0126] Finally, regarding the assessment of cross-individual generalization ability, such as... Figure 9 and Figure 10 The figures shown are the confusion matrices of the model under the 5-shot and 10-shot settings across subjects, respectively; Figure 11 and Figure 12 The figures shown are heatmaps for 5-shot and 10-shot data across subjects, respectively. The results demonstrate that even with unfamiliar subjects and a very small support set, this system can accurately identify various emotional states, maintaining high accuracy, particularly in distinguishing easily confused high-arousal emotions. This fully validates the superior performance of the low-rank supernetwork mechanism and small-sample strategy of this invention in overcoming individual feature bias and achieving personalized, high-precision adaptation.

[0127] Step S9: Construct a modular real-time emotion monitoring system architecture based on a deep learning framework using Python, integrating modules for audio data acquisition, signal preprocessing, online inference, and result output, and embedding the trained and validated integrated model parameters into the online inference module.

[0128] In this embodiment, to realize the real-time application of a lightweight, few-sample audio emotion recognition method for edge deployment, a modular real-time emotion monitoring system architecture is constructed based on a deep learning framework in Python. For example... Figure 13 As shown, the system architecture is divided into four core modules: audio data acquisition module, signal preprocessing module, online inference module, and result output module. The modules communicate with each other using a standardized data interface protocol to ensure efficient transmission of real-time data streams and low-latency compatibility between modules.

[0129] The audio data acquisition module integrates a microphone sensor interface, configured with a data sampling rate of 48 kHz, and the input data is a mono audio waveform signal. This module achieves real-time capture of continuous audio streams by calling the operating system's underlying audio API and pushes the raw audio data to the downstream processing module in the form of data blocks.

[0130] The signal preprocessing module follows the same signal processing flow as step S2, including: resampling the input 48kHz audio data to 16kHz using a polyphase filter; employing a short-time energy-based endpoint detection algorithm and de-mute technology to remove invalid silence segments; using a sliding window strategy with a window size of 320 sampling points and a sliding step of 160 sampling points, combined with a random pruning strategy to align the audio data into a standardized audio data window. For short audio segments less than 3 seconds long, a cyclic padding strategy is used to pad the length.

[0131] In the online inference module, the trained and validated ensemble model is serialized into ONNX format and loaded and its graph structure optimized in the C++ online inference module. By solidifying model weights and performing low-precision quantization inference (such as INT8 quantization), the computational structure is optimized, reducing memory bandwidth consumption and inference latency, thereby achieving efficient real-time sentiment prediction. Model parameters include the inter-layer fusion weights in step S4, the dual-stream encoder parameters in step S5, and the attention network parameters in step S6.

[0132] The results output module receives the sentiment category and posterior probability output by the online inference module and pushes the results to the upper-layer application through a standardized interface.

[0133] Step S10: Collect and store audio data in real time, input the audio signal into the modular emotion system architecture, and use a multi-threaded buffer mechanism to process the data stream in real time, execute the signal processing and online inference modules, and dynamically output the prediction results of the emotion category;

[0134] In this embodiment, after the system starts, the audio data acquisition module acquires audio waveform signals in real time and stores the raw data stream in a circular buffer. The circular buffer adopts a first-in-first-out structure, and its capacity is preset to buffer 10 seconds of audio data to prevent data frame loss in high-concurrency scenarios.

[0135] The system employs a multi-threaded buffering mechanism, allocating independent threads to perform signal preprocessing and online inference tasks. Specifically, one thread reads raw audio data blocks from the circular buffer and performs preprocessing operations such as real-time resampling, endpoint detection, demute, and sliding window segmentation; another thread inputs the preprocessed, standardized audio segments into the ensemble model for online inference and prediction. Data exchange between the two threads is achieved through shared memory, and synchronization and data transfer between threads are implemented using mutexes and condition variables to avoid resource contention and deadlock.

[0136] The signal preprocessing thread performs real-time resampling and noise reduction on the audio waveform signal, and uses a sliding window strategy combined with random pruning to generate standardized input segments. The generated segments are passed to the online inference thread via shared memory. The online inference thread loads the fixed ONNX format model, performs low-precision quantization inference, calculates the sentiment embedding vector of the input segment, and calculates the Euclidean distance with the sentiment prototype centers that support centralized pre-computation. The posterior probability distribution of the sentiment category is output through the Softmax function. The prediction results are transmitted to the results output module through a standardized interface.

[0137] In summary, this invention addresses the core challenges in audio emotion recognition tasks, such as poor model generalization ability, difficulty in personalized adaptation, and high deployment costs caused by the extreme scarcity of data in vertical scenarios like medical rehabilitation, huge differences in individual pronunciation characteristics, and limited computing power of edge devices. It proposes a lightweight, few-sample audio emotion recognition method and system for edge deployment. This invention constructs a scenario-based small-sample task, achieving effective feature organization and rapid calibration of target emotional prototypes with minimal labeled data. To balance feature representation depth and model lightweighting, it innovatively employs a parameter-freezing self-supervised pre-trained large audio model combined with an adaptive inter-layer mapping mechanism to construct the feature extraction backbone. Furthermore, it pioneeringly designs a dual-stream complementary encoder incorporating dilated temporal convolution and a low-rank supernetwork to accurately capture long-term globally stable prosody and short-term local personalized mutations, effectively overcoming feature distribution shifts between individuals. Simultaneously, it introduces a second-order statistical pooling mechanism for attention, explicitly modeling the intensity of emotional fluctuations while suppressing environmental noise interference. Finally, it achieves high-precision classification by calculating the Euclidean distance between the test sample and the center of the emotional prototype in the metric space, and uses negative log-likelihood loss for end-to-end optimization of a small number of adaptation parameters. Ultimately, this invention successfully deploys the lightweight model on edge computing devices and implements a visualized real-time personalized emotion monitoring system using frameworks such as PyQt or Streamlit. This invention has achieved significant breakthroughs in emotion assessment accuracy under small sample conditions, cross-individual generalization ability, and operating efficiency of resource-constrained devices. It completely breaks away from the dependence of traditional deep learning on large-scale, high-quality labeled data, and provides highly reliable, low-latency technical support for fields such as specific pathological pronunciation analysis, real-time monitoring of emotions in medical rehabilitation, and personalized human-computer interaction. It has clear and broad application prospects and value.

[0138] The above embodiments describe one implementation of the present invention. Those skilled in the art can make various adjustments, modifications, and improvements to the present invention. All adjustments, modifications, and improvements made within the scope of the core principles of the present invention are considered to be within the scope of the present invention.

Claims

1. A lightweight, few-shot audio emotion recognition method for edge deployment, characterized in that, Includes the following steps: S1. Collect original audio clips from multiple subjects under different emotional states, label them with emotional categories, and form an emotionally labeled audio dataset. S2. Preprocess the original audio segment to obtain an audio segment of fixed length; S3. Input the fixed-length audio segment into the self-supervised pre-trained model with frozen parameters to obtain multi-layer feature representations of the audio; based on different inter-layer weight parameters, sum the multi-layer feature representations by weight to generate a high-dimensional acoustic feature sequence. S4. Input the high-dimensional acoustic feature sequence into the static and dynamic branches of the dual-stream complementary feature encoder, and output the long-term global prosodic features and short-term local prosodic features of the audio segment respectively. Then, splice and fuse them to obtain the complementary enhanced acoustic feature representation. S5. Using an attention-based statistical pooling mechanism, the complementary enhanced acoustic feature sequences are aggregated, and the weighted mean and weighted standard deviation along the time dimension are calculated to generate a fixed-length sentiment embedding vector. S6. Construct a few-shot learning task based on the sentiment-annotated audio dataset and divide it into a support set and a query set; Using fixed-length sentiment embedding vectors as sample feature representations, the mean of each sentiment sample in the support set is calculated as the prototype center. The predicted probability of each sentiment category is calculated based on the Euclidean distance between the query set samples and each prototype center. The sentiment category with the highest predicted probability is taken as the sentiment classification result. S7. Using the negative log-likelihood loss between the sentiment classification result and the true label as the optimization objective, the inter-layer weight parameters, the two-stream complementary feature encoder parameters, and the attention statistical pooling mechanism parameters are updated through the backpropagation algorithm until the loss is less than the preset value.

2. The lightweight few-shot audio emotion recognition method for edge deployment as described in claim 1, characterized in that, Step S1 includes: using a microphone that supports a 48kHz sampling rate to collect original audio clips of multiple subjects in emotional states of happiness, sadness, anger, neutrality, and surprise, wherein the frequency response range of the microphone is 20Hz to 20kHz.

3. The lightweight few-shot audio emotion recognition method for edge deployment as described in claim 1, characterized in that, Step S2 includes: The original audio segment is resampled; Endpoint detection algorithm is used to remove silent segments. The resampled signal is divided into frames using sliding window technology. The short-time energy of each frame is calculated. Frames with short-time energy below a preset dynamic threshold are marked as silent frames and removed. Non-silent frames are spliced ​​together to form a clean signal. The system sets a target duration. For audio segments that exceed the target duration, a random trimming strategy is used to extract audio segments that meet the target duration. For audio segments that are shorter than the target duration, the system performs loop filling to obtain audio segments that meet the target duration.

4. The lightweight few-shot audio emotion recognition method for edge deployment as described in claim 1, characterized in that, In step S3, the weighted summation of the multi-layer feature representation includes: initializing a set of learnable weight parameters, which correspond to the output features of the L+1 hidden layers from layer 0 to layer L; normalizing the weights using a normalized exponential function to obtain the fusion coefficients of each layer, and then performing a weighted summation of the features of each layer accordingly to generate the fused acoustic feature sequence.

5. The lightweight few-shot audio emotion recognition method for edge deployment as described in claim 1, characterized in that, In step S4, the dual-stream complementary feature encoder includes: The static branch is composed of an exponentially dilated temporal convolutional network, which contains multiple cascaded temporal convolutional blocks. Each temporal convolutional block is composed of a dilated convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence. The dilation rate of each temporal convolutional block increases exponentially with the network depth. The high-dimensional acoustic feature sequence is sequentially input into each temporal convolutional block. Each temporal convolutional block gradually expands its receptive field through dilation convolution to capture long-term global prosodic patterns in the audio signal with a span greater than a preset value. The static feature is output by the last temporal convolutional block. The dynamic branching introduces a low-rank supernetwork mechanism, specifically: First, global average pooling is performed on the input feature sequence to obtain a global context vector representing the overall style of the current speech segment. A lightweight hypernetwork is then used to predict and generate mixing coefficients and channel scaling factors using the global context vector as input. Subsequently, the input features are deep convolved using the reconstructed dynamic convolution kernel, and the channel scaling factor is applied to scale the convolution result element by element to obtain dynamic features. During feature fusion, the static features output by the static branch and the dynamic features output by the dynamic branch are concatenated along the channel dimension. The concatenated features are then processed by a convolutional layer with a kernel size of 1 to perform inter-channel information exchange and dimensionality reduction, resulting in the final dual-stream complementary feature sequence.

6. The lightweight few-shot audio emotion recognition method for edge deployment as described in claim 1, characterized in that, Step S5 includes: The input feature sequence is processed by an attention network containing convolutional layers, hyperbolic tangent activation functions, and normalized exponential function normalization layers. For the feature vector at each time step, its attention score is calculated, and the attention scores of all time steps are converted into attention weights through the normalized exponential function. The larger the value of the attention weight, the higher the importance of the speech segment at that time in emotion recognition. Conversely, for background noise or invalid pause segments, the network automatically assigns lower weights to achieve soft noise suppression. Based on the calculated attention weights, the feature sequences are weighted and aggregated in the time dimension, and the weighted mean and weighted standard deviation are calculated respectively. The weighted mean and weighted standard deviation are then concatenated in the feature channel dimension to construct the final fixed-length sentiment embedding vector.

7. The lightweight few-shot audio emotion recognition method for edge deployment as described in claim 1, characterized in that, The specific construction process of the few-shot learning task in step S6 is as follows: S61. Define an N-way K-shot few-shot classification task; for each training or testing scenario, randomly select N emotion categories from the emotion-annotated audio dataset, and for each selected category, select K labeled samples to form a support set to construct the feature prototype of each emotion category; at the same time, select Q unlabeled samples to construct a query set to evaluate the model performance and calculate the loss. S62. For the selected nth sentiment category, n=1, ..., N, calculate the prototype center of the category using K samples from the support set. The prototype center is defined as the arithmetic mean of the embedding vectors of all samples in the support set.

8. The lightweight few-shot audio emotion recognition method for edge deployment as described in claim 1, characterized in that, The specific process of the task-based training method in step S7 is as follows: S71. In each round of the task, take one sample to be tested from the query set. First, calculate the Euclidean distance between its embedding vector and all N sentiment prototypes. The smaller the distance, the higher the similarity between the sample and the sentiment prototype. Subsequently, the Softmax function is used to convert the distance into the posterior probability of belonging to each sentiment category; S72. During the training phase, the average negative log-likelihood loss of all samples in the query set is used as the objective function. The inter-layer fusion weights in step S3, the parameters of the dual-stream complementary feature encoder in step S4, and the attention network parameters of the attention statistical pooling mechanism in step S5 are updated end-to-end through the backpropagation algorithm. When the decrease of the average negative log-likelihood loss on the validation set is less than the preset threshold in consecutive preset training rounds, the model is determined to have converged and training is terminated.

9. A lightweight, few-shot audio emotion recognition system for edge deployment, used to implement the method described in any one of claims 1-8, characterized in that, include: The audio data acquisition module integrates a microphone sensor interface, is configured with a data sampling rate of 48kHz, and inputs mono audio waveform signals. It is used to acquire raw audio segments from multiple subjects in different emotional states. The signal preprocessing module is used to resample the original audio segment and remove, cyclically fill or segment the resampled audio segment to obtain an audio segment of fixed length. An online inference module is used to input the fixed-length audio segment into the ensemble model for online inference; wherein, the ensemble model is composed of the self-supervised pre-trained model, the two-stream complementary feature encoder, and the attention statistical pooling mechanism. The results output module is used to generate online inference results in real time at time intervals, which serve as sentiment prediction results. The modules communicate with each other using a standardized data interface protocol.

10. The lightweight few-shot audio emotion recognition system for edge deployment according to claim 9, characterized in that, The system is built on a deep learning framework based on Python.

Images