A visual dominant multi-modal learning emotion recognition method for small sample music

Abstract

The present application relates to the technical field of sound emotion recognition, and provides a visual dominant multi-modal learning emotion recognition method for small sample music. The method comprises the following steps: synchronously extracting visual representation data and acoustic representation data from original music audio, and respectively encoding into high-dimensional visual feature vectors and low-dimensional acoustic feature vectors; based on an asymmetric feature fusion mechanism with visual representation as the dominant and acoustic semantics as the auxiliary, the two types of feature vectors are fused into a joint feature vector; the joint feature vector is input into a multi-task learning network to simultaneously perform a music emotion classification main task and a VAD dimension regression auxiliary task, and the network is optimized accordingly. The present application effectively combines the time-frequency texture and music theory semantic information of music through the visual dominant asymmetric fusion, and significantly improves the emotion recognition accuracy, the ability to distinguish fuzzy emotion categories and the model generalization performance in the small sample scene by using the fine-grained supervision of the VAD regression task.

Description

Technical Field

[0001] This invention relates to the field of voice emotion recognition technology, and more specifically, to a visually-driven multimodal learning emotion recognition method for small-sample music. Background Technology

[0002] In traditional machine learning, voice emotion recognition primarily relies on manually designed and extracted acoustic features, such as Mel-frequency cepstral coefficients (MFCC), fundamental frequency (F0), energy, speech rate, and prosodic features. These features are then fed into traditional machine learning classifiers such as Support Vector Machines (SVM), Random Forests, or Gaussian Mixture Models (GMM) for emotion discrimination. The advantage of this method is its relatively lightweight model and low computational cost. However, its fundamental limitation lies in the limited representational power of manually designed features, resulting in insufficient generalization ability and robustness when facing environmental noise, differences between speakers, and cross-scenario applications.

[0003] With the development of deep learning technology, models such as Convolutional Neural Networks (CNN), Long Short-Term Memory Networks (LSTM / Gated Recurrent Units (GRU), and Transformers have been introduced to achieve end-to-end feature learning and classification from raw audio or basic spectral features. Among them, CNN excels at capturing local time-frequency patterns in spectrograms, LSTM / GRU is suitable for modeling long temporal dependencies in speech signals, while Transformers capture global correlations of features through a self-attention mechanism. Although such single-modal recognition methods have significantly improved recognition accuracy, they still generally suffer from problems such as high-dimensional feature redundancy, loss of some fine temporal information, weak anti-interference ability in complex acoustic environments, and overfitting and poor generalization ability in scenarios with small samples or imbalanced data.

[0004] To compensate for the lack of semantic and contextual information in a single speech modality, current cutting-edge technologies are exploring the fusion of multiple modal information, such as speech acoustics, text semantics, and even facial expressions, to improve the ability to recognize complex emotions. However, multimodal fusion methods introduce new technical bottlenecks: First, it is difficult to achieve precise temporal alignment between data from different modalities (such as audio streams, text streams, and video streams); second, common fusion strategies such as feature-level concatenation or decision-level voting are often rigid and fail to fully consider the differences in the contribution of information from different modalities; in addition, constructing a multimodal model containing multiple sub-networks usually leads to a significant increase in the number of parameters and computational complexity, posing challenges to practical training and deployment.

[0005] Therefore, it is necessary to study a scheme that is more suitable for improving the accuracy of complex emotion recognition by utilizing multimodal information. Summary of the Invention

[0006] This invention addresses the technical problems existing in the prior art by providing a visually-driven multimodal learning-based emotion recognition method for small-sample music, which improves the accuracy of complex emotion recognition by combining visually-driven multimodal learning.

[0007] According to a first aspect of the present invention, a visually-dominated multimodal learning emotion recognition method for few-sample music is provided, comprising: Visual and acoustic representation data are extracted from the original music audio, and the visual and acoustic representation data are encoded to obtain visual feature vectors and acoustic feature vectors, respectively. Based on an asymmetric feature fusion mechanism that prioritizes visual representation and uses acoustic semantics as an auxiliary, the visual feature vector and the acoustic feature vector are fused into a joint feature vector. The joint feature vector is input into a multi-task learning network to simultaneously perform the main task of music emotion classification and the auxiliary task of VAD (Valence-Arousal-Dominance) dimension regression, so as to obtain the emotion classification result and the predicted values ​​of the three continuous dimensions of valence-arousal-dominance. The multi-task learning network is then optimized in reverse based on the emotion classification result and the predicted values ​​of the three continuous dimensions.

[0008] Based on the above technical solution, the present invention can also be improved as follows.

[0009] Optionally, visual representation data is extracted from the original music audio, and the visual representation data is encoded to obtain a visual feature vector, including: The raw music audio data is preprocessed to generate a Mel spectrogram of a specific size; The Mel spectrogram is converted into a three-channel RGB image using pseudo-color mapping to serve as the visual representation data; The visual representation data is input into a visual encoding network based on the Swing Transformer architecture; The visual coding network sequentially performs hierarchical feature extraction and downsampling on the visual representation data through its multiple layers; Obtain a visual feature vector of dimension 1×N1 from the final output layer of the visual coding network, where N1 is a preset visual feature dimension.

[0010] Optionally, the preprocessing of the original music audio data and the generation of a Mel spectrogram of a specific size includes: Preprocessing of the original music audio includes loudness normalization, resampling, and length alignment; The preprocessed audio is subjected to a short-time Fourier transform to obtain the spectrum; The spectrum is mapped to the Mel scale to generate a single-channel Mel spectrogram.

[0011] Optionally, acoustic representation data is extracted from the original music audio, and the acoustic representation data is encoded to obtain an acoustic feature vector, including: Extracting the underlying acoustic feature sequence from the original music audio; The low-level acoustic feature sequence is input into a pre-trained acoustic feature extraction network to obtain a high-level acoustic embedding sequence. The advanced acoustic embedding sequence is temporally aggregated to obtain a global acoustic representation vector. The global acoustic representation vector is linearly projected and normalized to obtain an acoustic feature vector of dimension 1×N2, where N2 is a preset acoustic feature dimension, and N2 <N1。

[0012] Optionally, the asymmetric feature fusion mechanism based on visual representation as the primary method and acoustic semantics as the secondary method fuses the visual feature vector and the acoustic feature vector into a joint feature vector, including: The acoustic feature vector is normalized so that the numerical distribution of the normalized acoustic feature vector matches the visual feature vector. The normalized acoustic feature vector is concatenated with the visual feature vector to obtain a preliminary fused feature vector, wherein the dimension of the visual feature vector is higher than the dimension of the normalized acoustic feature vector. The preliminary fused feature vector is input into a multilayer perceptron module for nonlinear transformation and feature interaction, and the joint feature vector is output. The multilayer perceptron module integrates a random deactivation layer.

[0013] Optionally, the multi-task learning network includes a classification head and a regression head that are configured in parallel and share the joint feature vector as input; wherein the classification head is used to output the emotion classification result, and the regression head is used to output the predicted values ​​of the three continuous dimensions of valence, arousal and control.

[0014] Optionally, the step of back-optimizing the multi-task learning network based on the emotion classification result and the predicted values ​​of the three continuous dimensions specifically includes: During training, the classification loss between the emotion classification result and the true emotion category label is calculated, and the regression loss between the predicted values ​​of the three continuous dimensions and the true VAD dimension coordinate labels is calculated. The network parameters of the multi-task learning network are optimized using the weighted sum of the classification loss and the regression loss as the total loss through the backpropagation algorithm.

[0015] Optionally, the classification loss is calculated using the cross-entropy loss function, and the regression loss is calculated using the mean squared error loss function; the true VAD dimension coordinate labels are obtained from a pre-established mapping relationship between discrete music emotion category labels and continuous VAD spatial coordinates.

[0016] Optionally, in the process of reverse optimization of the multi-task learning network, at least one optimization strategy is adopted, including hierarchical learning rate optimization, cosine annealing learning rate scheduling with a warm-up phase, label smoothing, and weight decay.

[0017] According to a second aspect of the present invention, a vision-dominated multimodal learning emotion recognition system for few-sample music is provided, comprising: The feature extraction and encoding module is used to extract visual representation data and acoustic representation data from the original music audio, and to encode the visual representation data and acoustic representation data respectively to obtain visual feature vectors and acoustic feature vectors. An asymmetric feature fusion module is used to fuse the visual feature vector and the acoustic feature vector into a joint feature vector based on an asymmetric feature fusion mechanism that is dominated by visual representation and assisted by acoustic semantics. The multi-task learning and optimization module is used to input the joint feature vector into the multi-task learning network to simultaneously perform the main task of music emotion classification and the auxiliary task of VAD dimension regression, to obtain the emotion classification result and the predicted values ​​of three continuous dimensions of valence-arousal-control, and to back-optimize the multi-task learning network based on the emotion classification result and the predicted values ​​of the three continuous dimensions.

[0018] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the processor is configured to implement the steps of the above-described visual-dominated multimodal learning emotion recognition method for few-sample music when executing a computer management program stored in the memory.

[0019] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer management program stored thereon, which, when executed by a processor, implements the steps of the above-described visual-dominated multimodal learning emotion recognition method for few-sample music.

[0020] This invention provides a visual-dominated multimodal learning emotion recognition method, system, electronic device, and storage medium for music with limited sample sizes. By simultaneously extracting and encoding visual and acoustic features from music audio, and then performing asymmetric fusion in a visual-dominated and acoustic-assisted manner, the fused features are finally classified for emotion and regressed in the VAD dimension using multi-task learning. This effectively improves the accuracy and generalization ability of music emotion recognition under limited sample conditions, and enhances the discriminative power for fuzzy emotion categories. Attached Figure Description

[0021] Figure 1 A flowchart of a vision-driven multimodal learning emotion recognition method for few-sample music provided by this invention; Figure 2 A schematic diagram of a vision-driven multimodal learning emotion recognition system architecture for few-sample music, provided for one embodiment; Figure 3 A block diagram of a vision-driven multimodal learning emotion recognition system for few-sample music provided by this invention; Figure 4 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention; Figure 5 This is a schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. Detailed Implementation

[0022] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0023] Figure 1 The flowchart of a vision-driven multimodal learning emotion recognition method for few-sample music provided by this invention is as follows: Figure 1 As shown, the method includes steps S1 to S3.

[0024] S1, extract visual representation data and acoustic representation data from the original music audio, and encode the visual representation data and acoustic representation data respectively to obtain visual feature vectors and acoustic feature vectors.

[0025] This step is responsible for data processing and dual-stream feature encoding. For example, the original music audio is first preprocessed using standardization, including uniform sampling rate, loudness normalization, and length alignment. Then, two complementary features are generated and encoded: one path converts the audio into a Mel spectrogram at a specific scale and inputs it into a pre-trained Swin Transformer network to extract a high-dimensional visual feature vector; the other path extracts a series of statistical acoustic features from the same original music audio, which are then processed through a pre-trained acoustic feature extraction network to obtain a low-dimensional acoustic feature vector, thus providing an informational basis for subsequent fusion.

[0026] S2, based on an asymmetric feature fusion mechanism with visual representation as the main component and acoustic semantics as the auxiliary component, the visual feature vector and the acoustic feature vector are fused into a joint feature vector.

[0027] This step achieves vision-dominated asymmetric feature fusion. For example, the low-dimensional acoustic feature vector is first normalized to match its distribution with that of the high-dimensional visual feature vector. Then, the two are directly concatenated along their feature dimensions. Since the visual feature dimension is higher, this operation naturally makes it dominant in the fusion. Finally, the concatenated features are fed into a multilayer perceptron (MLP) with a dropout layer for nonlinear transformation and dimensionality reduction, outputting a joint feature vector that fuses the deep texture of the vision and the semantic information of the acoustics.

[0028] S3, the joint feature vector is input into the multi-task learning network to simultaneously perform the main task of music emotion classification and the auxiliary task of VAD dimension regression, to obtain the emotion classification result and the predicted values ​​of three continuous dimensions of valence-arousal-control, and the multi-task learning network is optimized in reverse based on the emotion classification result and the predicted values ​​of the three continuous dimensions.

[0029] This step defines the model's workflow in both deployment and training / optimization modes. In the real-time detection (inference) phase, the joint feature vector generated in step S2 is input into a multi-task learning network. This network computes in parallel and outputs two results: a discrete music emotion classification result (e.g., probability distribution of 6 emotion categories) is output through the classification head; this result is the final functional output provided by the system. Simultaneously, prediction values ​​for three continuous dimensions—valence, arousal, and control—are calculated through the regression head. These are internal calculations during the inference phase and are typically not output externally. In the model training phase, labeled training data is used to simultaneously calculate the cross-entropy loss between the emotion classification result and the true category label, as well as the mean squared error regression loss between the VAD prediction value and the true VAD coordinate label. The two losses are weighted and summed to form the total loss, and the parameters of the entire network are optimized end-to-end using the backpropagation algorithm. This optimization process employs strategies such as hierarchical learning rate, cosine annealing scheduling, and label smoothing to improve the model's generalization ability on small sample data. This step enables auxiliary supervision and regularization using the VAD regression task during training, while only high-precision emotion classification results are output during actual application.

[0030] Understandably, given the shortcomings in the background technology, this invention proposes a visually-dominated multimodal learning emotion recognition method for small-sample music. This method first simultaneously extracts Mel spectrograms from the music audio as visual representation data and statistical acoustic features as acoustic representation data. These are then encoded into high-dimensional visual feature vectors and low-dimensional acoustic feature vectors through a pre-trained visual encoding network and acoustic feature extraction network, respectively. Subsequently, the two are fused into a joint feature vector through feature concatenation and other methods. This fusion process naturally assigns higher weights to visually richer features in the feature dimension, thus forming a visually-dominated asymmetric fusion. Finally, this joint feature vector is input into a multi-task learning network containing a classification head and a regression head, simultaneously outputting discrete emotion classification results and continuous valence-arousal-control (VAD) dimension prediction values. The entire model is optimized end-to-end by combining the loss functions of the two tasks.

[0031] This embodiment fully utilizes the temporal and frequency textures and music theory semantics of music through dual-stream feature extraction, strengthens the learning of discriminative patterns by leveraging visual-driven asymmetric fusion, and improves the model's accuracy in emotion recognition, ability to distinguish fuzzy emotion categories, and overall generalization performance by utilizing fine-grained supervision signals provided by the VAD regression task and multi-task learning strategies.

[0032] Based on the above technical solutions, the embodiments of the present invention can be further improved as follows.

[0033] In one possible embodiment, step S1 involves extracting visual representation data from the original music audio and encoding the visual representation data to obtain a visual feature vector, including: (1) Preprocessing the original music audio data and generating a Mel spectrogram of a specific size, specifically including: Preprocessing of the original music audio includes loudness normalization, resampling, and length alignment; The preprocessed audio is subjected to a short-time Fourier transform to obtain the spectrum; The spectrum is mapped to the Mel scale to generate a single-channel Mel spectrogram.

[0034] Understandably, Mel spectrograms, as a time-frequency analysis tool, can transform one-dimensional audio signals into two-dimensional images containing rich texture details, and their frequency scale conforms to the non-linear auditory perception characteristics of the human ear.

[0035] This embodiment first performs loudness normalization on all original music audio, standardizing it to -20dB to eliminate volume gain differences caused by different recording devices. Subsequently, all audio is resampled to 32kHz and uniformly length-aligned (padding or truncation) to ensure the input duration is consistent (e.g., 20 seconds). In the spectrum generation stage, short-time Fourier transform (STFT) is used to extract spectral information and map it to a Mel scale to obtain a single-channel Mel-spectrum.

[0036] (2) The Mel spectrogram is converted into a three-channel RGB image through pseudo-color mapping to serve as the visual representation data.

[0037] Understandably, in order to make the generated single-channel Mel spectrograms compatible with visual models pre-trained on the ImageNet image dataset, such as the Swin Transformer model, etc. Figure 2 As shown, the amplitude spectrum of the single-channel Mel spectrogram is converted into a three-channel RGB image through pseudo-color mapping, and the size of the three-channel RGB image is uniformly adjusted (e.g., through bicubic interpolation) to 384 pixels × 384 pixels, which is then used as the final visual representation data. This processing not only preserves the time-frequency energy distribution of the audio, but also transforms it into texture patterns that visual models are good at processing, such as high-frequency noise textures corresponding to harsh sounds, or regular harmonic stripes corresponding to harmonious music.

[0038] (3) Input the visual representation data (i.e., the resized RGB image) into a pre-trained visual encoding network based on the Swing Transformer architecture; The visual coding network sequentially passes through its constituent elements such as... Figure 2 The four stages shown perform hierarchical feature extraction and downsampling on the visual representation data; each stage contains several Swing TransformerBlocks and reduces the feature map resolution and increases the number of channels through the Patch Merging layer.

[0039] From the final output layer of the visual encoding network, such as the global average pooling layer output by the last stage, a visual feature vector of dimension 1×N1 is obtained, where N1 is a preset visual feature dimension, for example, N1=1024, thereby completing the encoding of visual features.

[0040] Understandably, this embodiment successfully converts the audio signal into a standard input format suitable for the pre-trained visual model by converting the original music audio into a Mel spectrogram and further processing it into a three-channel RGB image. This allows full utilization of the powerful feature extraction capabilities for texture, shape, and structure obtained by visual backbone networks such as the Swing Transformer through pre-training in the image domain. At the same time, through the hierarchical feature extraction and downsampling process of multiple layers of the visual coding network, it is possible to capture multi-level information from local fine textures to global time-frequency structures from the Mel spectrogram, ultimately generating a highly discriminative high-dimensional visual feature vector. This provides a high-quality feature foundation for the subsequent realization of high-precision emotion recognition dominated by visual information.

[0041] In one possible embodiment, step S1 involves extracting acoustic representation data from the original music audio and encoding the acoustic representation data to obtain an acoustic feature vector, including: (1) Extract the underlying acoustic feature sequence from the original music audio. The underlying acoustic feature sequence includes Mel frequency cepstral coefficients and their first and second order dynamic features used to describe timbre, chromaticity features and spectral contrast features used to describe tonality and harmonic structure, and rhythmic features and zero cross rate features used to describe rhythmic characteristics.

[0042] Understandably, while spectrograms preserve complete detail, they lack explicit musical semantics. Therefore, this embodiment extracts a set of 80-dimensional statistical acoustic feature sequences in parallel, aiming to capture the macroscopic properties of music from multiple dimensions. This statistical acoustic feature sequence set first includes the first 20 Mel-frequency cepstral coefficients (MFCCs) and their first and second-order differences, totaling 60 dimensions, used to finely describe timbre and its dynamic changes. Secondly, to characterize the tonality and harmonic structure of the music, a 12-dimensional chroma feature and a 6-dimensional spectral contrast feature are introduced; the latter reflects the clarity and fullness of the sound by measuring the peak-to-valley difference in the spectrum. Finally, the tempo and zero-crossing rate are calculated to capture global rhythmic patterns. All acoustic features were globally standardized before being input into the acoustic feature extraction network model, uniformizing them to zero mean and unit variance to eliminate the potential interference of differences in feature magnitudes on the stability of multimodal fusion.

[0043] (2) Input the underlying acoustic feature sequence into a pre-trained acoustic feature extraction network (e.g., Figure 2 The VGGish network shown is used to obtain the high-level acoustic embedding sequence; Temporally aggregate the high-level acoustic embedding sequence through temporal global average pooling (Mean Pooling) to obtain a global acoustic characterization vector; Reduce the dimension of the global acoustic characterization vector through a linear projection layer and perform layer normalization processing to obtain an acoustic feature vector with a dimension of 1×N2, where N2 is a preset acoustic feature dimension, and N2 < N1. For example, N1 = 1024 and N2 = 256.

[0044] It can be understood that in this embodiment, by extracting a low-level statistical acoustic feature sequence containing multi-dimensional information such as timbre, harmony, and rhythm from the original music audio and encoding it into high-level acoustic embeddings using a pre-trained VGGish network, auxiliary information with clear physical meaning and music theory semantics can be provided for the model; furthermore, through temporal aggregation and linear projection dimensionality reduction, the sequence information is condensed into an acoustic feature vector that matches the visual feature dimension but has a lower capacity, which not only ensures the effective integration of acoustic information but also naturally serves the vision-dominated setting in subsequent fusion by controlling its dimension, providing key and complementary semantic context for subsequent steps and enhancing the model's discriminative ability for complex music emotions. [[ID=�]]

[0045] In a possible embodiment, step S2 includes sub-steps S201 to S203.

[0046] S201, perform normalization processing on the acoustic feature vector so that the numerical distribution of the normalized acoustic feature vector adapts to the visual feature vector.

[0047] This step introduces layer normalization (Layer Normalization) at the end of the acoustic feature extraction and encoding branch to eliminate the statistical differences (Distribution Shift) between the feature distributions of different modalities. This step ensures that the acoustic features are consistent with the visually features output after layer normalization in terms of numerical magnitude, thus avoiding the risk that a certain modality dominates the gradient in subsequent optimization processes.

[0048] S202, perform a concatenation operation on the normalized acoustic feature vector and the visual feature vector to obtain a preliminary fusion feature vector, where the dimension of the visual feature vector is higher than that of the normalized acoustic feature vector.

[0049] This step constructs a joint representation using a feature concatenation (Concatenation) operation. The obtained preliminary fusion feature vector is defined by the following formula:

[0050] where is the visual feature vector, For acoustic eigenvectors, therefore, as Figure 2 As shown, the total dimension of the initially fused feature vectors This 4:1 asymmetric dimensional ratio implicitly assigns higher weight to visual features, which aligns with prior assumptions.

[0051] S203, the preliminary fused feature vector The input is fed into a Multilayer Perceptron (MLP) module, which consists of stacked linear layers, a nonlinear activation function (GELU), and a normalization layer to perform nonlinear transformations and feature interactions, outputting the joint feature vector. To mitigate the increased parameters and overfitting risks associated with feature fusion, a dropout layer is integrated into the MLP module. This deep nonlinear mapping effectively deconstructs semantic features strongly correlated with emotion categories from the high-dimensional joint space. Through this asymmetric fusion mechanism, the model not only retains a keen perception of spectrogram texture but also successfully utilizes acoustic context as auxiliary cues, thus achieving robust discrimination of musical emotions.

[0052] Understandably, effectively fusing features from heterogeneous data sources is a key challenge in multimodal learning. Traditional symmetric fusion strategies often project features from different modalities onto the same dimension for combination. However, in this embodiment, as... Figure 2 As shown, the visual representation (1024 dimensions) extracted by the pre-trained Swin Transformer contains extremely rich and high-dimensional texture and structural information, which constitutes the main basis for emotion discrimination. In contrast, although the acoustic representation (256 dimensions) has clear physical semantics, its information density is relatively low, mainly playing an auxiliary correction role. Therefore, the asymmetric feature fusion strategy proposed in this embodiment can effectively retain the dominance of the visual flow while effectively integrating the complementary information of the acoustic flow.

[0053] In one possible embodiment, in step S3, reference is made to... Figure 2 As shown, the multi-task learning network includes a classification head and a regression head that are set up in parallel and share the joint feature vector as input; wherein, the classification head is used to output the emotion classification result, and the regression head is used to output the predicted values ​​of the three continuous dimensions of valence-arousal-control.

[0054] The step of back-optimizing the multi-task learning network based on the emotion classification results and the predicted values ​​of the three continuous dimensions specifically includes: During training, the classification loss between the emotion classification result and the true emotion category label is calculated, and the regression loss between the predicted values ​​of the three continuous dimensions and the true VAD dimension coordinate labels is calculated. The network parameters of the multi-task learning network are optimized using the weighted sum of the classification loss and the regression loss as the total loss through the backpropagation algorithm.

[0055] The classification loss is calculated using the cross-entropy loss function, and the regression loss is calculated using the mean squared error loss function. The true VAD dimension coordinate labels are obtained from the pre-established mapping relationship between discrete music emotion category labels and continuous VAD spatial coordinates.

[0056] Understandably, during the deployment phase (inference phase), the model parameters are fixed and loaded. In this state, a multi-task learning network is used for forward computation. Specifically, the joint feature vector is simultaneously and in parallel input into two independent learnable modules (i.e., the classification head and the regression head): the classification head is one or more fully connected layers responsible for outputting the final emotion classification result, such as a 6-dimensional vector representing the probability distribution of six basic emotion categories; the other is the regression head, which is also one or more fully connected layers responsible for outputting predicted values ​​for the three continuous dimensions of valence, arousal, and control—that is, a 3-dimensional real-number vector. Although the classification head and regression head share the same input features, they work independently, producing classification and regression outputs respectively. In practical applications, only the output of the classification head is used as the result of voice emotion recognition presented to the user.

[0057] During the training phase, the multi-task learning network is the foundation for the model's multi-task learning. In the forward propagation of each training iteration, the joint feature vector is also input into the classification head and regression head in parallel. The key difference from the inference phase is that the outputs of the classification head and regression head are used simultaneously to calculate the loss: the output of the classification head is used with the true sentiment category label to calculate the classification loss (e.g., cross-entropy loss), and the output of the regression head is used with the true VAD dimension coordinate label to calculate the regression loss (e.g., mean squared error loss). The classification loss and regression loss are weighted and summed according to predefined weights to obtain a total loss function, expressed as:

[0058] in, The cross-entropy loss function is used to represent the loss of the model in the classification task; Mean squared error loss (MSE) is used to constrain VAD predictions. Its corresponding dimension label Consistency between them. Hyperparameters This is used to balance the gradient contributions of the two tasks. Through this multi-task constraint, the model is forced not only to distinguish "which category this is", but also to understand "where this emotion ranks in terms of arousal and pleasure", thereby significantly enhancing the semantic discriminative power of the features.

[0059] During the subsequent backpropagation process, this total loss function simultaneously and jointly optimizes the parameters of three parts: the parameters of the classification head itself, the parameters of the regression head itself, and the parameters of the upstream feature extraction network and fusion network that they share. Through this parallel structure with shared input but independent tasks, the model is able to learn a general feature representation that can effectively classify and perceive the continuous dimension of emotion during the optimization process.

[0060] In one possible embodiment, during step S3, the process of reverse optimizing the multi-task learning network employs at least one optimization strategy among hierarchical learning rate optimization, cosine annealing learning rate scheduling with a warm-up phase, label smoothing, and weight decay.

[0061] For example, in a specific implementation scenario, the following targeted optimization strategies were developed to address the challenge of unstable model training on small sample data.

[0062] (1) Training Strategy The model was trained using the AdamW optimizer. Considering the sensitivity of self-attention-based deep learning model architectures (such as the Transformer) to gradient noise, a fixed learning rate was not used; instead, a cosine annealing scheduler with a 20% warmup was employed. This strategy allows the model to start smoothly in the early stages of training and allows for fine-tuning of weights later. Training lasted for 50 epochs, with an early stopping mechanism of 20 epochs to conserve computational resources and prevent overfitting.

[0063] (2) Layer-wise Optimization & Gradient Stability Since the Swin Transformer's backbone network has been fully pre-trained on ImageNet (an image dataset), while the classification head of the multi-task learning network is randomly initialized, their learning rate requirements differ significantly. Therefore, a layer-wise learning rate decay (LLRD) strategy is implemented. Specifically, the learning rate of the classification head is set to a higher value. To accelerate convergence, the base learning rate of the backbone network is set to a low value. A layer decay rate of 0.9 is employed. This differentiated configuration effectively avoids catastrophic forgetting and preserves the pre-trained general texture features to the greatest extent. Furthermore, to stabilize the statistical properties of gradients under limited GPU memory, a gradient accumulation strategy is adopted to maintain the effective batch size at 32.

[0064] (3) Regularization Overfitting is a major bottleneck in few-sample fine-tuning tasks. Preliminary experiments show that directly transferring pixel-level enhancements from natural image domains (such as mixups) disrupts the time-frequency structure of the spectrogram, leading to performance degradation. Therefore, this implementation shifts the focus of regularization to the model structure itself. A stochastic depth / drop path technique is employed, with a dropout rate of 0.2 and a weight decay of 0.02. Simultaneously, to address the subjectivity and ambiguity of sentiment labels, label smoothing with a coefficient of 0.1 is introduced into the cross-entropy loss function. This combined strategy significantly improves generalization ability while suppressing model overconfidence.

[0065] The present invention provides a visual-dominated multimodal learning emotion recognition method for small-sample music. It adopts a multimodal fusion framework with visual representation as the main driver and acoustic semantics as the auxiliary driver, and introduces VAD (valence-arousal-control) dimension regression as an auxiliary training task. This not only solves the overfitting problem in small-sample scenarios, but also effectively improves the accuracy of emotion recognition.

[0066] Figure 3 This is a structural diagram of a vision-driven multimodal learning emotion recognition system for few-sample music, provided by an embodiment of the present invention. Figure 3 As shown, a vision-driven multimodal learning emotion recognition system for few-sample music includes a feature extraction and encoding module, an asymmetric feature fusion module, and a multi-task learning and optimization module, wherein: The feature extraction and encoding module is used to extract visual representation data and acoustic representation data from the original music audio, and to encode the visual representation data and acoustic representation data respectively to obtain visual feature vectors and acoustic feature vectors. An asymmetric feature fusion module is used to fuse the visual feature vector and the acoustic feature vector into a joint feature vector based on an asymmetric feature fusion mechanism that is dominated by visual representation and assisted by acoustic semantics. The multi-task learning and optimization module is used to input the joint feature vector into the multi-task learning network to simultaneously perform the main task of music emotion classification and the auxiliary task of VAD dimension regression, to obtain the emotion classification result and the predicted values ​​of three continuous dimensions of valence-arousal-control, and to back-optimize the multi-task learning network based on the emotion classification result and the predicted values ​​of the three continuous dimensions.

[0067] It is understood that the visual-dominated multimodal learning emotion recognition system for few-sample music provided by this invention corresponds to the visual-dominated multimodal learning emotion recognition method for few-sample music provided in the foregoing embodiments. The relevant technical features of the visual-dominated multimodal learning emotion recognition system for few-sample music can be referred to the relevant technical features of the visual-dominated multimodal learning emotion recognition method for few-sample music, and will not be repeated here.

[0068] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating an embodiment of the electronic device provided in this invention. For example... Figure 4 As shown, this embodiment of the invention provides an electronic device 400, including a memory 410, a processor 420, and a computer program 411 stored in the memory 410 and executable on the processor 420. When the processor 420 executes the computer program 411, it performs the following steps: S1, extract visual representation data and acoustic representation data from the original music audio, and encode the visual representation data and acoustic representation data respectively to obtain visual feature vectors and acoustic feature vectors; S2, based on an asymmetric feature fusion mechanism with visual representation as the main component and acoustic semantics as the auxiliary component, the visual feature vector and the acoustic feature vector are fused into a joint feature vector; S3, the joint feature vector is input into the multi-task learning network to simultaneously perform the main task of music emotion classification and the auxiliary task of VAD dimension regression, to obtain the emotion classification result and the predicted values ​​of three continuous dimensions of valence-arousal-control, and the multi-task learning network is optimized in reverse based on the emotion classification result and the predicted values ​​of the three continuous dimensions.

[0069] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by the present invention. (See diagram below.) Figure 5 As shown, this embodiment provides a computer-readable storage medium 500 on which a computer program 411 is stored. When the computer program 411 is executed by a processor, it performs the following steps: S1, extract visual representation data and acoustic representation data from the original music audio, and encode the visual representation data and acoustic representation data respectively to obtain visual feature vectors and acoustic feature vectors; S2, based on an asymmetric feature fusion mechanism with visual representation as the main component and acoustic semantics as the auxiliary component, the visual feature vector and the acoustic feature vector are fused into a joint feature vector; S3, the joint feature vector is input into the multi-task learning network to simultaneously perform the main task of music emotion classification and the auxiliary task of VAD dimension regression, to obtain the emotion classification result and the predicted values ​​of three continuous dimensions of valence-arousal-control, and the multi-task learning network is optimized in reverse based on the emotion classification result and the predicted values ​​of the three continuous dimensions.

[0070] This invention provides a visually-driven multimodal learning method, system, electronic device, and storage medium for emotion recognition based on small sample music. It constructs a complementary dual-stream representation by simultaneously extracting and encoding Mel spectrogram visual features and statistical acoustic features from music audio. Furthermore, it enhances the multi-task learning network's ability to capture discriminative patterns of music emotion through an asymmetric fusion method that prioritizes high-dimensional visual features and assists with low-dimensional acoustic features. Further, by introducing VAD dimensionality regression as an auxiliary task alongside the main classification task for multi-task learning, and employing optimization strategies such as hierarchical learning rate and label smoothing, it effectively improves the model's generalization ability on small sample data and its accuracy in distinguishing fuzzy emotion categories, ultimately achieving efficient end-to-end training and more accurate music emotion recognition.

[0071] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0072] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0077] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A visually-driven multimodal learning-based emotion recognition method for small-sample music, characterized in that, include: Visual and acoustic representation data are extracted from the original music audio, and the visual and acoustic representation data are encoded to obtain visual feature vectors and acoustic feature vectors, respectively. Based on an asymmetric feature fusion mechanism that prioritizes visual representation and uses acoustic semantics as an auxiliary, the visual feature vector and the acoustic feature vector are fused into a joint feature vector. The joint feature vector is input into a multi-task learning network to simultaneously perform the main task of music emotion classification and the auxiliary task of VAD dimension regression, so as to obtain the emotion classification result and the predicted values ​​of three continuous dimensions: valence-arousal-control. The multi-task learning network is then optimized in reverse based on the emotion classification result and the predicted values ​​of the three continuous dimensions.

2. The visual-dominated multimodal learning emotion recognition method for few-sample music as described in claim 1, characterized in that, Extracting visual representation data from the original music audio and encoding the visual representation data to obtain a visual feature vector includes: The raw music audio data is preprocessed to generate a Mel spectrogram of a specific size; The Mel spectrogram is converted into a three-channel RGB image using pseudo-color mapping to serve as the visual representation data; The visual representation data is input into a visual encoding network based on the Swing Transformer architecture; The visual coding network sequentially performs hierarchical feature extraction and downsampling on the visual representation data through its multiple layers; Obtain a visual feature vector of dimension 1×N1 from the final output layer of the visual coding network, where N1 is a preset visual feature dimension.

3. The visual dominant multi-modal learning emotion recognition method for small sample music of claim 2, wherein, The process of preprocessing the original music audio data and generating a Mel spectrogram of a specific size includes: Preprocessing of the original music audio includes loudness normalization, resampling, and length alignment; The preprocessed audio is subjected to a short-time Fourier transform to obtain the spectrum; The spectrum is mapped to the Mel scale to generate a single-channel Mel spectrogram.

4. The visual dominant multi-modal learning emotion recognition method for small sample music of claim 2, wherein, The process involves extracting acoustic representation data from the original music audio and encoding the acoustic representation data to obtain acoustic feature vectors, including: Extracting the underlying acoustic feature sequence from the original music audio; The low-level acoustic feature sequence is input into a pre-trained acoustic feature extraction network to obtain a high-level acoustic embedding sequence. The advanced acoustic embedding sequence is temporally aggregated to obtain a global acoustic representation vector. The global acoustic representation vector is linearly projected and normalized to obtain an acoustic feature vector of dimension 1×N2, where N2 is a preset acoustic feature dimension, and N2 <N1。 5. The visual dominant multi-modal learning emotion recognition method for small sample music according to any one of claims 1-4, characterized in that, The asymmetric feature fusion mechanism, based on visual representation as the primary method and acoustic semantics as the secondary method, fuses the visual feature vector and the acoustic feature vector into a joint feature vector, including: The acoustic feature vector is normalized so that the numerical distribution of the normalized acoustic feature vector matches the visual feature vector. The normalized acoustic feature vector is concatenated with the visual feature vector to obtain a preliminary fused feature vector, wherein the dimension of the visual feature vector is higher than the dimension of the normalized acoustic feature vector. The preliminary fused feature vector is input into a multilayer perceptron module for nonlinear transformation and feature interaction, and the joint feature vector is output. The multilayer perceptron module integrates a random deactivation layer.

6. The visual dominant multi-modal learning emotion recognition method for small sample music of claim 5, wherein, The multi-task learning network includes a classification head and a regression head that are configured in parallel and share the joint feature vector as input; wherein, the classification head is used to output the emotion classification result, and the regression head is used to output the predicted values ​​of the three continuous dimensions of valence, arousal and control.

7. The visual dominant multi-modal learning emotion recognition method for small sample music of claim 5, wherein, The step of back-optimizing the multi-task learning network based on the emotion classification results and the predicted values ​​of the three continuous dimensions specifically includes: During training, the classification loss between the emotion classification result and the true emotion category label is calculated, and the regression loss between the predicted values ​​of the three continuous dimensions and the true VAD dimension coordinate labels is calculated. The network parameters of the multi-task learning network are optimized using the weighted sum of the classification loss and the regression loss as the total loss through the backpropagation algorithm.

8. The visual dominant multi-modal learning emotion recognition method for small sample music of claim 7, wherein, The classification loss is calculated using the cross-entropy loss function, and the regression loss is calculated using the mean squared error loss function; the true VAD dimension coordinate labels are obtained from the pre-established mapping relationship between discrete music emotion category labels and continuous VAD spatial coordinates.

9. The visual dominant multi-modal learning emotion recognition method for small sample music of claim 7, wherein, In the process of reverse optimization of the multi-task learning network, at least one optimization strategy is adopted, including hierarchical learning rate optimization, cosine annealing learning rate scheduling with a warm-up phase, label smoothing, and weight decay.

10. A visual dominant multi-modal learning emotion recognition system for small sample music, characterized in that, include: The feature extraction and encoding module is used to extract visual representation data and acoustic representation data from the original music audio, and to encode the visual representation data and acoustic representation data respectively to obtain visual feature vectors and acoustic feature vectors. An asymmetric feature fusion module is used to fuse the visual feature vector and the acoustic feature vector into a joint feature vector based on an asymmetric feature fusion mechanism that is dominated by visual representation and assisted by acoustic semantics. The multi-task learning and optimization module is used to input the joint feature vector into the multi-task learning network to simultaneously perform the main task of music emotion classification and the auxiliary task of VAD dimension regression, to obtain the emotion classification result and the predicted values ​​of three continuous dimensions of valence-arousal-control, and to back-optimize the multi-task learning network based on the emotion classification result and the predicted values ​​of the three continuous dimensions.

Images