A speech emotion recognition method across granular weight reuse and heterogeneous multitasking

Abstract

The application discloses a speech emotion recognition method for cross-granularity weight reuse and heterogeneous multi-tasking, and relates to the technical field of speech signal processing and artificial intelligence, and the method comprises the following steps: a nested mutual exclusion data division system is constructed, and a training set is divided into a teacher training set and a target reasoning set; based on the additive linear semantic consistency assumption of emotional features, the sentence-level classification weights of the teacher model are reused to perform frame-by-frame reasoning on the target reasoning set, and frame-level hard pseudo labels are generated; a multi-task student network with heterogeneous feature extractors is constructed, which comprises a parallel sentence-level main task branch and a frame-level auxiliary task branch; and the student network is cooperatively trained by using the sentence-level real labels and the frame-level pseudo labels through a joint loss function. Without artificial frame-level labeling, the application realizes the construction and utilization of fine-grained emotional supervision signals, and effectively improves the modeling capability of the model for local emotional changes of speech and the overall recognition performance.

Description

Technical Field

[0001] This invention relates to the fields of speech signal processing and artificial intelligence, and in particular to a speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task. Background Technology

[0002] Speech Emotion Recognition (SER) is a core technology in fields such as human-computer interaction, intelligent customer service, in-vehicle voice systems, and mental health assistance. It aims to automatically identify the speaker's emotional state by analyzing speech signals. With the rapid development of deep learning technology, existing SER methods generally employ deep neural networks to model the spectral features, temporal structure, and vocalization patterns of speech, using sentence-level emotion annotation as a supervisory signal to complete the emotion classification task. Current mainstream techniques are typically based on pre-trained acoustic models (such as Wav2Vec2 and HuBERT), combined with attention mechanisms or end-to-end Transformer architectures, achieving good benchmark performance on multiple public SER datasets.

[0003] Although existing methods have improved the accuracy of emotion recognition to some extent, the following significant limitations still exist:

[0004] 1. Limited Granularity of Supervision Signals: Most existing methods rely on coarse-grained sentiment labels at the sentence level for training, aggregating variable-length speech features into fixed-dimensional vectors through methods such as global pooling. This approach ignores the local emotional changes in the speech signal over time (e.g., brief bursts of anger interspersed within an otherwise calm sentence), making it difficult for the model to capture fine-grained emotional fluctuations and hierarchical emotional structures. Consequently, its recognition ability is limited when faced with complex speech phenomena such as emotional reversals and abrupt changes in intonation.

[0005] 2. Insufficient utilization of pre-trained models: Current transfer learning in SER mainly adopts the "pre-training + sentence-level fine-tuning" model, that is, adding a sentence-level classifier for fine-tuning on the basis of the pre-trained acoustic model. This approach fails to fully explore the hierarchical representation capabilities of the pre-trained model at different time scales, especially lacking the effective reuse of potential weights at the frame or segment level, which limits the model's modeling and transfer effects on deep emotional cues.

[0006] 3. Homogeneity of Multi-Task Learning Frameworks: Existing multi-task learning methods in SER mostly focus on homogeneous or homogeneous tasks (such as sentiment classification and speaker recognition, or sentence-level sentiment and sentence-level semantic tasks), lacking heterogeneous multi-task mechanisms that can effectively integrate supervision signals of different granularities (such as sentence-level and frame-level). Due to the lack of hierarchical alignment and semantic interaction between tasks, the synergistic optimization effect of multi-task learning is not fully realized, making it difficult to enhance the discriminative power of sentence-level sentiment discrimination through fine-grained supervision.

[0007] 4. Lack of fine-grained labeled data: Due to the high cost and strong subjectivity of manually labeling frame-level or fragment-level sentiment data, existing SER datasets generally lack fine-grained sentiment supervision. Although some studies have attempted to alleviate the scarcity of labels through methods such as pseudo-label generation, traditional methods often suffer from problems such as label noise, overfitting, or data leakage, making it difficult to build a high-quality, generalizable fine-grained supervision system, which in turn limits the model's ability to model local sentiment features in a refined manner.

[0008] In summary, existing speech emotion recognition technologies have significant shortcomings in terms of emotion supervision granularity, pre-training weight reuse strategies, multi-task collaborative mechanisms, and fine-grained label construction, which limit the performance improvement and generalization ability of models in complex real-world scenarios. Therefore, there is an urgent need for a new speech emotion recognition method that can effectively utilize multi-granularity supervision, achieve cross-level feature fusion, and perform robust learning even in the absence of fine-grained annotations. Summary of the Invention

[0009] To address this, this invention provides a speech emotion recognition method that combines cross-granularity weight reuse and heterogeneous multi-task processing. This method solves the problems in existing technologies where the lack of fine-grained emotion supervision signals and insufficient multi-task collaboration mechanisms result in weak modeling capabilities for local emotional changes in speech and limited overall recognition performance.

[0010] To address the aforementioned technical problems, embodiments of the present invention provide a speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task implementation, the method comprising:

[0011] S1. Construct a nested and mutually exclusive data partitioning system: Divide the original speech training dataset with sentence-level sentiment labels into multiple outer folds; for the training set of each outer fold, further divide it into multiple mutually exclusive inner subsets; in each round of inner processing, select one inner subset as the target inference set, and the union of the remaining inner subsets as the teacher training set.

[0012] S2. Generate frame-level pseudo-labels based on cross-granularity weight reuse: In each round of inner layer processing, a teacher model is trained using the teacher training set. The teacher model obtains classification weights by minimizing sentence-level sentiment classification loss. For speech samples in the target inference set, the pooling layer of the teacher model is removed, while keeping the classification weights unchanged. The extracted frame-level feature sequences are classified frame by frame to generate corresponding frame-level hard pseudo-label sequences.

[0013] S3. Construct and train a heterogeneous multi-task student network: Construct a student model that includes a pre-trained feature extractor heterogeneous with the teacher model, as well as parallel sentence-level main task branches and frame-level auxiliary task branches; Use the sentence-level real labels and the frame-level hard pseudo-label sequences from the original speech training dataset to train the student model end-to-end through a joint loss function, which is composed of a weighted sum of sentence-level main task loss and frame-level auxiliary task loss;

[0014] S4. Model Inference: Use the trained student model to perform speech emotion recognition, using only its sentence-level main task branch to perform sentence-level emotion classification on the input speech.

[0015] Preferably, in step S1, the nested mutually exclusive data partitioning system specifically includes:

[0016] Let the original speech dataset be... ,in For the first The original audio of each speech sample. For the first The actual sentence-level sentiment labels of each voice sample The total number of samples in the original speech dataset. For the speech sample index, it is divided into One outer fold;

[0017] For the Outer training set of the fold Further divide it into Let be mutually exclusive inner subsets. ;

[0018] For the Secondary inner-level rotation, inner-level teacher training program Inner target reasoning set And satisfy ,in Indicates the first an inner subset, This represents the set subtraction operation. This represents the empty set.

[0019] Preferably, in step S2, the generation of frame-level pseudo-tags based on cross-granularity weight reuse specifically includes:

[0020] Based on the assumption that sentiment features have additive linear semantic consistency in vector space—that is, sentence-level sentiment feature vectors are linearly superimposed from frame-level sentiment feature vectors—the classification weight hyperplane used for classifying sentence-level features is reused for classifying frame-level features. For speech samples in the target inference set, its... Frame-level hard pseudo-labels The formula for generating it is:

[0021] ;

[0022] ;

[0023] ;

[0024] in, This is an operation to retrieve the index of the maximum value. For sentiment category indexing, Indicates the first Frame belongs to the Unnormalized score of sentiment-like characteristics For time frame indexing, Indicates the first Unnormalized score of the frame, It is a non-linear activation function. and For the weight matrix and bias vector, For projection layer parameters, For the first The input feature vector of the frame, Indicates that it is subject to, Represents the sentence-level loss function. This represents the actual sentence-level sentiment label corresponding to the speech sample. This represents a function that transforms a vector of logical values ​​into a probability distribution. The weights of the classifier to be optimized will be equal to the weights after optimization. , Indicates the total number of frames. For the first The input feature vector of the frame.

[0025] Preferably, in step S3, the feature extractor of the heterogeneous multi-task student network The feature extractor used in the teacher model Different architectures of pre-trained models, and For general acoustic models, For the emotion-focused model, that is .

[0026] Preferably, in step S3, the sentence-level main task branch includes a temporal aggregation module and a classifier, through... To achieve temporal aggregation, where This is a sentence-level global feature vector. This indicates a time-domain aggregation operation. Heterogeneous frame-level feature sequences extracted for the student model For the first Heterogeneous feature vectors of frames Total number of frames The time frame index is used; the frame-level auxiliary task branch is a point-to-point classifier that does not contain any temporal aggregation operations and directly performs sentiment classification on single-frame features.

[0027] Preferably, in step S3, the joint loss function is defined as:

[0028] ;

[0029] in, For the joint loss function, For sentence-level main task loss, For frame-level auxiliary task loss, Hyperparameters to balance the contributions of the two tasks.

[0030] Preferably, the sentence-level main task loss and frame-level auxiliary task loss The specific calculation method is as follows:

[0031] ;

[0032] ;

[0033] in, The total number of samples in the original speech dataset. For speech sample indexing, The total number of emotion categories to be identified. For sentiment category indexing, For indicator functions, For the first The actual sentence-level sentiment labels of each voice sample For the first The sample is predicted as the first in the sentence-level main task branch. The probability value of emotion-like qualities. For the first Total number of frames corresponding to each speech sample For time frame indexing, For the first The first voice sample Frame-level hard pseudo-labels For the first The first voice sample In the frame-level auxiliary task branch, the predicted frame is the [number]. The probability value of a type of emotion.

[0034] Preferably, after the training in step S3 is completed, in the model inference stage of step S4, the frame-level auxiliary task branch and its related parameters are discarded, and only the feature extractor and the sentence-level main task branch are retained for speech emotion recognition.

[0035] This invention also provides a speech emotion recognition system with cross-granularity weight reuse and heterogeneous multi-task implementation. This system is used to implement the aforementioned speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task implementation, specifically including:

[0036] The data partitioning module is used to construct a nested and mutually exclusive data partitioning system: the original speech training dataset with sentence-level sentiment labels is divided into multiple outer folds; for the training set of each outer fold, it is further divided into multiple mutually exclusive inner subsets; in each round of inner processing, one inner subset is selected as the target inference set, and the union of the remaining inner subsets is used as the teacher training set.

[0037] The pseudo-label generation module is used to generate frame-level pseudo-labels based on cross-granularity weight reuse: In each round of inner layer processing, a teacher model is trained using the teacher training set. The teacher model obtains classification weights by minimizing sentence-level sentiment classification loss. For speech samples in the target inference set, the pooling layer of the teacher model is removed, the classification weights are kept unchanged, and the extracted frame-level feature sequences are classified frame by frame to generate the corresponding frame-level hard pseudo-label sequences.

[0038] The model training module is used to construct and train a heterogeneous multi-task student network: a student model is constructed, which includes a pre-trained feature extractor heterogeneous with the teacher model, as well as parallel sentence-level main task branches and frame-level auxiliary task branches; using the sentence-level ground truth labels of the original speech training dataset and the frame-level hard pseudo-label sequences, the student model is trained end-to-end through a joint loss function, which is composed of a weighted sum of sentence-level main task loss and frame-level auxiliary task loss;

[0039] The inference module is used for model inference: it performs speech emotion recognition using the trained student model, and performs sentence-level emotion classification on the input speech using only its sentence-level main task branch.

[0040] This invention also provides an electronic device, which includes a processor, a memory, and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to realize the above-described method for cross-granularity weight reuse and heterogeneous multi-task speech emotion recognition.

[0041] As can be seen from the above technical solutions, this invention application has the following beneficial effects:

[0042] (1) This invention innovatively proposes the assumption that the emotion vector space is approximately decomposable and a cross-granularity weight reuse mechanism. By reusing sentence-level classification weights, frame-level pseudo-labels are directly generated through reasoning. Combined with a nested mutually exclusive data partitioning strategy, the self-confirmation bias of pseudo-labels is effectively avoided. This scheme automatically constructs a large-scale, highly reliable fine-grained emotion labeling system without any additional manual annotation, fundamentally solving the core problems of high cost and data scarcity of fine-grained annotation in the field of speech emotion recognition.

[0043] (2) By constructing a student network that includes a heterogeneous feature extractor and parallel multi-task branches, this invention can simultaneously leverage the complementary advantages of a general acoustic model and an emotion-specialized model. Using generated frame-level pseudo-labels as dense regularization terms, the joint loss function forces the model to focus on and fit local fine-grained emotion changes while optimizing global sentence-level classification. This heterogeneous multi-task collaborative mechanism effectively guides the model to learn more discriminative hierarchical features, significantly improving its ability to model complex speech phenomena such as emotional fluctuations and abrupt changes in intonation.

[0044] (3) The method of this invention applies fine-grained constraints and enhancements to the model through frame-level auxiliary tasks during the training phase, but retains only the lightweight sentence-level main task branch during the final inference phase. Therefore, when the trained model is deployed, its computational complexity and inference time are no different from those of conventional sentence-level classification models, and no additional overhead is introduced. However, because the model internalizes fine-grained emotional cues during training, its recognition accuracy and overall robustness for complex samples with long speech, sparse emotions, or local emotional reversals are substantially improved. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Referring to the drawings will make the features and advantages of the present invention clearer. The drawings are illustrative and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0046] Figure 1 This is a flowchart of a speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task provided by the present invention;

[0047] Figure 2 This is a schematic diagram of the overall architecture of the method of the present invention;

[0048] Figure 3 This is a block diagram of a speech emotion recognition system with cross-granularity weight reuse and heterogeneous multi-task provided by the present invention.

[0049] Explanation of the reference numerals in the instruction manual's attached diagrams: 100, Data partitioning module; 200, Pseudo-label generation module; 300, Model training module; 400, Inference module. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Example 1: To address the problem in existing technologies where the lack of fine-grained emotion supervision signals and insufficient multi-task collaboration mechanisms result in weak modeling ability for local emotion changes in speech and limited overall recognition performance. For example... Figure 1 and Figure 2 As shown, this invention proposes a speech emotion recognition method based on cross-granularity weight reuse and heterogeneous multi-task architecture. This method achieves fine-grained enhanced speech emotion recognition without the need for manual frame-level annotation by constructing a nested mutually exclusive data partitioning system, generating high-quality frame-level pseudo-labels based on cross-granularity weight reuse, and building and jointly optimizing a heterogeneous multi-task student network.

[0052] S1. Construct a nested and mutually exclusive data partitioning system: Divide the original speech training dataset with sentence-level sentiment labels into multiple outer folds; for the training set of each outer fold, further divide it into multiple mutually exclusive inner subsets; in each round of inner processing, select one inner subset as the target inference set, and the union of the remaining inner subsets as the teacher training set.

[0053] S2. Generate frame-level pseudo-labels based on cross-granularity weight reuse: In each round of inner layer processing, a teacher model is trained using the teacher training set. The teacher model obtains classification weights by minimizing sentence-level sentiment classification loss. For speech samples in the target inference set, the pooling layer of the teacher model is removed, the classification weights are kept unchanged, and the extracted frame-level feature sequences are classified frame by frame to generate the corresponding frame-level hard pseudo-label sequences.

[0054] S3. Construct and train a heterogeneous multi-task student network: Construct a student model that includes a pre-trained feature extractor that is heterogeneous with the teacher model, as well as parallel sentence-level main task branches and frame-level auxiliary task branches; Use sentence-level real labels and frame-level hard pseudo-label sequences from the original speech training dataset to train the student model end-to-end through a joint loss function, which consists of a weighted sum of sentence-level main task loss and frame-level auxiliary task loss;

[0055] S4. Model Inference: Use the trained student model for speech emotion recognition, and only use its sentence-level main task branch to perform sentence-level emotion classification on the input speech.

[0056] As can be seen from the above technical solution, this invention proposes a speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task approach. Firstly, by constructing a nested and mutually exclusive data partitioning system, the propagation path of label noise is cut off at the data level, laying the foundation for generating unbiased, high-quality pseudo-labels. Secondly, based on the linear consistency assumption of emotion features, the sentence-level classification weights of the teacher model are reused to perform frame-by-frame inference on frame-level features, thereby automatically generating reliable fine-grained supervision signals without manual annotation, effectively solving the core problem of scarce annotation resources. On this basis, a heterogeneous multi-task student network is designed and trained: by employing a feature extractor with a different architecture than the teacher model, the complementary advantages between models are utilized to enhance representation capabilities; and by jointly optimizing the sentence-level main task and the frame-level auxiliary task, the model is forced to learn both global emotion trends and local fine-grained changes simultaneously, achieving deep fusion and regularization of cross-granularity features, significantly improving the model's discriminative ability. Finally, only lightweight sentence-level branches are retained in the inference stage, allowing the model to achieve the aforementioned performance gains without increasing the computational overhead during actual deployment, achieving synergistic optimization of recognition accuracy and operational efficiency.

[0057] In step S1, a nested mutually exclusive data partitioning system is constructed. To address the "self-confirmation bias" caused by directly generating pseudo-labels on the training set and using them for self-training, this invention first constructs a nested mutually exclusive data partitioning system. By physically isolating training samples and inference samples, the objectivity and unbiasedness of pseudo-label generation are ensured.

[0058] Specifically, let the original speech dataset be... ,in For the first The original audio of each speech sample. For the first The actual sentence-level sentiment labels of each voice sample The total number of samples in the original speech dataset. Indexing for speech samples. First, [the following is done]. Divided into An outer layer fold (such as) or ), employing a cross-validation strategy. For the ), Outer training set of the fold Further divide it into Let be mutually exclusive inner subsets. .

[0059] In each round of inner layer processing, select one inner layer subset. The union of the remaining subsets serves as the Teacher Set, which is the target reasoning set.

[0060] ;

[0061] ;

[0062] And it satisfies:

[0063] ;

[0064] in, This refers to the inner-level teacher training set. This represents the inner target inference set. This represents set subtraction. This represents the empty set. This partitioning method cuts off the memory transmission path of label noise in the model parameters at the data level, laying the foundation for subsequent high-quality pseudo-label generation.

[0065] In step S2, frame-level pseudo-labels are generated based on cross-granularity weight reuse. This step is based on the assumption that sentiment features have "additive linear semantic consistency" in the vector space, meaning that sentence-level sentiment feature vectors can be linearly superimposed from frame-level sentiment feature vectors. Therefore, the trained sentence-level classification weights can be directly reused for frame-level classification. Based on this, an "asymmetric train-inference" paradigm is used to generate dense frame-level hard pseudo-labels.

[0066] The specific process is as follows:

[0067] 1. Teacher Model Training: In the inner layer teacher training set Train an emotion-focused teacher model by minimizing the sentence-level global loss function. Obtaining classification weights and bias The loss function is typically the cross-entropy loss.

[0068] 2. Frame-level pseudo-tag generation: For the target inference set In the speech samples, remove the pooling layers (such as global average pooling) of the teacher model, and maintain Without changing the order, perform frame-by-frame classification on the extracted frame-level feature sequences. Let the first frame be... The input feature vector of the frame is After the projection layer Then, through classification weights Calculate the unnormalized score :

[0069] ;

[0070] ;

[0071] in, It is a non-linear activation function (such as ReLU or GELU). For projection layer parameters, Indicates that it is subject to, Represents the sentence-level loss function. This represents the actual sentence-level sentiment label corresponding to the speech sample. This represents a function that transforms a vector of logical values ​​into a probability distribution. The weights of the classifier to be optimized will be equal to the weights after optimization. , Indicates the total number of frames. For the first The input feature vector of the frame.

[0072] 3. Hard tag generation: For Perform the Argmax operation along the sentiment category dimension to obtain the hard pseudo-label for the frame:

[0073] ;

[0074] in, For the first Frame-level hard pseudo-labels For sentiment category indexing, Indicates the first Frame belongs to the Unnormalized emotion-based scores. This step yields a time-aligned, frame-level pseudo-label sequence for each speech sample. , This represents the total number of frames.

[0075] This invention proposes a cross-granularity weight reuse mechanism, aiming to directly induce fine-grained time-aligned labels using sentence-level supervision signals without manual annotation. Furthermore, to isolate teacher-model-specific acoustic biases, hard labels are used instead of soft probabilities as the final supervision signal.

[0076] In step S3, a heterogeneous multi-task student network is constructed and trained. To fully utilize multi-granularity supervision and avoid model homogenization, this step constructs a heterogeneous multi-task student network, which includes a heterogeneous feature extractor, a sentence-level main task branch, and a frame-level auxiliary task branch, and performs end-to-end training using a joint loss.

[0077] 1. Heterogeneous Feature Extractor: A pre-trained model with a different architecture than the teacher's model is selected as the student's feature extractor. Preferably, the teacher model is an emotion-specialized model (such as a model fine-tuned on large-scale emotion-based speech data), and the student model is a general-purpose acoustic model (such as Wav2Vec2, HuberT, etc.). For input audio Extract heterogeneous frame-level feature sequences:

[0078] ;

[0079] in, For the first Heterogeneous feature vectors of frames.

[0080] This invention constructs The heterogeneous architecture forces the student model to approximate the pseudo-labels generated by the teacher model from a new acoustic perspective, thereby achieving feature enhancement across model perspectives rather than simple knowledge replication.

[0081] 2. Multi-task branch building:

[0082] This invention constructs two physically parallel task processing branches at the back end of the backbone network, which are used to capture global semantics and local details, respectively.

[0083] (1) Sentence-level main task branch: includes a temporal aggregation module (such as global average pooling) and a classifier. First, for... Perform time-domain aggregation:

[0084] ;

[0085] in, This is a sentence-level global feature vector. This indicates a time-domain aggregation operation. For the first Heterogeneous feature vectors of frames Total number of frames For time frame indexing;

[0086] Then, the sentence-level sentiment probability distribution is calculated using a classifier:

[0087] ;

[0088] in, For sentence-level sentiment prediction probability distribution, For normalized activation functions, This is the weight matrix for the sentence-level classification heads. This is the bias vector for the sentence-level classification head;

[0089] (2) Frame-level auxiliary task branch: This is a point-to-point classifier that does not perform any temporal aggregation operations. It directly processes the features of each frame. Categorize:

[0090] ;

[0091] in, For the first Frame-level sentiment prediction probability distribution The weight matrix of the frame-level classification header. This is the bias vector for the frame-level classification header.

[0092] 3. Joint Loss Function and Training: The student model simultaneously receives real sentence-level sentiment labels. and frame-level hard pseudo-labels Supervision. The total loss function consists of sentence-level main task loss. and frame-level auxiliary task loss Weighted composition:

[0093] ;

[0094] Among them, sentence-level main task loss Used to constrain the model's accuracy in judging the overall sentiment tone, frame-level auxiliary task loss. Used to constrain the model's ability to fine-grained align with the emotional state of each frame. To balance the hyperparameters of the contributions of the two tasks, This forces the model to reach agreement with the teacher model's fine-grained judgments at local time steps before making a global decision, effectively preventing "shortcut learning" caused by the model ignoring local emotional details. Hard-label supervision is used here to filter out uncertainty noise. Both losses are expressed as cross-entropy.

[0095] ;

[0096] ;

[0097] in, The total number of samples in the original speech dataset. For speech sample indexing, The total number of emotion categories to be identified. For sentiment category indexing, For indicator functions, For the first The actual sentence-level sentiment labels of each voice sample For the first The sample is predicted as the first in the sentence-level main task branch. The probability value of emotion-like qualities. For the first Total number of frames corresponding to each speech sample For time frame indexing, For the first The first voice sample Frame-level hard pseudo-labels For the first The first voice sample In the frame-level auxiliary task branch, the predicted frame is the [number]. The probability value of sentiment-like characteristics. The entire outer training set is used during training. Perform end-to-end optimization.

[0098] In step S4, model inference occurs. After training, only the sentence-level main task branch of the student model is retained during the inference phase, while the frame-level auxiliary task branches and their parameters are discarded. For any input speech... ,pass Extracting heterogeneous frame-level feature sequences After pooling and aggregation, the data is fed into a sentence-level classifier, which outputs the final sentiment category prediction. This design ensures that no additional computational overhead is added during the inference stage, while internalizing fine-grained sentiment feature representations.

[0099] In summary, this invention eliminates pseudo-label bias through nested mutually exclusive partitioning, achieves zero-cost fine-grained label generation through cross-granularity weight reuse, and enhances the model's ability to model local emotional changes through heterogeneous multi-task joint training. Ultimately, it significantly improves the accuracy and robustness of speech emotion recognition without requiring frame-level manual annotation. This method is particularly suitable for complex speech scenarios such as emotion reversal and abrupt intonation changes, demonstrating good practicality and promotional value.

[0100] Example 2: Figure 3 As shown, this invention provides a speech emotion recognition system with cross-granularity weight reuse and heterogeneous multi-task implementation. This system is used to implement the speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task implementation of Embodiment 1 above, specifically including:

[0101] The data partitioning module 100 is used to construct a nested and mutually exclusive data partitioning system: the original speech training dataset with sentence-level sentiment labels is divided into multiple outer folds; for the training set of each outer fold, it is further divided into multiple mutually exclusive inner subsets; in each round of inner processing, one inner subset is selected as the target inference set, and the union of the remaining inner subsets is used as the teacher training set.

[0102] The pseudo-label generation module 200 is used to generate frame-level pseudo-labels based on cross-granularity weight reuse: In each round of inner layer processing, a teacher model is trained using the teacher training set. The teacher model obtains classification weights by minimizing sentence-level sentiment classification loss. For speech samples in the target inference set, the pooling layer of the teacher model is removed, the classification weights are kept unchanged, and the extracted frame-level feature sequences are classified frame by frame to generate the corresponding frame-level hard pseudo-label sequences.

[0103] The model training module 300 is used to construct and train a heterogeneous multi-task student network: a student model is constructed, which includes a pre-trained feature extractor heterogeneous with the teacher model, as well as parallel sentence-level main task branches and frame-level auxiliary task branches; using sentence-level ground truth labels and frame-level hard pseudo-label sequences from the original speech training dataset, the student model is trained end-to-end through a joint loss function, which is composed of a weighted sum of sentence-level main task loss and frame-level auxiliary task loss;

[0104] The inference module 400 is used for model inference: it uses the trained student model to perform speech emotion recognition, and only uses its sentence-level main task branch to perform sentence-level emotion classification on the input speech.

[0105] This system can be implemented through software, hardware, or a combination of both, and is suitable for embedded devices, servers, or cloud-based voice emotion recognition services.

[0106] This embodiment provides a speech emotion recognition system with cross-granularity weight reuse and heterogeneous multi-task implementation, used to implement the aforementioned speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task implementation. Therefore, the specific implementation of the speech emotion recognition system with cross-granularity weight reuse and heterogeneous multi-task implementation can be found in the embodiment section of the speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task implementation described above. For example, the data partitioning module 100, pseudo-label generation module 200, model training module 300, and inference module 400 are respectively used to implement steps S1, S2, S3, and S4 in the aforementioned speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task implementation. Therefore, its specific implementation can be referred to the description of the corresponding embodiments. To avoid redundancy, it will not be repeated here.

[0107] Example 3: This embodiment of the invention provides an electronic device, which includes a processor, a memory, and a bus system. The processor and the memory are connected through the bus system. The memory stores instructions, and the processor executes the instructions stored in the memory to implement the aforementioned cross-granularity weight reuse and heterogeneous multi-task speech emotion recognition method. This electronic device can be a smartphone, smart speaker, in-vehicle terminal, server, or other device with speech processing capabilities.

[0108] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0111] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task approach, characterized in that, include: S1. Construct a nested and mutually exclusive data partitioning system: Divide the original speech training dataset with sentence-level sentiment labels into multiple outer folds; For each outer fold's training set, it is further divided into multiple mutually exclusive inner subsets; in each round of inner processing, one inner subset is selected as the target reasoning set, and the union of the remaining inner subsets is used as the teacher training set. S2. Generate frame-level pseudo-labels based on cross-granularity weight reuse: In each round of inner layer processing, a teacher model is trained using the teacher training set. The teacher model obtains classification weights by minimizing sentence-level sentiment classification loss. For the speech samples in the target inference set, the pooling layer of the teacher model is removed, the classification weights are kept unchanged, and the extracted frame-level feature sequences are classified frame by frame to generate the corresponding frame-level hard pseudo-label sequences. S3. Construct and train a heterogeneous multi-task student network: Construct a student model that includes a pre-trained feature extractor heterogeneous with the teacher model, as well as parallel sentence-level main task branches and frame-level auxiliary task branches; Use the sentence-level real labels and the frame-level hard pseudo-label sequences from the original speech training dataset to train the student model end-to-end through a joint loss function, which is composed of a weighted sum of sentence-level main task loss and frame-level auxiliary task loss; S4. Model Inference: Use the trained student model to perform speech emotion recognition, using only its sentence-level main task branch to perform sentence-level emotion classification on the input speech.

2. The speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task as described in claim 1, characterized in that, In step S1, the nested mutually exclusive data partitioning system specifically includes: Let the original speech dataset be... ,in For the first The original audio of each speech sample. For the first The actual sentence-level sentiment labels of each voice sample The total number of samples in the original speech dataset. For the speech sample index, it is divided into One outer fold; For the Outer training set of the fold Further divide it into Let be mutually exclusive inner subsets. ; For the Secondary inner-level rotation, inner-level teacher training program Inner target reasoning set And satisfy ,in Indicates the first an inner subset, This represents the set subtraction operation. This represents the empty set.

3. The speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task as described in claim 1, characterized in that, In step S2, the generation of frame-level pseudo-tags based on cross-granularity weight reuse specifically includes: Based on the assumption that sentiment features have additive linear semantic consistency in vector space—that is, sentence-level sentiment feature vectors are linearly superimposed from frame-level sentiment feature vectors—the classification weight hyperplane used for classifying sentence-level features is reused for classifying frame-level features. For speech samples in the target inference set, its... Frame-level hard pseudo-labels The formula for generating it is: ； ； ； in, This is an operation to retrieve the index of the maximum value. For sentiment category indexing, Indicates the first Frame belongs to the Unnormalized score of sentiment-like characteristics For time frame indexing, Indicates the first Unnormalized score of the frame, It is a non-linear activation function. and For the weight matrix and bias vector, For projection layer parameters, For the first The input feature vector of the frame, Indicates that it is subject to, Represents the sentence-level loss function. This represents the actual sentence-level sentiment label corresponding to the speech sample. This represents a function that transforms a vector of logical values ​​into a probability distribution. The weights of the classifier to be optimized will be equal to the weights after optimization. , Indicates the total number of frames. For the first The input feature vector of the frame.

4. The speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task as described in claim 1, characterized in that, In step S3, the feature extractor of the heterogeneous multi-task student network The feature extractor used in the teacher model Different architectures of pre-trained models, and For general acoustic models, For the emotion-focused model, that is .

5. The speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task as described in claim 1, characterized in that, In step S3, the sentence-level main task branch includes a temporal aggregation module and a classifier, through... To achieve temporal aggregation, where This is a sentence-level global feature vector. This indicates a time-domain aggregation operation. Heterogeneous frame-level feature sequences extracted for the student model For the first Heterogeneous feature vectors of frames Total number of frames The time frame index is used; the frame-level auxiliary task branch is a point-to-point classifier that does not contain any temporal aggregation operations and directly performs sentiment classification on single-frame features.

6. The speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task as described in claim 1, characterized in that, In step S3, the joint loss function is defined as: ； in, For the joint loss function, For sentence-level main task loss, For frame-level auxiliary task loss, Hyperparameters to balance the contributions of the two tasks.

7. The speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task as described in claim 6, characterized in that, The sentence-level main task loss and frame-level auxiliary task loss The specific calculation method is as follows: ； ； in, The total number of samples in the original speech dataset. For speech sample indexing, The total number of emotion categories to be identified. For sentiment category indexing, For indicator functions, For the first The actual sentence-level sentiment labels of each voice sample For the first The sample is predicted as the first in the sentence-level main task branch. The probability value of emotion-like qualities. For the first Total number of frames corresponding to each speech sample For time frame indexing, For the first The first voice sample Frame-level hard pseudo-labels For the first The first voice sample In the frame-level auxiliary task branch, the predicted frame is the [number]. The probability value of a type of emotion.

8. The speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task as described in claim 1, characterized in that, After training is completed in step S3, in the model inference stage of step S4, the frame-level auxiliary task branch and its related parameters are discarded, and only the feature extractor and the sentence-level main task branch are retained for speech emotion recognition.

9. A speech emotion recognition system with cross-granularity weight reuse and heterogeneous multi-task implementation, characterized in that, The system is used to implement the speech emotion recognition method with cross-granularity weight reuse and heterogeneous multi-task as described in any one of claims 1 to 8, specifically including: The data partitioning module is used to construct a nested and mutually exclusive data partitioning system: the original speech training dataset with sentence-level sentiment labels is divided into multiple outer folds; for the training set of each outer fold, it is further divided into multiple mutually exclusive inner subsets; in each round of inner processing, one inner subset is selected as the target inference set, and the union of the remaining inner subsets is used as the teacher training set. The pseudo-label generation module is used to generate frame-level pseudo-labels based on cross-granularity weight reuse: In each round of inner layer processing, a teacher model is trained using the teacher training set. The teacher model obtains classification weights by minimizing sentence-level sentiment classification loss. For speech samples in the target inference set, the pooling layer of the teacher model is removed, the classification weights are kept unchanged, and the extracted frame-level feature sequences are classified frame by frame to generate the corresponding frame-level hard pseudo-label sequences. The model training module is used to construct and train a heterogeneous multi-task student network: a student model is constructed, which includes a pre-trained feature extractor heterogeneous with the teacher model, as well as parallel sentence-level main task branches and frame-level auxiliary task branches; using the sentence-level ground truth labels of the original speech training dataset and the frame-level hard pseudo-label sequences, the student model is trained end-to-end through a joint loss function, which is composed of a weighted sum of sentence-level main task loss and frame-level auxiliary task loss; The inference module is used for model inference: it performs speech emotion recognition using the trained student model, and performs sentence-level emotion classification on the input speech using only its sentence-level main task branch.

10. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the speech emotion recognition method with cross-granularity weight reuse and heterogeneous multitasking as described in any one of claims 1 to 8.

Images