A human-computer interaction voice perception method and system based on gradient intelligent subnetwork pool

Abstract

The present application relates to the technical field of speech emotion recognition, in particular to a human-computer interaction speech perception method and system based on gradient intelligent tonal net pool. The method comprises: obtaining an emotion dataset; constructing a human-computer interaction speech perception model based on gradient intelligent tonal net pool, which comprises an acoustic cue perception purification module, a hierarchical acoustic essence coding module, a gradient harmonization sub-net pool module, a task-specific feature extraction module, a focus and confidence joint calibration module, an adaptive optimization strategy module, and a real-time reasoning and decision fusion module; using the constructed human-computer interaction speech perception model to make an emotion decision; and outputting the decision result. The present application fundamentally solves the problem of emotion information distortion and identity feature confusion caused by real environment noise through the acoustic cue perception purification module and the hierarchical acoustic essence coding.

Description

Technical Field

[0001] This invention relates to the field of speech emotion recognition technology, and in particular to a human-computer interaction speech perception method and system based on a gradient intelligent tuning subnet pool. Background Technology

[0002] Virtual digital human technology is experiencing explosive growth and is rapidly becoming widespread in areas such as virtual anchors, intelligent customer service, emotional companions, and human-computer interfaces. Unlike traditional task-oriented robots, the core value of modern virtual digital humans lies in their "human-like" nature—providing an interactive experience rich in emotion, warmth, and empathy. To overcome the "inhuman" feeling and the "uncanny valley" effect, virtual digital humans must not only be able to "understand" what users are saying but also "understand" their emotional state and provide real-time, appropriate emotional feedback.

[0003] Against this backdrop, Speech Emotion Recognition (SER) has become a core technology for endowing virtual digital humans with "empathy." However, directly applying existing SER technology to real-time interactive virtual digital humans has serious limitations and technical defects, resulting in a clunky interactive experience and frequent errors, specifically manifested in:

[0004] (1) The application scenarios for virtual digital humans are not quiet laboratories. Users may interact with them in noisy streets, cars, or living rooms where many people are talking. Existing SER models will experience a sharp decline in performance when faced with such real-world environmental noise. Traditional independent noise reduction modules are prone to erasing key acoustic cues of emotion, such as breath and faint vibrato, when removing noise, causing the virtual digital human to receive "flat" or "distorted" emotional signals, thus making incorrect judgments. (2) The confusion between emotion and identity features leads to misjudgments of "one-size-fits-all": Virtual digital humans need to have "memory" and be able to recognize specific users, such as owners or family members. However, existing SER models are prone to feature confusion. For example, the model may mistakenly identify a child's naturally high fundamental frequency as "excitement" or forcibly associate a man's deep voice with "depression". This makes it impossible for the virtual digital human to establish a personalized emotional baseline for specific users. When facing users of different genders, ages, or identities, it will make a large number of "misattributed" incorrect emotional judgments, making the interaction experience very poor. (3) To address the aforementioned confusion, existing technologies attempt to introduce multi-task learning (MTL), such as simultaneous identification of emotion, speaker (SR), and gender (GR). However, these methods generally employ a rigid "hard parameter sharing" architecture, meaning that all tasks (such as emotion and gender) must share the same feature extraction backbone. This leads to severe "negative transfer": the SER task attempts to learn generalization features, while the SR task attempts to learn identity features, and the two "fight" in the shared layer, resulting in limited accuracy for all tasks. The "brain" of the virtual digital human is thus in a state of internal imbalance. (4) An advanced virtual digital human should simultaneously understand "what" and "how" the user said it. However, existing technologies are blind when fusing ASR information. In noisy environments, the ASR module may identify the user's "(frustrated) too good" as "(happy) too good." If the virtual digital human "blindly trusts" this erroneous ASR result and gives a happy response, this "opposite" reaction will instantly destroy the user's trust and cause a disastrous interaction failure. Existing technologies generally lack a mechanism to dynamically evaluate the real-time credibility of auxiliary tasks, such as ASR and SR. (5) The training process of a complex model that can simultaneously process information such as noise reduction, emotion, semantics, and identity is very unstable. Mixing highly subjective "advanced" tasks such as SER with relatively objective "basic" tasks such as SR and ASR from the beginning will lead to difficulties in model convergence.

[0005] In summary, existing technologies suffer from serious deficiencies in noise resistance, feature purity, architectural conflicts, fusion reliability, and training efficiency, preventing virtual digital humans from making robust and accurate emotional interactions in complex real-world environments. Therefore, there is an urgent need to develop a novel voice emotion recognition method and system that can fundamentally solve these problems from the perspectives of architecture and training mechanisms, enabling virtual digital humans to truly possess robust, accurate, and intelligent "empathy" capabilities. Summary of the Invention

[0006] To address the problems of poor robustness due to environmental noise interference, severe confusion between emotional features and irrelevant features such as identity and gender, rigid conflicts and negative transfer in multi-task learning architectures, and unreliable auxiliary information fusion in real-time emotional interaction of virtual digital humans, this invention provides a human-computer interaction speech perception method and system based on a gradient-based intelligent tuning subnet pool. This fundamentally solves the above problems from the perspectives of architecture and training mechanism, enabling virtual digital humans to truly possess robust, accurate, and intelligent "empathic" capabilities.

[0007] Firstly, the present invention provides a human-computer interaction speech perception method based on a gradient intelligent tuning subnet pool, which adopts the following technical solution:

[0008] A human-computer interaction speech perception method based on gradient intelligent tuning subnet pooling includes:

[0009] Obtain the sentiment dataset;

[0010] A human-computer interaction speech perception model based on gradient intelligent tuning subnet pool is constructed, including an acoustic cue perception and purification module, a hierarchical acoustic essence encoding module, a gradient harmonization subnet pool module, a task-specific feature extraction module, a focus and confidence joint calibration module, an adaptive optimization strategy module, and a real-time reasoning and decision fusion module.

[0011] Emotional decision-making is carried out using a constructed human-computer interaction speech perception model.

[0012] Output the decision results.

[0013] Secondly, the present invention provides a human-computer interaction speech perception system based on a gradient intelligent tuning subnet pool, comprising:

[0014] The data acquisition module is configured to acquire sentiment datasets;

[0015] The model building module is configured to build a human-computer interaction speech perception model based on a gradient intelligent tuning subnet pool, including an acoustic cue perception and purification module, a hierarchical acoustic essence encoding module, a gradient harmonization subnet pool module, a task-specific feature extraction module, a focus and confidence joint calibration module, an adaptive optimization strategy module, and a real-time reasoning and decision fusion module.

[0016] The decision-making module is configured to make emotion decisions using a constructed human-computer interaction voice perception model.

[0017] The output module is configured to output the decision results.

[0018] The present invention provides a computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the human-computer interaction voice perception method based on a gradient intelligent tuning subnet pool.

[0019] Thirdly, the present invention provides a terminal device, including a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, the instructions being adapted to be loaded and executed by the processor to provide the human-computer interaction voice perception method based on a gradient intelligent tuning subnet pool.

[0020] In summary, the present invention has the following beneficial technical effects:

[0021] Compared to existing technologies, which suffer from limitations such as poor noise resistance in real-world environments, confusion between emotional and identity features, negative transfer in multi-task architectures, insufficient reliability of fusion decisions, and instability in training complex models, this invention addresses the core need of "endowing virtual digital humans with robust, accurate, and intelligent 'empathy' capabilities." It constructs a novel end-to-end multi-task speech perception solution based on deep architecture and dynamic strategies. The beneficial effects of this invention are mainly reflected in the following aspects:

[0022] First, this invention fundamentally solves the problem of emotional information distortion and identity feature confusion caused by real-world environmental noise through (1) an acoustic cue perception and purification module and (2) a hierarchical acoustic essence encoding. The former (purification module) intelligently filters out interfering noise while retaining acoustic cues such as breath and vibrato that are crucial to emotion; the latter (encoding module) extracts "acoustic essence" features that remain unchanged for the speaker's identity (such as fundamental frequency) through self-supervised pre-training. This provides a high-purity and highly robust data foundation for all subsequent tasks.

[0023] Secondly, the gradient harmonization subnet pool (3) designed in this invention overcomes the limitation of "negative transfer" caused by traditional "hard parameter sharing" in terms of architecture. This module intelligently allocates different sparse subnet paths for emotion tasks (SER) seeking generalization features and speaker tasks (SR) seeking identity features through a dynamic gating mechanism. This allows tasks with different optimization objectives to "coexist harmoniously" while sharing parameters, fundamentally solving gradient conflicts and breaking through the performance bottleneck of multi-task learning.

[0024] Furthermore, the task-specific feature extraction module (4) and the joint calibration module (5) of the present invention work together to solve the problems of "extensive" prediction and "imbalance" in loss calculation. The former (extraction module) refines the optimal decision basis for each of the different needs of ASR (focus on local) and SER (focus on global); the latter (calibration module) focuses on class imbalance and easily confused errors such as "neutral" and "anger" through mechanisms such as "modulation-type focus loss", which significantly improves the model's refined recognition ability.

[0025] Then, this invention perfectly solves the problems of "unstable training" and "blind fusion" in complex models through the (6) adaptive optimization strategy module. This module ensures stable convergence through the "progressive task activation" strategy; more importantly, its "confidence inverse weight adjustment" mechanism can dynamically evaluate the real-time credibility of auxiliary tasks such as ASR and SR. When the confidence of ASR is low under noise, the system will automatically reduce its weight to avoid catastrophic interaction failure caused by "blindly believing" erroneous semantics, which greatly enhances the decision-making intelligence and reliability of the system.

[0026] Ultimately, this invention fundamentally solves the problem of "inconsistency between training and inference" in traditional models during deployment through the (7) real-time inference and multi-stream decision fusion module. Traditional models, in pursuit of low latency, prune auxiliary task branches used during training during inference, leading to a significant decrease in robustness in real-world scenarios. This invention reuses the uncertainty assessment capability trained in module (6) and constructs a "multi-stream decision fusion gate" during inference. This fusion gate can, in real time, (1) utilize the "pure" emotional features obtained from adversarial training in module (6) and (2) introduce high-confidence information from auxiliary tasks such as ASR, SR, and GR to "calibrate" them. This ensures that the final emotional decision of the virtual digital human maintains both low latency and robustness to multi-dimensional information, enabling it to make the most reliable emotional feedback in complex and ever-changing real-world interactions.

[0027] In summary, the test results of the method of this invention on a publicly available dataset in simulated high-noise and multi-interference real-world interaction scenarios such as home and vehicle environments show that: under a 15dB signal-to-noise ratio environment, the emotion recognition accuracy (SER-ACC) is improved to 86.2%, and the overall F1 score (SER-F1) reaches 85.5%; in multi-task processing, the word error rate (ASR-WER) is reduced to 18.4%, and the speaker recognition accuracy (SR-ACC) reaches 97.1%; and while maintaining high performance, the single inference response time is only 280 milliseconds. This invention demonstrates superior performance in multiple core dimensions such as perception accuracy, environmental robustness, and decision intelligence, and possesses strong practical deployment value and engineering promotion potential. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of a human-computer interaction speech perception method based on a gradient intelligent tuning subnet pool according to Embodiment 1 of the present invention;

[0029] Figure 2 This is a performance graph of voice emotion recognition according to Embodiment 1 of the present invention;

[0030] Figure 3 This is a multi-task collaborative performance diagram of Embodiment 1 of the present invention;

[0031] Figure 4 This is the inference efficiency diagram of Embodiment 1 of the present invention;

[0032] Figure 5 This is a comprehensive performance diagram of Embodiment 1 of the present invention;

[0033] Figure 6 This is a schematic diagram of the overall structure of Embodiment 1 of the present invention.

[0034] Figure 7 This is a schematic diagram of the task-specific feature extraction module in Embodiment 1 of the present invention. Detailed Implementation

[0035] The present invention will be further described in detail below with reference to the accompanying drawings.

[0036] Example 1

[0037] Reference Figure 1 This embodiment of a human-computer interaction speech perception method based on a gradient intelligent tuning subnet pool includes:

[0038] A human-computer interaction speech perception method and system based on gradient intelligent tuning subnet pool. The system consists of seven key modules: acoustic cue perception and purification module, hierarchical acoustic essence encoding module, gradient harmonization subnet pool module, task-specific feature extraction module, focus and confidence joint calibration module, adaptive optimization strategy module, and real-time reasoning and decision fusion module.

[0039] (1) Acoustic cue perception and purification module:

[0040] Because human-computer interaction scenarios for virtual digital humans, such as home, in-vehicle, and outdoor environments, are often accompanied by complex background noise, reverberation, and interference from multiple conversations, the original speech signal has a low signal-to-noise ratio, which seriously affects the accuracy of downstream emotion and semantic recognition. To solve this problem, this invention deploys an end-to-end adaptive speech enhancement module at the front end of the system. Its purpose is to intelligently filter out interference noise while retaining acoustic cues that are crucial for downstream tasks such as emotion and identity recognition, rather than performing indiscriminate "one-size-fits-all" noise reduction.

[0041] 1) Configure the system in tThe original single-channel noisy voice input signal at any given time is a time-series waveform:

[0042] ,

[0043] in Let be the length of the signal's time series. To perform refined processing in the time-frequency domain, first... Performing a short-time Fourier transform (STFT) yields the complex spectrum:

[0044] ,

[0045] in F For frequency point index number, N This represents the frame index number.

[0046] Decompose the complex spectrum into its amplitude spectrum and phase spectrum :

[0047] ,

[0048] ,

[0049] 2) Amplitude spectrum Input into a deep neural network The network learns a multiplication mask that adapts to the downstream task objective. :

[0050] ,

[0051] The mask The range of values ​​is constrained to [0, 1], and is used to weight the original amplitude spectrum.

[0052] 3) Generate a mask Compared with the original noisy amplitude spectrum Element-wise multiplication yields the enhanced amplitude spectrum. :

[0053] ,

[0054] Subsequently, the enhanced amplitude spectrum was used and the original phase spectrum The enhanced timing waveform is reconstructed using inverse short-time Fourier transform (iSTFT). :

[0055] ,

[0056] (2) Layered acoustic essence coding module:

[0057] This module is the core hub connecting the underlying audio signal and the high-level multi-task semantics. It receives the enhanced waveform from (1). This module aims to solve two major technical challenges: Although the waveform output by module (1) is enhanced, it may still contain residual noise or environmental artifacts that have not been completely filtered out. Secondly, speech emotion (prosody, intonation) and speech content (phonemes, vocabulary) have complex long and short distance dependencies on the time scale and must be captured simultaneously. To solve the above problems, this module is constructed with a "multi-scale convolutional front-end" and a "hierarchical Transformer encoder". Its core capability comes from a unique "dual-objective self-supervised pre-training" stage, which forces the model to learn acoustic features that are invariant to noise, thereby providing highly robust and context-rich shared feature representations for all subsequent tasks (SER, ASR, SR, GR).

[0058] 1) Simultaneously sent in P Group of parallel one-dimensional convolutional layers Each group has a different convolutional kernel width. :

[0059] ,

[0060] Will P The group output features are concatenated along the channel dimension and passed through a projective convolutional layer. Dimensionality reduction and fusion are performed to obtain the initial frame sequence. :

[0061] ,

[0062] The sequence Z It will be used as the input to the hierarchical Transformer encoder.

[0063] 2) Initial frame sequence (as input to layer 0) Send in one Layer Transformer Encoder To model long-range temporal dependencies:

[0064] ,

[0065] in, It is the first The output hidden state sequence of the layer encoder.

[0066] 3) To provide optimal features that integrate low-level acoustic details and high-level semantics for subsequent tasks, this module performs the following on all... The output sequence of the layer ( to Perform dynamic weighted summation. First, calculate... trainable scalar weights Normalized attention score :

[0067] ,

[0068] Then, the feature sequences of all layers according to Weighted summation, and passed through a trainable global scaling factor. Adjustments are made to obtain the output:

[0069] ,

[0070] in, This is the robust context feature sequence that is finally output by this module.

[0071] (3) Gradient Harmonization Subnet Pool Module:

[0072] Traditional "hard parameter sharing" architectures force all downstream tasks, such as sentiment, semantics, and identity, to share the same set of parameters, which easily leads to conflicting optimization objectives and "negative transfer" problems between tasks. To address this issue, this module proposes a gradient harmonization subnet pool. This architecture uses a pool containing... A "pool" of parallel subnets replaces the single shared backbone. This is achieved by creating a "pool" for each task. Equipped with a dedicated gating network The system can dynamically and sparsely select the most suitable subnet combination for each input frame, thereby achieving efficient parameter sharing while fundamentally resolving gradient conflicts between tasks.

[0073] 1) Set the output feature sequence of module (2) as follows:

[0074] ,

[0075] in For the first The feature vector of the frame; For time frames; For feature dimensions. PGSA is composed of Parallel, parameter-independent subnets and a containing One task ( (dedicated gating network) constitute.

[0076] 2) For any of the tasks and its gating The system will Each frame in Calculate its route separately Log-values ​​of preferences for each subnet:

[0077] ,

[0078] in It is a gating Trainable linear layers.

[0079] Using a Top-k gating mechanism, only select The largest median There are 1 index, denoted as _ ... For the first This subnet will The sparse route weights are obtained by performing Softmax calculation on each value. :

[0080] ,

[0081] 3) Through each task Corresponding sparse weights For all K Gezi.com The output results are weighted and summed, and each task is dynamically synthesized. Output characteristics :

[0082] ,

[0083] Repeat this process, and this module will eventually become Each task generated its own unique feature sequence. :

[0084] ,

[0085] in They will be sent to their respective downstream task heads for processing.

[0086] 4) To prevent gate control Training crashes due to activating only a few "hot" subnets, leading to the introduction of auxiliary loss. For each task Its load balancing loss The calculation is as follows:

[0087] ,

[0088] in, For subnet Task The selected frame rate; For subnet In the mission The average route weight obtained from the above. It will be added as a regularization term to the total system loss. middle.

[0089] (4) Task-specific feature extraction module:

[0090] Due to the output of the gradient harmonization subnet pool module in (3) The corresponding downstream tasks have significantly different feature requirements. For example, the ASR task focuses on frame-by-frame local acoustic information, while the SER, SR, and GR tasks rely on global prosody, statistics, and focus features. Traditional methods often use simple average pooling or MLP, ignoring this task specificity, leading to information loss and suboptimal predictions. This module introduces focus temporal pooling, temporal weighted statistical pooling, and mapping mechanisms to extract the decision basis required by each task from the high-dimensional sequence to the maximum extent, providing high-precision prediction logarithms for the (5) focus and confidence joint calibration module.

[0091] 1) Set the dedicated input for the SER task as First, a trainable "sentiment focus query vector" is computed. With each frame Alignment score and normalized focus weights :

[0092] ,

[0093] ,

[0094] in, For trainable query vectors; The projection matrix; For the projection dimension.

[0095] Subsequently, a single context vector is obtained through weighted pooling. The final logarithmic value is obtained through a two-layer nonlinear mapping network:

[0096] ,

[0097] ,

[0098] in, The projection matrix is ​​a value. These are the weights of the mapping layer; For bias; This represents the logarithmic value for the sentiment category.

[0099] 2) Set the exclusive input for ASR tasks as follows ASR is a frame-by-frame transcription task, and its logarithmic value is directly calculated through a linear projection layer:

[0100] ,

[0101] in, is a trainable projection weight matrix; It is the bias vector; The size of the ASR vocabulary (including spaces).

[0102] 3) Set tasks The exclusive input is First, calculate the first order (mean) of the sequence. ) and second-order (standard deviation) Statistic:

[0103] ,

[0104] ,

[0105] Secondly, to capture time-sensitive characteristics, a trainable "temporal decay scalar" is introduced. ), calculate time series weights and weighted average :

[0106] ,

[0107] ,

[0108] Then, all statistics are concatenated into an enhanced statistical vector. And the logarithm is obtained through nonlinear mapping:

[0109] ,

[0110] ,

[0111] in, This is the concatenated statistical vector; and For the task Dedicated trainable parameters; for or The logarithm of the dimension.

[0112] (5) Focus and confidence joint calibration module:

[0113] It receives four sets of predicted logarithmic values ​​from the (4) task-specific feature extraction module. and the load balancing loss term from the gradient harmonization subnet pool module (3). Traditional methods often employ standard cross-entropy or CTC losses. These loss functions are sensitive to class imbalance (e.g., "neutral" far outnumbers "angry" in sentiment data) and lack targeted penalties for "confusing errors" (e.g., misclassifying "happy" as "angry"). This module designs "Modulation-Focused Loss" (MFL) and "Confidence-Gated CTC" (CG-CTC) to address these issues, and ultimately synthesizes the system's fundamental joint loss. .

[0114] 1) Set the prediction logarithm of the SER task to be... The real label is First, the probability is calculated using the Softmax function. :

[0115] ,

[0116] Then, the modulation focus loss (MFL) is calculated. :

[0117] ,

[0118] in, The true category predicted by the model The probability of. For category Static balancing weights. For focus modulation terms, To focus on hyperparameters. To obfuscate the penalty matrix. This is a weighted confusion penalty item.

[0119] 2) Both SR and GR tasks adopt the same approach. Perform calculations, but using their own independent parameters ( , , ):

[0120] ,

[0121] in .

[0122] 3) Set the prediction logarithm value for the ASR task to be... The actual label sequence is To address the lack of constraints on prediction uncertainty in the standard CTC loss, this invention employs confidence-gated CTC (CG-CTC). First, the standard CTC loss is calculated:

[0123] ,

[0124] Secondly, calculate a "non-empty confidence entropy penalty term". In each frame Calculate the sum of probabilities of non-empty characters. and the conditional entropy on this non-empty distribution :

[0125] ,

[0126] ,

[0127] ,

[0128] in For frames Predicted as character The probability of.

[0129] Calculate the mean of the "confidence-weighted entropy" for all frames. :

[0130] ,

[0131] Penalize frames that the model believes contain a character but is unsure which character it is.

[0132] CG-CTC loss for:

[0133] ,

[0134] in This is a hyperparameter for the penalty term.

[0135] 4) The system's fundamental joint loss The losses from all tasks and the load balancing losses from module (3) ( The weighted sum is:

[0136] ,

[0137] ,

[0138] in and These are hyperparameters used to balance different loss terms. It will serve as input for subsequent advanced training and optimization strategies.

[0139] (6) Adaptive optimization strategy module:

[0140] This module is the core optimization strategy center of the system. It does not directly participate in forward propagation, but receives the basic joint loss from the (5) focus and confidence joint calibration module during the training and backpropagation phases. and its components ( , , Traditional training methods employ static loss weighting and fixed gradient flow, leading to instability and a tendency to fall into suboptimal solutions. This module designs three dynamic strategies—"progressive task activation," "adaptive gradient polarity modulation," and "confidence inverse weighting"—to intelligently regulate the training process and the gradients themselves, fundamentally solving the problems of uneven convergence speeds across tasks, feature confusion, and unreliable auxiliary information.

[0141] 1) To address the significant difference in convergence difficulty between advanced emotion tasks (SER) and basic acoustic tasks (ASR, SR, GR), a progressive task activation strategy is dynamically adjusted. Mid-core mission loss weight This allows it to smoothly grow from 0 to the target value. Calculate the... training cycles weight :

[0142] ,

[0143] in, for The final target weight. For auxiliary tasks (ASR, SR, GR) in the first The average error rate or loss on a validation set over several periods. This is the initial error rate for the auxiliary task at the start of training. The error rate threshold for "fully converged" is preset for auxiliary tasks. The smoothing factor is η=2. This strategy ensures that the smoothing only occurs when the auxiliary tasks (ASR, SR, GR) have substantially converged on the validation set (i.e., ...). When the weight λSER(t) of the main task SER is fully activated, a smooth transition from "basic representation learning" to "advanced sentiment reasoning" is achieved.

[0144] 2) The adaptive gradient polarity modulation strategy corresponds to the specific implementation of the task-specific feature extraction module (4). During backpropagation, it... Gradients generated by the adversarial task (SR, GR) Perform dynamic polarity reversal. Set the shared parameter to... (i.e., the parameters of modules (2) and (3), the cooperative gradient is) (from) and ), adversarial gradient is (from) and Dynamic modulation factor Losses from the current batch of the combat mission Dynamically determined:

[0145] ,

[0146] in The maximum reversal coefficient, This is a loss sensitivity scalar.

[0147] Modulation is used for updating Total gradient :

[0148] ,

[0149] This mechanism is mandatory. Learning "pure" emotional characteristics unrelated to identity or gender, and in adversarial tasks that are already "difficult" ( When the intensity of the resistance is very high, the resistance intensity is automatically reduced to ensure convergence.

[0150] 3) To address the issue of the auxiliary task being "unreliable" under noise, a confidence inverse weighting control strategy is used for dynamic adjustment. Mid-level auxiliary tasks ( loss weights First, calculate the output of module (4). In the current batch Average prediction entropy on frames As a measure of the "uncertainty" of the task:

[0151] ,

[0152] ,

[0153] Then, auxiliary tasks Dynamic loss weights Modulated by its uncertainty:

[0154] ,

[0155] in This serves as the base weight for the task. This is the uncertainty sensitivity factor. The final... It is dynamically calculated during training as follows:

[0156] ,

[0157] in for or This mechanism makes it possible for ASR tasks to be "confused" by noise interference. When the value is very high, its loss weight This will be automatically reduced exponentially, thus preventing the "erroneous" auxiliary task from misleading the update of shared parameters in the current batch.

[0158] (7) Real-time reasoning and multi-stream decision fusion module

[0159] This module is the final execution unit when the present invention is deployed in interactive systems such as virtual digital humans. Traditional models only run the main task branch during inference in pursuit of low latency, but this discards the valuable contextual information provided by auxiliary tasks ASR, SR, and GR during training, resulting in inconsistency between training and inference and reducing the robustness of decision-making in complex scenarios. This module aims to construct a "multi-stream decision fusion gate" by activating all relevant task flows and reusing the "uncertainty" and "decoupling" characteristics trained in module (6) to perform real-time "calibration" of sentiment predictions from adversarial decoupling, so as to output the sentiment decision with the lowest latency and the strongest robustness.

[0160] 1) Set the initial noisy audio input for the reasoning phase as follows: First, the signal The system sequentially passes through the trained and optimized (1) acoustic cue perception and purification module. and (2) Layered acoustic essence coding module To obtain robust context feature sequences :

[0161] ,

[0162] Secondly, in order to make multi-flow decisions, They are fed into the (3) gradient harmonization subnet pool module and activated in parallel. Dedicated gating for each task (SER, ASR, SR, GR) And calculate their respective characteristic sequences. :

[0163] ,

[0164] in This step is consistent with the forward propagation of module (3) in the training phase.

[0165] 2) The data are fed into the corresponding task header in the (4) task-specific feature extraction module to obtain two key outputs: a) a pre-log feature vector for fusion and b) an uncertainty score for weighting.

[0166] The sentiment context vector is calculated using the "focused temporal pooling" mechanism. That is, the output of module 4, 1).

[0167] ,

[0168] and The global statistical vector is calculated using the "Temporal Weighted Statistical Pooling (TWSP)" mechanism. That is, the output of module 4, 3).

[0169] To obtain the credibility of the auxiliary task, the system continues to calculate. ( (as defined in Module 4), and reuse the average prediction entropy calculation logic defined in Module (6) to obtain the quantified uncertainty score. :

[0170] ,

[0171] in The sequence length (ASR is) SR and GR are 1). .

[0172] 3) Use As a "gating signal," it is used to dynamically integrate emotional representations. and auxiliary task representation First, calculate the confidence weights for the auxiliary task. :

[0173] ,

[0174] in This is a trained and solidified uncertainty sensitivity factor. A value close to 1 represents "trustworthy", while a value close to 0 represents "untrustworthy".

[0175] Secondly, to unify the dimensions, the statistical vectors of the auxiliary tasks are... Through a trainable projection layer (trained during the training phase along with Module 6). Mapping to The same Dimensions:

[0176] ,

[0177] Finally, all representation vectors are summed by confidence level to obtain the "calibrated" fused sentiment vector. :

[0178] ,

[0179] 4) Merge vector Input the same nonlinear mapping network as in module (4) 1) to calculate the final calibration logarithm value. :

[0180] ,

[0181] in These are all pre-trained and fixed parameters.

[0182] Ultimately, for application The function yields the category index with the highest confidence. :

[0183] ,

[0184] in This is the final sentiment decision result output by this system after robust calibration using multi-stream information.

[0185] Experimental verification:

[0186] To systematically verify the performance advantages of the method of this invention in multi-task speech perception (especially emotion recognition) under complex interactive scenarios, we used a publicly available emotion dataset that simulates a real noisy environment. The data was collected from multi-channel audio and labeled with: ① emotion category (neutral, happy, angry, etc.); ② speech-to-text; ③ speaker ID. The dataset contains approximately 1000 hours of speech. We divided the dataset into a training set (70%), a validation set (10%), and a test set (20%), ensuring that speakers and noise types in the test set did not appear in the training set.

[0187] To comprehensively evaluate the performance of the method of this invention, the following four mainstream comparative methods, which are representative of both academia and industry, are set up:

[0188] 1. SER-Only (benchmark): A standard, single-task model that only handles emotion recognition (such as CNN+LSTM), trained directly on noisy speech.

[0189] 2. Denoise + SER (Serial Concatenation): A traditional serial concatenation scheme that first uses an independent speech enhancement module (such as RNNoise) to reduce noise, and then sends its output into the SER-Only model.

[0190] 3. Hard-MTL (Hard Sharing): A standard multi-task learning method that forces all tasks such as ASR, SER, and SR to share the same feature extraction backbone, representing the "hard parameter sharing" architecture in the background technology.

[0191] 4. MMoE (Soft Sharing): An advanced soft parameter sharing multi-task learning method that assigns different expert combinations to different tasks through multiple gating networks, and is a strong competitor in the current field of multi-task learning.

[0192] All methods were evaluated under the same training and test set conditions. Evaluation metrics included: sentiment recognition accuracy (SER-ACC, %), sentiment recognition F1 score (SER-F1, %), word error rate (ASR-WER, %), speaker recognition accuracy (SR-ACC, %), and inference time (Time, ms).

[0193] Table 1. Comparison of data from different methods under five major indicators.

[0194] Method Name SER-ACC (%) SER-F1 (%) ASR-WER (%) SR-ACC (%) Time (ms) SER-Only 51.3 49.8 N / A N / A 190 Denoise+ SER 58.6 57.2 N / A N / A 320 Hard-MTL 70.4 69.1 29.5 88.3 310 MMoE 79.5 78.8 24.1 92.5 345 Method of the present invention 86.2 85.5 18.4 97.1 280

[0195] The experimental results are shown in Table 1 and Figure 2 , Figure 3 , Figure 4 , Figure 5 As shown. In order to... Figure 5 This invention provides an intuitive comparison of the comprehensive performance of five methods under different dimensional indicators. The indicators are standardized: for positive indicators (the higher the value, the better, i.e., SER-ACC, SER-F1, SR-ACC), the normalized value is retained; for negative indicators (the lower the value, the better, i.e., ASR-WER, Time), (1 - normalized value) is used to reverse them, so that all indicators are uniformly converted into the form of "the higher the value, the better".

[0196] Voice emotion recognition performance such as Figure 2As shown, the method of this invention performs best in the core task—emotion recognition. The benchmark SER-Only model performs the worst under noise (ACC 51.3%), demonstrating its lack of robustness. The Denoise+SER concatenation scheme (ACC 58.6%) shows some improvement, but the improvement is limited, which verifies the defect of traditional noise reduction modules that indiscriminately filter out key emotional cues (such as breath and vibrato). Hard-MTL (ACC 70.4%) and MMoE (ACC 79.5%) both outperform the former two, proving that multi-task learning does indeed have an advantage, and that MMoE's soft sharing mechanism is superior to Hard-MTL's hard sharing. The method of this invention (ACC 86.2%, F1 85.5%) significantly outperforms all competitors, including MMoE. This is thanks to the fact that (1) the acoustic cue perception purification module and (2) the hierarchical acoustic essence coding module provide a high-purity and high-robust feature base, while (5) the focus and confidence joint calibration module effectively solves the problem of imbalance of emotion categories (F1 score and ACC remain at a high level in sync).

[0197] Multi-task collaborative performance such as Figure 3 As shown, the advantages of this invention are equally evident in both the ASR and SR auxiliary tasks. SER-Only and Denoise + SER are single-task models, so this metric is not applicable. The Hard-MTL architecture suffers from the "negative transfer" effect, with an ASR word error rate (WER) as high as 29.5% and an SR accuracy of only 88.3%. MMoE mitigates the conflict through an expert network and outperforms Hard-MTL. The method of this invention achieves the lowest ASR-WER (18.4%) and the highest SR-ACC (97.1%). This strongly demonstrates the superiority of the gradient harmonization subnet pool module (3): this architecture fundamentally solves the gradient conflict between SER seeking generalization and SR seeking identity through dynamic sparse routing, enabling all tasks (main and auxiliary tasks) to achieve optimal performance.

[0198] Reasoning efficiency such as Figure 4 As shown, while ensuring the highest accuracy, the method of this invention also has significant efficiency advantages. SER-Only, as a simple model, is the fastest (190ms). Denoise + SER (320ms) and Hard-MTL (310ms) are both slower, the former being limited by the serial process and the latter having a bloated model structure. MMoE (345ms) has the highest computational overhead and the slowest inference because it needs to calculate all experts and gating networks. The method of this invention (280ms) is faster than all other multi-task and serial schemes. This is due to the "sparse activation" characteristic of the gradient harmonization subnet pool (3), which only activates some subnets during inference, and the computational efficiency is much higher than the "dense" gating of MMoE, perfectly meeting the needs of real-time interaction of virtual digital humans.

[0199] Overall performance such as Figure 5 As shown in the radar chart, this invention achieves the most balanced and powerful overall performance. SER-Only and Denoise + SER, lacking multi-tasking capabilities, exhibit "collapsed" profiles and limited capabilities. Hard-MTL and MMoE have relatively balanced performance profiles, but are completely enveloped by the profile of this invention's method (light green area). This invention's method has the largest performance profile area and is at the outermost edge in all four dimensions: speech emotion recognition, F1 score, speaker recognition, and word error rate. This indicates that the proposed solution achieves comprehensive leadership in perceptual accuracy, environmental robustness, decision intelligence, and real-time efficiency, representing a solution with optimal overall performance.

[0200] A computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the aforementioned human-computer interaction speech perception method based on a gradient intelligent tuning subnet pool.

[0201] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, the instructions being adapted to be loaded and executed by the processor to provide a human-computer interaction voice perception method based on a gradient intelligent tuning subnet pool.

[0202] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A human-computer interaction speech perception method based on gradient intelligent tuning subnet pool, characterized in that, include: Obtain the sentiment dataset; Constructing a human-computer interaction speech perception model based on a gradient-adjusted subnet pool, including: An acoustic cue perception and purification module is used to filter out interference noise; a hierarchical acoustic essence coding module uses a multi-scale convolutional front-end and a Transformer encoder to extract acoustic and semantic features; a gradient harmonization subnet pooling module, which includes introducing a gradient harmonization subnet pool, first setting the output feature sequence of the hierarchical acoustic essence coding module as follows: ,in For the first The feature vector of the frame; For time frames; As a feature dimension, the gradient harmonization subnet pool module consists of Parallel, parameter-independent subnets and contain The tasks include: emotion recognition (SER), speaker recognition (SR), speaker gender recognition (GR), and dedicated gating networks for speech recognition (ASR). Composition; then for any of the tasks and its gating ,right Each frame in Calculate its route separately Log-values ​​of preferences for each subnet: ,in It is a gating A trainable linear layer is then used; subsequently, a Top-k gating mechanism is employed to select... The largest median Each index will The sparse route weights are obtained by performing Softmax calculation on each value. Finally, through each task Corresponding sparse weights For all K Gezi.com The output results are weighted and summed, and each task is dynamically synthesized. Output characteristics Obtain the auxiliary losses of SER, SR, GR, and ASR, and calculate the load balancing loss value based on the auxiliary losses of SER, SR, GR, and ASR. The task-specific feature extraction module calculates the feature prediction values ​​for each of the task's SER, SR, GR, and ASR. The focus and confidence joint calibration module calculates the corresponding focus loss value using the predicted values ​​of SER, SR, and GR; calculates the corresponding ASR loss value using the predicted value of ASR; and weights the focus loss value of SER, SR, and GR, the ASR loss value, and the load balancing loss value to obtain the basic joint loss value. The adaptive optimization strategy module dynamically updates model parameters; A real-time reasoning and multi-stream decision fusion module is used to output sentiment decision results; Emotional decision-making is carried out using a constructed human-computer interaction speech perception model. Output the decision results.

2. The human-computer interaction speech perception method based on gradient intelligent tuning subnet pool according to claim 1, characterized in that, The construction process of the acoustic cue perception purification module includes setting the system in t The original single-channel noisy voice input signal at any given time is a time-series waveform: ,in The time series length of the signal, for Perform a short-time Fourier transform (STFT) to obtain the complex spectrum, and then decompose the complex spectrum into its amplitude spectrum. and phase spectrum : , Then the amplitude spectrum Input into deep neural network The original amplitude spectrum is weighted; finally, the generated mask is... Compared with the original noisy amplitude spectrum Element-wise multiplication yields the enhanced amplitude spectrum. Using the enhanced amplitude spectrum and the original phase spectrum, the enhanced time-series waveform is reconstructed through inverse short-time Fourier transform. .

3. The human-computer interaction speech perception method based on gradient intelligent tuning subnet pool according to claim 2, characterized in that, The construction process of the hierarchical acoustic essential coding module includes firstly... Simultaneously sent in P Group of parallel one-dimensional convolutional layers ,Will P The group output features are concatenated along the channel dimension and then passed through a projective convolutional layer. Dimensionality reduction and fusion are performed to obtain the initial frame sequence. : ,sequence Z As input to a hierarchical Transformer encoder, the initial frame sequence is then used. As input to layer 0 Send in Layer Transformer Encoder To model long-range temporal dependencies: ,in, It is the first The layer encoder outputs a sequence of hidden states, ultimately providing features that fuse acoustic and semantic information for all... The output sequences of the layer are dynamically weighted and summed, and trainable scalar weights are calculated. The normalized attention score is used to obtain the context feature sequence.

4. The human-computer interaction speech perception method based on gradient intelligent tuning subnet pool according to claim 3, characterized in that, The task-specific feature extraction module first sets the specific input for the SER task as... Calculate trainable sentiment focus query vectors With each frame Alignment score The focus weights are normalized, and then a single context vector is obtained through weighted pooling. The final predicted logarithmic value is obtained through a two-layer nonlinear mapping network. Then set the exclusive input for ASR tasks as follows: ASR is a frame-by-frame transcription task, and its logarithmic value is directly calculated through a linear projection layer: in, is a trainable projection weight matrix; It is the bias vector; Define the size of the ASR vocabulary; finally, set the task. The exclusive input is Calculate the first-order statistic of the sequence and second-order statistics Calculate time series weights and weighted average Then all statistics are concatenated into an enhanced statistical vector. The predicted logarithm is obtained through nonlinear mapping: , ,in, This is the concatenated statistical vector; and For the task Dedicated trainable parameters; for or The predicted logarithm of the dimension.

5. The human-computer interaction speech perception method based on gradient intelligent tuning subnet pool according to claim 4, characterized in that, The construction process of the focus and confidence joint calibration module includes: first, setting the prediction logarithm of the SER task to be... And calculate the modulation focus loss MFL : ,in, The true category predicted by the model The probability, For category Static balance weights, For focus modulation terms, To focus on hyperparameters, To obfuscate the penalty matrix, For weighted confusion penalty items; also adopt Calculate the focus loss values ​​for SR and GR, and then set the prediction logarithm value for the ASR task to be... The actual label sequence is The standard CTC loss is calculated using confidence-gated CTC. Secondly, in each frame Calculate the sum of probabilities of non-empty characters. and the conditional entropy on a non-empty distribution Finally, the mean of the confidence-weighted entropy of all frames is calculated. According to the formula Gaining loss ,in The hyperparameters for the penalty term are used; the weighted loss values ​​of SER, SR, GR, ASR, and load balancing are used to obtain the basic joint loss value.

6. The human-computer interaction speech perception method based on gradient intelligent tuning subnet pool according to claim 5, characterized in that, The construction process of the adaptive optimization strategy module includes receiving the basic joint loss, SER, SR, and GR focal loss values ​​and ASR loss values ​​from the focal and confidence joint calibration module, and adjusting the gradient.

Images