Encoder training method, audio-visual speech recognition method and device, and storage medium
By training a self-attention mechanism encoder to extract paralinguistic features from audio and fusing them with a large language model, the problem of semantic understanding bias in complex scenarios of existing audiovisual speech recognition models is solved, achieving higher recognition accuracy and environmental adaptability.
Patent Information
- Application Number
- CN202511812332.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-03
- Publication Date
- 2026-02-06
AI Technical Summary
Existing audiovisual speech recognition models neglect paralinguistic information such as speaker's emotions, intonation, and rhythm when dealing with complex and ever-changing real-world scenarios, leading to semantic comprehension biases and transcription errors in dialogues rich in emotion or complex expressions.
By training an encoder with a self-attention mechanism, the emotion, emotional intensity, and prosodic information of audio samples are extracted, and a large language model is used for fusion processing to dynamically adjust the quality of audio and video signals to improve recognition accuracy.
It significantly improves the accuracy of audiovisual speech recognition in complex scenarios, eliminates ambiguity in transcription and understanding, and enhances the robustness and computational efficiency of the model in variable environments.
Smart Images
Figure CN121483232A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to the technical field of speech recognition, and in particular to an encoder training method, an audio-visual speech recognition method and device thereof, and a storage medium. BACKGROUND
[0002] Audio-Visual Speech Recognition (AVSR) aims to integrate the visual information (mainly the lip movement) and the audio information of the speaker to perform speech recognition. Its core advantage lies in simulating the multi-modal perception ability of humans, and when the audio signal is disturbed by noise (such as street noise or restaurant noise), the visual information can be used as a key supplement to significantly improve the robustness of recognition. In recent years, with the great success of Large Language Model (LLM) in the field of natural language processing, researchers have begun to combine it with AVSR (such as the MMS-LLaMA model). The introduction of the large language model enables the AVSR model not only to "understand" and "understand", but also to use the powerful context reasoning ability, world knowledge and semantic coherence generation ability of the LLM to correct recognition errors and fill in missing information, so that the recognition result is more coherent and accurate in semantics. SUMMARY
[0003] The inventors have noticed that in the related art, although LLM-based AVSR (LLM-based AVSR) has made progress, in complex and variable real-world scenarios, existing AVSR models (including models combined with LLM) mainly focus on "what is said" (linguistic content, i.e. text content), and completely ignore "how to say" (paralinguistic cues). Paralinguistic information, such as the speaker's emotion (happy, angry), tone (question, statement) and prosody (stress, pause), carries a lot of semantic information. For example, the sentence "You are really good" has completely opposite meanings in a happy and sarcastic tone. Existing models cannot distinguish this difference, resulting in serious semantic understanding bias and transcription errors when processing emotionally rich or complex conversations.
[0004] Accordingly, the present disclosure provides an encoder training method, which trains an encoder with a self-attention mechanism, so that the trained encoder can output audio situation information, emotional intensity information and prosody information, so that the audio-visual speech recognition method based on the trained encoder not only recognizes "what is said", but also understands "how to say", by introducing explicit modeling of paralinguistic features such as emotion and tone, providing more rich context for the large language model to eliminate ambiguity in transcription and understanding, and further improving the accuracy of audio-visual speech recognition in complex scenarios.
[0005] In a first aspect of the present disclosure, an encoder training method is provided, including: extracting a first feature sequence of an audio sample; inputting the first feature sequence into an encoder with a self-attention mechanism, so that the encoder captures long-distance dependencies of the first feature sequence by using the self-attention mechanism to generate a second feature sequence; performing average pooling on the second feature sequence in a time dimension to obtain an utterance feature; processing the utterance feature by using a sentiment classification head and an intensity classification head in the encoder to obtain sentiment prediction information and sentiment intensity prediction information, respectively; processing the second feature sequence by using a prosody regression head in the encoder to obtain prosody prediction information; determining a loss function value according to the sentiment prediction information, the sentiment intensity prediction information, the prosody prediction information, and labeled information of the audio sample; and training the encoder by using the loss function value.
[0006] In some embodiments, the determining the loss function value includes: determining a first sub-loss function value according to the sentiment prediction information and sentiment labeled information in the labeled information; determining a second sub-loss function value according to the sentiment intensity prediction information and intensity labeled information in the labeled information; determining a third sub-loss function value according to the prosody prediction information and prosody labeled information in the labeled information; and obtaining the loss function value according to the first sub-loss function value, the second sub-loss function value, and the third sub-loss function value.
[0007] In some embodiments, the obtaining the loss function value includes: calculating a weighted sum of the first sub-loss function value, the second sub-loss function value, and the third sub-loss function value to obtain the loss function value.
[0008] In some embodiments, the extracting the first feature sequence of the audio sample includes: processing the audio sample by using an audio feature extractor to obtain a spectrogram; and projecting the spectrogram to a preset embedding dimension to obtain the first feature sequence.
[0009] In a second aspect of the present disclosure, an encoder training apparatus is provided, including: a memory; and a processor coupled to the memory, the processor being configured to perform the encoder training method according to any of the above embodiments based on instructions stored in the memory.
[0010] In a third aspect of the present disclosure, an audio-visual speech recognition method is provided, comprising: processing a video frame in a to-be-processed video using a visual encoder to obtain a video feature; processing an audio in the to-be-processed video using a speech encoder to obtain an audio feature; processing the audio using a paralanguage encoder to obtain a paralanguage feature, wherein the paralanguage feature comprises emotional information, emotional intensity information and prosody information of the audio, and the paralanguage encoder is trained using the encoder training method of any of the above embodiments; fusing the audio feature and the paralanguage feature to obtain a first fused feature; weighting and fusing the first fused feature and the video feature to obtain a second fused feature; processing the second fused feature using a large language model to obtain an audio-visual speech recognition result.
[0011] In some embodiments, the weighting and fusing the first fused feature and the video feature comprises: determining a gating parameter according to a real-time quality indicator of the to-be-processed video; determining a first weight of the first fused feature and a second weight of the video feature according to the gating parameter; calculating a product of the first weight and the first fused feature to obtain a first sub-feature; calculating a product of the second weight and the video feature to obtain a second sub-feature; and calculating a sum of the first sub-feature and the second sub-feature to obtain the second fused feature.
[0012] In some embodiments, the first weight is the gating parameter; and a sum of the first weight and the second weight is a predetermined value.
[0013] In some embodiments, the real-time quality indicator comprises at least one of a signal-to-noise ratio of the audio and a degree of visual lip occlusion in the video frame.
[0014] In some embodiments, the fusing the audio feature and the paralanguage feature comprises: element-level adding the audio feature and the paralanguage feature to obtain the first fused feature.
[0015] In some embodiments, the obtaining an audio-visual speech recognition result comprises: predicting a complexity of each audio segment in the audio using an audio complexity predictor; determining a number of query tokens to be allocated to each audio segment according to the complexity of each audio segment and a speaking speed; and inputting the number of query tokens and the second fused feature into the large language model, so that the large language model processes the second fused feature using a computing resource corresponding to the number of query tokens to obtain the audio-visual speech recognition result.
[0016] In some embodiments, the determining the number of query tokens allocated to each audio segment comprises: calculating a weighted sum of a complexity and a speech rate of each audio segment to obtain a first allocation value; and obtaining the number of query tokens allocated to each audio segment according to the first allocation value, a query frequency, a frame rate of the audio feature, and a total number of frames.
[0017] In some embodiments, the obtaining the number of query tokens allocated to each audio segment comprises: calculating a product of the first allocation value, the query frequency, and the total number of frames of the audio feature to obtain a second allocation value; and calculating a ratio of the second allocation value to the frame rate of the audio feature to obtain the number of query tokens allocated to each audio segment.
[0018] In some embodiments, the training the audio complexity predictor comprises: performing random masking on an audio frame sequence sample to obtain a to-be-processed sample; processing the to-be-processed sample by using a teacher model to obtain predicted content of a masked part in the to-be-processed sample; obtaining a first loss value reflecting complexity of the masked part according to a deviation between the predicted content and original content of the masked part; processing an audio training sample by using a student model to obtain complexity of the audio training sample, wherein the audio training sample is the masked part in the to-be-processed sample; determining a second loss value according to the complexity and the first loss value; and training the student model by using the second loss value to obtain the audio complexity predictor.
[0019] In a fourth aspect of the present disclosure, an audio-visual speech recognition apparatus is provided, comprising: a memory; a processor coupled to the memory, the processor being configured to implement the audio-visual speech recognition method according to any one of the above embodiments based on instructions stored in the memory.
[0020] In a fifth aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the instructions, when executed by a processor, implement the method according to any one of the above embodiments.
[0021] In a sixth aspect of the present disclosure, a computer program product is provided, comprising computer instructions, and the computer instructions, when executed by a processor, implement the method according to any one of the above embodiments.
[0022] Other features and advantages of the present disclosure will be apparent from the following detailed description of exemplary embodiments thereof, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS
[0023] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 This is a flowchart illustrating an encoder training method according to an embodiment of the present disclosure;
[0025] Figure 2 This is a schematic diagram of the process for generating secondary language features according to an embodiment of the present disclosure;
[0026] Figure 3 This is a schematic diagram of the structure of an encoder training device according to an embodiment of the present disclosure;
[0027] Figure 4 This is a flowchart illustrating an audiovisual speech recognition method according to an embodiment of the present disclosure;
[0028] Figure 5 This is a flowchart illustrating another embodiment of the audiovisual speech recognition method of this disclosure;
[0029] Figure 6 This is a flowchart illustrating an audio complexity predictor training method according to an embodiment of the present disclosure.
[0030] Figure 7 This is a schematic diagram of the process for generating complexity pseudo-labels according to an embodiment of this disclosure;
[0031] Figure 8 This is a schematic diagram of the training process for an audio complexity predictor according to an embodiment of the present disclosure.
[0032] Figure 9 This is a flowchart illustrating another embodiment of the audiovisual speech recognition method of this disclosure;
[0033] Figure 10 This is a schematic diagram of the structure of an audiovisual speech recognition device according to an embodiment of the present disclosure. Detailed Implementation
[0034] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.
[0035] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of this disclosure.
[0036] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.
[0037] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.
[0038] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.
[0039] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0040] Figure 1 This is a schematic flowchart of an encoder training method according to an embodiment of the present disclosure. In some embodiments, the following encoder training method is performed by an encoder training device, including steps 11-17.
[0041] In step 11, the first feature sequence of the audio sample is extracted.
[0042] In some embodiments, an audio feature extractor (e.g., the Whisper model) is used to process the audio samples to obtain a spectrogram. ,in For time frames, This represents the Mel-band number. Next, the spectrogram is projected onto the preset embedding dimension. The first feature sequence is obtained.
[0043] In step 12, the first feature sequence is input into an encoder with a self-attention mechanism so that the encoder can use the self-attention mechanism to capture the long-distance dependencies of the first feature sequence to generate the second feature sequence.
[0044] In some embodiments, the encoder with a self-attention mechanism is... A shared backbone network consisting of standard Transformer encoder layers. This network utilizes a self-attention mechanism to capture long-range dependencies in speech sequences, outputting a context-rich feature sequence. ,in It refers to the number of time frames.
[0045] In step 13, the second feature sequence is averaged and pooled over time to obtain the discourse features.
[0046] It should be noted here that for feature sequences Perform average pooling over the time dimension to obtain a discourse feature (i.e., a global feature vector) that represents the entire sentence. .
[0047] In step 14, the utterance features are processed using the sentiment classification head and intensity classification head in the encoder to obtain sentiment prediction information and sentiment intensity prediction information, respectively.
[0048] It's important to note that the sentiment classification head is a linear layer that can process discourse features. Logical values mapped to multiple emotion categories For example, various emotion categories can include anger, fear, sadness, joy, surprise, disgust, trust, and anticipation.
[0049] The intensity classification head is a linear layer that can classify speech features. Logical values mapped to multiple levels of emotional intensity For example, various levels of emotional intensity include Level 1 (weak), Level 2 (moderate), and Level 3 (intense).
[0050] In step 15, the second feature sequence is processed using the prosodic regression head in the encoder to obtain prosodic prediction information.
[0051] It's important to note that the prosodic regression head is a lightweight linear layer that acts independently on the features at each time step. The fundamental frequency and energy of the frame are predicted to generate prosodic prediction information.
[0052] In step 16, the loss function value is determined based on the sentiment prediction information, sentiment intensity prediction information, prosody prediction information, and the annotation information of the audio samples.
[0053] In some embodiments, the step of determining the loss function value includes steps S101-S104.
[0054] S101. Determine the value of the first sub-loss function based on the sentiment prediction information and the sentiment annotation information in the annotation information. .
[0055] S102. Determine the value of the second sub-loss function based on the sentiment intensity prediction information and the intensity labeling information in the labeling information. .
[0056] S103. Determine the value of the third sub-loss function based on the prosody prediction information and the prosody annotation information in the annotation information. .
[0057] S104. Based on the value of the first sub-loss function Second sub-loss function value and the third sub-loss function value The loss function value is obtained.
[0058] In some embodiments, the value of the first sub-loss function is calculated. Second sub-loss function value and the third sub-loss function value The weighted sum of the values yields the loss function value, as shown in formula (1).
[0059] (1)
[0060] In formula (1), , and For weights.
[0061] In step 17, the encoder is trained using the loss function value.
[0062] In some embodiments, during training, the sentiment classification head is trained using cross-entropy loss, the intensity classification head is trained using cross-entropy loss, and the prosody regression head is trained using mean squared error loss to obtain the sub-language encoder.
[0063] In the encoder training method provided in the above embodiments of this disclosure, the speech features of audio samples are processed by the emotion classification head and intensity classification head in the encoder to obtain emotion prediction information and emotion intensity prediction information, respectively. The prosody regression head in the encoder is used to process the feature sequence obtained based on the self-attention mechanism to obtain prosody prediction information. Based on the emotion prediction information, emotion intensity prediction information, prosody prediction information and the annotation information of the audio samples, the loss function value is determined. The encoder is trained using the loss function value so that the trained encoder can output audio context information, emotion intensity information and prosody information. Thus, the audiovisual speech recognition method based on the trained encoder can not only recognize "what to say" but also understand "how to say it". By introducing explicit modeling of paralinguistic features such as emotion and intonation, richer context is provided for the large language model to eliminate ambiguity in transcription and understanding, and further improve the accuracy of audiovisual speech recognition in complex scenarios.
[0064] Figure 2 This is a schematic diagram of the process for generating secondary language features according to an embodiment of the present disclosure.
[0065] like Figure 2 As shown, the audio is processed using an audio feature extractor (e.g., the Whisper model) to obtain a spectrogram. The spectrogram is then projected onto a preset embedding dimension. The first feature sequence is obtained.
[0066] Next, the first feature sequence is input into an encoder with a self-attention mechanism (by... A shared backbone network consisting of standard Transformer encoder layers is used to enable the encoder to capture long-range dependencies in the first feature sequence using a self-attention mechanism, thus outputting a context-rich second feature sequence, i.e., a deep feature sequence. The deep feature sequence is then averaged and pooled over time to obtain the discourse features.
[0067] Next, a linear layer, acting as the head of sentiment classification, is used to process the discourse features to obtain sentiment classification information. A linear layer, acting as the head of intensity classification, is then used to process the discourse features to obtain sentiment intensity information. Finally, a linear layer, acting as the head of prosody regression, is used to process the deep feature sequence to obtain prosodic information.
[0068] Figure 3 This is a schematic diagram of the structure of an encoder training device according to an embodiment of the present disclosure.
[0069] like Figure 3 As shown, the encoder training device 30 can be represented in the form of a general-purpose computing device. The encoder training device 30 includes a memory 31, a processor 32, and a bus 33 connecting different system components.
[0070] The memory 31 may include, for example, system memory, non-volatile storage media, etc. System memory may store, for example, an operating system, application programs, a boot loader, and other programs. System memory may include volatile storage media, such as random access memory (RAM) and / or cache memory. Non-volatile storage media may store, for example, instructions for a corresponding embodiment of an encoder training method being executed. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, etc.
[0071] Processor 32 can be implemented using a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic devices, discrete hardware components such as discrete gates or transistors. Accordingly, each module, such as the acquisition module, calculation module, and adjustment module, can be implemented by executing instructions in the central processing unit (CPU) running memory to perform the corresponding steps, or by implementing dedicated circuitry to perform the corresponding steps.
[0072] For example, processor 32 is configured to execute instructions stored in memory 31, such as Figure 1 The method involved in any of the embodiments.
[0073] Bus 33 can use any of the various bus architectures. For example, bus architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, and the Peripheral Component Interconnect (PCI) bus.
[0074] The interfaces 34, 35, and 36 of the encoder training device 30, as well as the memory 31 and processor 32, can be connected via bus 33. Input / output interface 34 provides a connection interface for input / output devices such as monitors, mice, and keyboards. Network interface 35 provides a connection interface for various networked devices. Storage interface 36 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.
[0075] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations thereof, can be implemented by computer-readable program instructions.
[0076] These computer-readable program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device to produce a machine, such that execution of the instructions by the processor produces means for implementing the functions specified in one or more boxes of the flowchart and / or block diagram.
[0077] These computer-readable program instructions may also be stored in a computer-readable storage medium. These instructions cause a computer to work in a particular manner to produce an article of manufacture, including instructions that implement the functions specified in one or more boxes in a flowchart and / or block diagram.
[0078] This disclosure may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.
[0079] This disclosure also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement... Figure 1 The method involved in any of the embodiments.
[0080] This disclosure also provides a computer program product, including computer instructions, wherein the computer instructions, when executed by a processor, implement as follows: Figure 1 The method involved in any of the embodiments.
[0081] Figure 4This is a schematic flowchart of an audiovisual speech recognition method according to an embodiment of the present disclosure. In some embodiments, the following audiovisual speech recognition method steps are performed by an audiovisual speech recognition device, including steps 41-46.
[0082] In step 41, the video frames in the video to be processed are processed using a visual encoder to obtain video features.
[0083] In step 42, the audio in the video to be processed is processed using a speech encoder to obtain audio features.
[0084] In step 43, the audio is processed using a paralinguistic encoder to obtain paralinguistic features, which include emotional information, emotional intensity information, and prosodic information of the audio.
[0085] It should be noted here that the sub-language encoder utilizes... Figure 1 The encoder is trained using any one of the following methods.
[0086] In step 44, the audio features and the sub-language features are fused to obtain the first fused feature.
[0087] In some embodiments, audio features and sub-language features are added element-wise to obtain a first fused feature.
[0088] In step 45, the first fusion feature and the video feature are weighted and fused to obtain the second fusion feature.
[0089] In some embodiments, the step of weighted fusion of the first fusion feature and the video feature includes steps S201-S205.
[0090] S201. Determine the gating parameters based on the real-time quality indicators of the video to be processed.
[0091] For example, real-time quality metrics include at least one of the signal-to-noise ratio of audio and the degree of visual lip occlusion in video frames.
[0092] S202. Based on the gating parameters, determine the first weight of the first fusion feature and the second weight of the video feature.
[0093] For example, the first weight is a gating parameter, and the sum of the first weight and the second weight is a predetermined value.
[0094] For example, if the gating parameter is Then the first weight is The second weight is .
[0095] S203. Calculate the product of the first weight and the first fused feature to obtain the first sub-feature.
[0096] S204. Calculate the product of the second weight and the video feature to obtain the second sub-feature.
[0097] S205. Calculate the sum of the first sub-feature and the second sub-feature to obtain the second fused feature.
[0098] For example, let the first fusion feature be... The video features are Then the second fusion feature As shown in formula (2).
[0099] (2)
[0100] It's important to note that existing AVSR models typically employ static feature fusion strategies, such as simple feature concatenation or fixed-weighted averaging. However, in real-world scenarios, the quality of audio and video signals is dynamically changing. For example, in dimly lit but quiet environments, audio quality is high while video quality is low. In noisy but well-lit outdoor environments, video quality is high while audio quality is low. Static fusion strategies cannot adaptively adjust their dependence on audio and visual modalities based on real-time signal quality, leading to a sharp decline in overall system performance when a single modality is compromised.
[0101] To address this issue, this disclosure provides a dynamic adaptive modal fusion mechanism that enables the model to intelligently assess the quality of the current audio and video signals and dynamically adjust the fusion weights. When either modality is compromised, the model relies more on the other modality, thereby improving robustness in variable environments.
[0102] In step 46, the second fusion feature is processed using a large language model to obtain the audiovisual speech recognition result.
[0103] In the audiovisual speech recognition provided in the above embodiments of this disclosure, by weighted fusion of video features, audio features and sub-language features in the video to be processed to obtain fused features, and by using a large language model to process the fused features, the audiovisual speech recognition results in complex scenes can be effectively improved.
[0104] Figure 5 This is a schematic flowchart of an audiovisual speech recognition method according to an embodiment of the present disclosure. In some embodiments, the following audiovisual speech recognition method steps are performed by an audiovisual speech recognition device, including steps 51-58.
[0105] In step 51, the video frames in the video to be processed are processed using a visual encoder to obtain video features.
[0106] In step 52, the audio in the video to be processed is processed using a speech encoder to obtain audio features.
[0107] In step 53, the audio is processed using a paralinguistic encoder to obtain paralinguistic features, which include emotional information, emotional intensity information, and prosodic information of the audio.
[0108] It should be noted here that the sub-language encoder utilizes... Figure 1 The encoder is trained using any one of the following methods.
[0109] In step 54, the audio features and the sub-language features are fused to obtain the first fused feature.
[0110] In some embodiments, audio features and sub-language features are added element-wise to obtain a first fused feature.
[0111] In step 55, the first fusion feature and the video feature are weighted and fused to obtain the second fusion feature.
[0112] In some embodiments, the step of weighted fusion of the first fusion feature and the video feature is as shown in steps S201-S205 above.
[0113] In step 56, the complexity of each audio segment in the audio is predicted using an audio complexity predictor.
[0114] In some embodiments, an audio complexity predictor is trained using a teacher-student training paradigm, and the corresponding training methods and steps are described in the following embodiments.
[0115] In step 57, the number of query tokens assigned to each audio segment is determined based on the complexity and speaking rate of each audio segment.
[0116] In some embodiments, the step of determining the number of query tokens assigned to each audio segment includes steps S301-S302.
[0117] S301. Calculate the weighted sum of the complexity and speech rate of each audio segment to obtain the first assignment value.
[0118] For example, let The speaking speed of the audio clip. For the complexity of audio segments, For speech rate weights, As the complexity weight, then the first allocation value As shown in formula (3).
[0119] (3)
[0120] S302. Based on the first allocation value, query frequency, frame rate of audio features, and total number of frames, obtain the number of query tokens allocated to each audio segment.
[0121] In some embodiments, a second allocation value is obtained by multiplying the first allocation value, the query frequency, and the total number of frames of the audio features. Next, the ratio of the second allocation value to the frame rate of the audio features is calculated to obtain the number of query tokens allocated to each audio segment.
[0122] For example, the number of query tokens assigned to each audio segment. As shown in formula (4).
[0123] (4)
[0124] In formula (4), The speaking speed of the audio clip. For the complexity of audio segments, For speech rate weights, For complexity weights, For query frequency, The total number of frames for audio features. The frame rate (in Hertz) is the audio feature.
[0125] In step 58, the number of query tokens and the second fusion feature are input into the large language model so that the large language model can use the computing resources corresponding to the number of query tokens to process the second fusion feature and obtain the audiovisual speech recognition result.
[0126] For example, in determining Subsequently, the large model further allocates these query tokens to various time steps for feature compression based on the relative complexity ratio of each frame. Ultimately, this strategy significantly improves overall computational efficiency while maintaining high accuracy by directing more computational resources to high-complexity speech segments.
[0127] It's important to note that existing models (especially large LLM-based models) typically employ a "one-size-fits-all" strategy when processing speech signals. This means allocating the same computational resources (e.g., a fixed number of query tokens in a Transformer architecture) to both simple, clear words and complex, fast-paced, or noisy phrases. This static allocation strategy is extremely inefficient. It wastes a significant amount of unnecessary computation on simple segments, while potentially failing to recognize truly difficult segments due to insufficient computational power. This not only increases the overall computational cost of the model but also raises latency, hindering its deployment in real-time applications.
[0128] To address this issue, this disclosure provides a complexity-aware resource allocation strategy that enables the model to predict the "comprehension difficulty" of speech segments in real time and intelligently and unevenly allocate limited computing resources (such as query tokens in LLM) to the places where they are most needed (i.e. complex segments), thereby significantly reducing computational overhead and latency while maintaining or even improving accuracy.
[0129] Figure 6 This is a schematic flowchart illustrating an audio complexity predictor training method according to an embodiment of the present disclosure. In some embodiments, the following audio complexity predictor training method steps are performed by an audiovisual speech recognition device, including steps 61-68.
[0130] In step 61, the audio frame sequence samples are randomly masked to obtain the samples to be processed.
[0131] In step 62, the teacher model is used to process the sample to be processed to obtain the predicted content of the occluded part in the sample to be processed.
[0132] In step 63, a first loss value is obtained to reflect the complexity of the occluded part based on the deviation between the predicted content of the occluded part and the original content.
[0133] For example, such as Figure 7 As shown, the sample to be processed carrying the mask is input into the trained teacher model, so that the teacher model outputs the predicted content of the mask, where the bounding box in the sample corresponds to the mask part. Based on the deviation between the predicted content of the mask and the actual content of the mask, a first loss value L1 is obtained to reflect the complexity of the masked part, where a higher first loss value L1 indicates a higher complexity of the masked part. In this case, the obtained first loss value L1 can also be regarded as a pseudo-label of complexity.
[0134] In step 64, the audio training samples are processed using the student model to obtain the complexity of the audio training samples, where the audio training samples are the occluded parts of the samples to be processed.
[0135] In step 65, the second loss value is determined based on the complexity and the first loss value.
[0136] In step 66, the student model is trained using the second loss value to obtain the audio complexity predictor.
[0137] For example, such as Figure 8As shown, audio training samples are input into the student model so that the student model can predict the complexity C of the audio training samples, where the audio training samples are the occluded parts of the samples to be processed. Based on the complexity C and the first loss value L1, a second loss value L2 is determined, and then the student model is trained using the second loss value L2 to obtain the audio complexity predictor.
[0138] Figure 9 This is a flowchart illustrating another embodiment of the audiovisual speech recognition method disclosed herein.
[0139] like Figure 9 As shown, a visual encoder is used to process video frames in the video to obtain video features. A speech encoder is used to process the audio in the video to obtain audio features. A paralinguistic encoder is used to process the audio in the video to obtain paralinguistic features, which include emotional information, emotional intensity information, and prosodic information of the audio.
[0140] Next, the audio features and sub-language features are element-wise summed to obtain the first fused feature. Gating parameters are determined based on the real-time quality metrics of the video to be processed. For example, real-time quality metrics include at least one of the audio signal-to-noise ratio and the degree of visual lip occlusion in the video frame. Based on the gating parameters, a first weight for the first fused feature and a second weight for the video features are determined.
[0141] For example, if the gating parameter is Then the first weight is The second weight is .
[0142] Next, during the feature fusion process, the product of the first weight and the first fused feature is calculated to obtain the first sub-feature. The product of the second weight and the video feature is calculated to obtain the second sub-feature. The sum of the first sub-feature and the second sub-feature is calculated to obtain the second fused feature.
[0143] Furthermore, an audio complexity predictor is used to predict the complexity of each audio segment. Based on the complexity and speech rate of each audio segment, the number of query tokens assigned to each audio segment is determined. The number of query tokens and the second fusion feature are input into a large language model, so that the large language model can use the computational resources corresponding to the number of query tokens to process the second fusion feature, obtaining the output text describing the audiovisual speech recognition results.
[0144] Figure 10 This is a schematic diagram of the structure of an audiovisual speech recognition device according to an embodiment of this disclosure. Figure 10 As shown, the audiovisual voice recognition device 100 includes a memory 101, a processor 102, a bus 103, an input / output interface 104, a network interface 105, and a storage interface 106.
[0145] Figure 10 and Figure 3 The difference is that, in Figure 10 In the illustrated embodiment, processor 102 is configured to execute instructions stored in memory 101 as follows: Figures 4-6 The method involved in any of the embodiments.
[0146] This disclosure also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement... Figures 4-6 The method involved in any of the embodiments.
[0147] This disclosure also provides a computer program product, including computer instructions, wherein the computer instructions, when executed by a processor, implement as follows: Figures 4-6 The method involved in any of the embodiments.
[0148] By implementing the above embodiments of this disclosure, the following beneficial effects can be obtained.
[0149] 1. Paralinguistic Encoder: This type of encoder explicitly extracts paralinguistic features from raw audio through multi-task learning, including information such as emotion, emotional intensity, and prosody. Therefore, it can also be called a multi-task paralinguistic encoder. Its beneficial effects include:
[0150] (a) Eliminating semantic ambiguity: The “how to say” features extracted by the encoder provide crucial context for subsequent LLM. LLM can use these features to determine whether expressions such as “you’re great” are sincere praise or sarcasm, thereby outputting more accurate transcription results that better reflect human intentions.
[0151] (b) Improved accuracy in complex scenarios: When dealing with emotionally rich dialogues such as dramas, movies or heated daily discussions, paralinguistic information is the core of understanding the content. This disclosure significantly improves the recognition accuracy in such scenarios by capturing this information.
[0152] 2. Dynamic Modality Gating Mechanism: This mechanism replaces the traditional static fusion mechanism. It receives visual features and fused audio features (original audio features + paralinguistic features) in real time and performs weighted fusion based on weight parameters dynamically learned by the network. The learning process of the weight parameters implicitly considers the quality of the input signal. Its beneficial effects include:
[0153] (a) Enhanced environmental robustness: When high noise is detected at the audio input (leading to a decrease in feature quality), the gating mechanism automatically reduces the weight of audio features while increasing the weight of visual features (lip reading). Conversely, when the video input is dimly lit or occluded, it becomes more reliant on audio. This adaptive capability enables the model to maintain high performance in real-world environments with fluctuating audio and video quality, such as video conferencing and in-vehicle environments.
[0154] 3. Complexity-Guided Resource Allocation: By directly linking the "complexity score" output by the predictor to the allocation of LLM computational resources (specifically, query tokens in the Transformer), its beneficial effects include:
[0155] (a) Significantly improve computational efficiency: This is the core optimization for LLM applications. For simple fragments with low complexity (such as a clear "hello"), the system allocates only a very small number (e.g., 1-2) of query tokens, while for difficult fragments with high complexity (such as noisy, fast-paced terms), more query tokens (e.g., 8-10) are allocated for fine-grained processing.
[0156] (b) Reduced latency and cost: Through this intelligent allocation that "uses the best resources where they are most needed", the model runs with very low computational cost most of the time (processing simple segments), significantly reducing average computational cost and recognition latency, making it more suitable for resource-constrained edge devices or online services that require real-time response.
[0157] 4. Self-supervised Audio Complexity Predictor: This model employs a teacher-student training paradigm by constructing a lightweight prediction module. The "teacher" model (a more complex model) learns to reconstruct the masked audio signal, and its "reconstruction difficulty" (e.g., loss magnitude) is used as a pseudo-label of the "complexity." The lightweight "student" model learns to predict this complexity, with beneficial effects including:
[0158] (a) No manual annotation required: The core advantage is its "self-supervised" feature. In existing technologies, annotating the "comprehension difficulty" of an audio clip is extremely expensive and subjective. This disclosure automatically generates massive amounts of complexity labels through the proxy task of "reconstruction difficulty," thus solving the bottleneck of training data.
[0159] In summary, the solution provided in this disclosure has high accuracy, high robustness, and high efficiency in audiovisual speech recognition, making it widely applicable and promising.
[0160] 1. Scenarios that can be directly applied.
[0161] (a) Video Conferencing and Online Education: In applications such as Zoom, Tencent Meeting, and DingTalk, the network and physical environments (noise, lighting) of participants vary greatly. The dynamic fusion mechanism disclosed herein ensures accurate meeting subtitles even when network fluctuations lead to poor audio and video quality; the paralinguistic understanding capability can also be used in the future to assist in analyzing the meeting atmosphere or students' listening status.
[0162] (b) Voice Assistants and Interaction in Noisy Environments: Audio signals are severely compromised in in-vehicle scenarios (high tire noise, wind noise), public places (restaurants, streets), or industrial environments (factories). This disclosure can dynamically switch to a more visually-dependent (lip-reading) mode, while utilizing complexity prediction to centrally process noise-affected segments, significantly improving the wake-up rate and command recognition rate of voice assistants (such as Siri, Tmall Genie) and in-vehicle systems.
[0163] (c) Media content production and analysis: Used for post-production of movies, TV series, podcasts, and short videos. This disclosure not only generates more accurate subtitles, but its secondary language encoder can also automatically tag materials with emotional labels such as "sarcasm," "excitement," and "question," greatly facilitating the retrieval and editing of video content.
[0164] 2. Potential application scenarios.
[0165] (a) Intelligent Cockpit and Human-Vehicle Interaction: In the intelligent cockpit of the future, the driver's intentions will be expressed not only through language, but also through emotions and tone of voice. This disclosure enables the car to "read between the lines," understanding whether the driver's urging to "hurry up" is due to anxiety or excitement, thereby providing a more intelligent and human-centered navigation or entertainment response.
[0166] (b) Assisted Medicine and Mental Health: The paralinguistic comprehension capabilities disclosed herein can be used to develop assistive diagnostic tools. For example, by analyzing the flat emotional and prosodic features in the speech of patients with depression, or by analyzing the emotional expression disorders in children with autism, objective quantitative references can be provided to physicians.
[0167] (c) Virtual Human and Human-Computer Interaction: In metaverse or virtual anchor applications, this disclosure can serve as the "ears" of a virtual human. This enables it not only to understand user commands but also to perceive user emotions, thereby providing more empathetic and "human-like" interactive feedback (e.g., offering comfort when a user is frustrated).
[0168] In some embodiments, the functional units described above may be implemented as general-purpose processors, programmable logic controllers (PLCs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any suitable combination thereof for performing the functions described herein.
[0169] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0170] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.
Claims
1. An encoder training method, comprising: Extract the first feature sequence from the audio sample; The first feature sequence is input into an encoder with a self-attention mechanism, so that the encoder can use the self-attention mechanism to capture the long-distance dependencies of the first feature sequence to generate a second feature sequence. The second feature sequence is averaged and pooled over the time dimension to obtain the discourse features; The utterance features are processed using the sentiment classification head and intensity classification head in the encoder to obtain sentiment prediction information and sentiment intensity prediction information, respectively. The second feature sequence is processed using the prosodic regression head in the encoder to obtain prosodic prediction information; The loss function value is determined based on the sentiment prediction information, the sentiment intensity prediction information, the prosody prediction information, and the annotation information of the audio samples; The encoder is trained using the loss function value.
2. The encoder training method according to claim 1, wherein, The determination of the loss function value includes: The value of the first sub-loss function is determined based on the sentiment prediction information and the sentiment annotation information in the annotation information; The value of the second sub-loss function is determined based on the emotion intensity prediction information and the intensity annotation information in the annotation information; The value of the third sub-loss function is determined based on the prosody prediction information and the prosody annotation information in the annotation information; The loss function value is obtained based on the first sub-loss function value, the second sub-loss function value, and the third sub-loss function value.
3. The encoder training method according to claim 2, wherein, The process of obtaining the loss function value includes: The loss function value is obtained by calculating the weighted sum of the first sub-loss function value, the second sub-loss function value, and the third sub-loss function value.
4. The encoder training method according to any one of claims 1-3, wherein, The first feature sequence extracted from the audio samples includes: The audio samples are processed using an audio feature extractor to obtain a spectrogram; The spectrogram is projected onto a preset embedding dimension to obtain the first feature sequence.
5. An encoder training device, comprising: Memory; A processor, coupled to a memory, is configured to implement the encoder training method as described in any one of claims 1-4 based on the memory-stored instruction execution.
6. An audiovisual speech recognition method, comprising: The video frames in the video to be processed are processed using a visual encoder to obtain video features; The audio in the video to be processed is processed using a speech encoder to obtain audio features; The audio is processed using a paralinguistic encoder to obtain paralinguistic features, wherein the paralinguistic features include emotional information, emotional intensity information, and prosodic information of the audio, and the paralinguistic encoder is trained using the encoder training method according to any one of claims 1-4; The audio features and the secondary language features are fused to obtain the first fused feature; The first fusion feature and the video feature are weighted and fused to obtain the second fusion feature; The second fusion feature is processed using a large language model to obtain the audiovisual speech recognition result.
7. The audiovisual speech recognition method according to claim 6, wherein, The weighted fusion of the first fused feature and the video feature includes: The gating parameters are determined based on the real-time quality indicators of the video to be processed; Based on the gating parameters, determine the first weight of the first fused feature and the second weight of the video feature; Calculate the product of the first weight and the first fused feature to obtain the first sub-feature; Calculate the product of the second weight and the video feature to obtain the second sub-feature; The sum of the first sub-feature and the second sub-feature is calculated to obtain the second fused feature.
8. The audiovisual speech recognition method according to claim 7, wherein, The first weight is the gating parameter; The sum of the first weight and the second weight is a predetermined value.
9. The audiovisual speech recognition method according to claim 7, wherein, The real-time quality metrics include at least one of the signal-to-noise ratio of the audio and the degree of visual lip occlusion in the video frame.
10. The audiovisual speech recognition method according to claim 6, wherein, The fusion of the audio features and the secondary language features includes: The first fused feature is obtained by adding the audio features and the sub-language features element-wise.
11. The audiovisual speech recognition method according to any one of claims 6-10, wherein, The obtained audiovisual speech recognition results include: The complexity of each audio segment in the audio is predicted using an audio complexity predictor; The number of query tokens assigned to each audio segment is determined based on the complexity and speech rate of each audio segment; The query token count and the second fusion feature are input into the large language model so that the large language model can use the computing resources corresponding to the query token count to process the second fusion feature and obtain the audiovisual speech recognition result.
12. The audiovisual speech recognition method according to claim 11, wherein, The number of query tokens allocated to each audio segment includes: Calculate the weighted sum of complexity and speech rate for each audio segment to obtain the first allocation value; The number of query tokens allocated to each audio segment is obtained based on the first allocation value, the query frequency, the frame rate of the audio feature, and the total number of frames.
13. The audiovisual speech recognition method according to claim 12, wherein, The number of query tokens assigned to each audio segment includes: Calculate the product of the first allocation value, the query frequency, and the total number of frames of the audio features to obtain the second allocation value; Calculate the ratio of the second allocation value to the frame rate of the audio feature to obtain the number of query tokens allocated to each audio segment.
14. The audiovisual speech recognition method according to claim 11 further includes: Training the audio complexity predictor, wherein training the audio complexity predictor includes: The audio frame sequence samples are randomly masked to obtain the samples to be processed; The teacher model is used to process the sample to be processed to obtain the predicted content of the occluded part in the sample to be processed. Based on the deviation between the predicted content and the original content of the occluded part, a first loss value is obtained to reflect the complexity of the occluded part. The audio training samples are processed using a student model to obtain the complexity of the audio training samples, wherein the audio training samples are the occluded parts of the samples to be processed. Based on the complexity and the first loss value, determine the second loss value; The student model is trained using the second loss value to obtain the audio complexity predictor.
15. An audiovisual speech recognition device, comprising: Memory; A processor, coupled to a memory, is configured to implement the audiovisual speech recognition method as described in any one of claims 6-14 based on the execution of instructions stored in the memory.
16. A computer-readable storage medium, wherein, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-4 and 6-14.
17. A computer program product comprising computer instructions, wherein the computer instructions, when executed by a processor, implement the method as described in any one of claims 1-4, 6-14.