An audio and video fusion method and device, electronic equipment and storage medium

Abstract

This application discloses an audio and video fusion method, apparatus, electronic device, and storage medium, relating to the field of data processing technology. The method includes: acquiring the audio and video streams of a target object; extracting and analyzing audio and video features respectively to generate multimodal data; inputting the multimodal data into a pre-trained multimodal encoder containing at least one cascaded encoder layer; dynamically adjusting the fusion threshold based on the multimodal data in each encoder layer; selectively performing cross-modal fusion based on the comparison result between the fusion threshold and the fusion necessity weight to obtain the output features of that layer; and decoding the features output by the last encoder layer to obtain the speech recognition result. This method can ensure speech recognition accuracy in complex acoustic environments while achieving on-demand computing power allocation, reducing redundant model computation and terminal deployment power consumption, and improving the model's environmental adaptability and recognition robustness.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an audio and video fusion method, apparatus, electronic device and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence and Internet of Things technologies, voice interaction has become the mainstream human-computer interaction method. Automatic speech recognition technology is widely used in various terminal devices such as smartphones, smart homes, and in-vehicle systems. End-to-end speech recognition models based on deep learning have achieved near-human level recognition accuracy in quiet environments, but their performance still faces huge challenges in complex real-world acoustic environments.

[0003] In related technologies, audiovisual speech recognition solutions generally adopt a full-time fusion strategy, which forces cross-modal attention calculations regardless of the current environmental noise level. This results in serious computational redundancy, leading to high model power consumption and large response latency, which severely restricts its deployment and application on resource-constrained terminal devices. At the same time, existing multimodal fusion algorithms mostly use fixed fusion weights and thresholds, lacking the ability to finely perceive and adapt to different types of noise. They cannot dynamically adjust the fusion strategy according to the noise interference, and are prone to poor fusion results in specific noise environments, or even problems such as the introduction of irrelevant visual information interfering with recognition performance. Summary of the Invention

[0004] This application provides an audio and video fusion method, apparatus, electronic device, and storage medium. It can solve the problems in related technologies where the use of a full-time fusion strategy in audiovisual speech recognition leads to redundant computing power, high power consumption, and large response latency. Furthermore, the use of fixed fusion weights and thresholds lacks the ability to finely adapt to different noise environments, resulting in poor recognition performance in complex noisy scenes and susceptibility to interference from irrelevant visual information.

[0005] According to a first aspect of this application, an audio and video fusion method is provided, comprising:

[0006] The audio and video streams of the target object are acquired, audio features and video features are extracted respectively, and the audio and video features are analyzed to generate multimodal data. The multimodal data includes at least audio features, audio clarity score, noise type, noise features, video features, and video clarity score. Multimodal data is input into a pre-trained multimodal encoder, which includes at least one cascaded encoder layer; In each encoder layer of the multimodal encoder, the fusion threshold is dynamically adjusted according to the multimodal data, and cross-modal fusion is selectively performed based on the comparison result of the fusion threshold and the calculated fusion necessity weight to obtain the output features of the encoder layer. The features output by the last encoder layer in the multimodal encoder are decoded to obtain the speech recognition result of the target object.

[0007] According to a second aspect of this application, an audio and video fusion apparatus is provided, comprising: The generation module is configured to acquire the audio stream and video stream of the target object, extract audio features and video features respectively, and analyze the audio features and video features to generate multimodal data. The multimodal data includes at least audio features, audio clarity score, noise type, noise features, video features, and video clarity score. The input module is configured to input multimodal data into a pre-trained multimodal encoder, the multimodal encoder including at least one cascaded encoder layer; The fusion module is configured to dynamically adjust the fusion threshold based on the multimodal data in each encoder layer of the multimodal encoder, and selectively perform cross-modal fusion based on the comparison result of the fusion threshold and the calculated fusion necessity weight to obtain the output features of the encoder layer. The decoding module is configured to decode the features output by the last encoder layer in the multimodal encoder to obtain the speech recognition result of the target object.

[0008] According to a third aspect of this application, an electronic device is provided, comprising: At least one processor; and memory that is communicatively connected to at least one processor; The memory stores instructions that can be executed by at least one processor, which are executed by at least one processor to enable the at least one processor to perform the audio and video fusion method of the first aspect described above.

[0009] According to a fourth aspect of this application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to perform the audio and video fusion method of the first aspect described above.

[0010] According to a fifth aspect of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the audio and video fusion method as described in the first aspect above.

[0011] This application addresses the following issues in audio-visual fusion speech recognition: First, features are extracted and analyzed from the audio and video streams of the target object, generating multimodal data including audio features, audio clarity scores, noise types, noise features, video features, and video clarity scores. This multimodal data is then input into a multimodal encoder containing cascaded encoder layers. In each encoder layer, the fusion threshold is dynamically adjusted based on the multimodal data. Cross-modal fusion is selectively performed based on the comparison between the fusion threshold and the fusion necessity weights, ultimately decoding the speech recognition result. Therefore, this approach solves the problems of computational redundancy, high power consumption, and large response latency caused by the all-time fusion strategy in audiovisual speech recognition, as well as the lack of fine-grained adaptation to different noise environments due to fixed fusion weights and thresholds. These issues lead to poor recognition performance in complex noise scenarios and susceptibility to interference from irrelevant visual information. This approach achieves the technical effect of ensuring speech recognition accuracy in complex acoustic environments while realizing sparse inference with on-demand computational power allocation, significantly reducing redundant model computation and terminal deployment power consumption, and significantly improving the model's adaptive adaptability and recognition robustness to different noise environments.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0013] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart illustrating an audio and video fusion method provided in an embodiment of this application; Figure 2 A flowchart illustrating another audio and video fusion method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an audio and video fusion device provided in an embodiment of this application. Detailed Implementation

[0015] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0016] The audio and video fusion method, apparatus, electronic device, and storage medium of this application are described below with reference to the accompanying drawings.

[0017] Figure 1 This is a flowchart illustrating an audio and video fusion method provided in an embodiment of this application.

[0018] like Figure 1 As shown, the method includes the following steps: Step 101: Obtain the audio stream and video stream of the target object, extract audio features and video features respectively, and analyze the audio features and video features to generate multimodal data. The multimodal data includes at least audio features, audio clarity score, noise type, noise features, video features, and video clarity score.

[0019] In some embodiments, an audio stream of the target object is acquired through an audio acquisition device, and a video stream of the target object is acquired through an image acquisition device. The audio stream is continuous temporal audio data containing the speech information of the target object, and the video stream is continuous temporal image data containing the face, especially the lip region, of the target object. After preprocessing operations such as frame segmentation, windowing, and fast Fourier transform on the acquired audio stream, a high-dimensional audio feature sequence that can characterize the acoustic characteristics of speech is extracted. The acquired video stream is first used to locate the region of interest in the lips through face detection and key point localization, and then a temporal video feature sequence that can characterize the lip movement characteristics of the target object is extracted through a visual feature extraction network. The extracted audio features are subjected to deep analysis. An audio clarity score between 0 and 1 is calculated through a regression network, and a noise classification network is used to identify noise types and generate corresponding noise features. The noise types include competitive speech noise, non-steady-state noise, impulse noise, and steady-state noise. The extracted video features are analyzed, and a video clarity score between 0 and 1 is calculated through a regression network. Finally, the above data are integrated to generate multimodal data.

[0020] This step can comprehensively acquire and quantify the core information and quality status of audio and video modalities, providing accurate and comprehensive data support for subsequent multimodal processing, and ensuring the accuracy and environmental adaptability of subsequent fusion decisions.

[0021] Step 102: Input the multimodal data into a pre-trained multimodal encoder, which includes at least one cascaded encoder layer.

[0022] In some embodiments, after the multimodal data is generated, the multimodal data, which includes audio features, audio clarity scores, noise types, noise features, video features, and video clarity scores, is input into the pre-trained multimodal encoder.

[0023] A multimodal encoder is a core network structure used for encoding and processing multimodal features of audio and video and for context modeling. It adopts a cascaded hierarchical architecture, containing at least one encoder layer. Multiple encoder layers are arranged in series. The output features of the previous encoder layer are used as the input data of the next encoder layer, realizing progressive feature processing and deep encoding.

[0024] Before being put into use, the multimodal encoder is pre-trained and its parameters are optimized using multimodal speech datasets containing different noise environments and different audio and video qualities. This enables it to have the feature encoding capability to adapt to complex scenarios and to perform stable and efficient feature processing on the input multimodal data. It provides a standardized network carrier and basic processing framework for subsequent hierarchical feature processing.

[0025] This step utilizes a cascaded architecture pre-trained multimodal encoder to handle multimodal data, providing a unified processing path for layer-by-layer processing of multimodal features. This ensures the consistency and continuity of feature encoding, while the pre-trained encoder can adapt to diverse input scenarios, improving the stability and generalization ability of the entire speech recognition process.

[0026] Step 103: In each encoder layer of the multimodal encoder, the fusion threshold is dynamically adjusted according to the multimodal data, and cross-modal fusion is selectively performed based on the comparison result of the fusion threshold and the calculated fusion necessity weight to obtain the output features of the encoder layer.

[0027] In some embodiments, after multimodal data is input into a multimodal encoder, the corresponding feature processing flow is completed in each encoder layer. Within each encoder layer, the fusion threshold corresponding to that encoder layer is dynamically adjusted based on core information such as noise type, noise features, audio clarity score, and video clarity score contained in the input multimodal data. The fusion threshold is a critical value used to determine whether to trigger cross-modal fusion. Its adjustment process combines the environmental noise state and audio / video signal quality represented by the current multimodal data, adjusting inversely with the noise intensity; the higher the noise intensity, the lower the fusion threshold.

[0028] Subsequently, the fusion necessity weight of the current layer is calculated through a gating network. The fusion necessity weight is then compared with the dynamically adjusted fusion threshold. Based on the comparison result, it is determined whether to perform cross-modal fusion operation. When the fusion necessity weight reaches or exceeds the fusion threshold, cross-modal fusion processing of audio features and video features is performed. Otherwise, cross-modal fusion is skipped and the feature processing of this layer is completed directly, ultimately generating the output features corresponding to this encoder layer.

[0029] This step enables on-demand triggering of fusion operations through dynamic threshold adjustment and conditional cross-modal fusion selection, effectively reducing unnecessary computational consumption. At the same time, it can adapt to different noise environments, improving the environmental adaptability of feature processing and the robustness of speech recognition.

[0030] Step 104: Decode the features output by the last encoder layer in the multimodal encoder to obtain the speech recognition result of the target object.

[0031] In some embodiments, after all encoder layers of the multimodal encoder have completed layer-by-layer feature processing, the final encoded feature output by the last encoder layer in the cascaded architecture is obtained. This feature is a high-dimensional feature representation that has been processed layer by layer and completed context modeling, fully carrying effective speech information and modal complementarity information adapted to the current acoustic environment.

[0032] The final encoded features are input into a pre-trained decoder that works in conjunction with the multimodal encoder. The decoder uses a Transformer-based decoding architecture, which can perform sequence mapping on the input deep encoded features, converting the high-dimensional feature sequence into the corresponding text sequence, completing the accurate transcription of the speech content, and finally outputting the speech recognition result corresponding to the target object.

[0033] The decoder completes joint optimization with the multimodal encoder during the pre-training stage, which can fully adapt to the feature format and feature distribution of the encoder output, avoiding recognition deviation caused by feature adaptation problems.

[0034] This step completes end-to-end decoding by taking the deep coding features output by the encoder. It can make full use of the robust features that have been optimized in a hierarchical manner to achieve accurate restoration of speech content, ensure the accuracy of speech recognition results in complex acoustic environments, and fully realize the entire process of speech recognition that integrates audio and video.

[0035] Compared with related technologies, in this embodiment, by acquiring the audio and video streams of the target object, audio features and video features are extracted respectively, and the audio and video features are analyzed to generate multimodal data. The multimodal data includes at least audio features, audio clarity score, noise type, noise features, video features, and video clarity score. The multimodal data is input into a pre-trained multimodal encoder, which includes at least one cascaded encoder layer. In each encoder layer of the multimodal encoder, the fusion threshold is dynamically adjusted according to the multimodal data, and cross-modal fusion is selectively performed based on the comparison result of the fusion threshold and the calculated fusion necessity weight to obtain the output features of that encoder layer. The features output by the last encoder layer in the multimodal encoder are decoded to obtain the speech recognition result of the target object. This technology can solve the problems of redundant computing power, high power consumption, and large response latency caused by the full-time fusion strategy in audiovisual speech recognition, as well as the lack of fine-grained adaptation to different noise environments when using fixed fusion weights and thresholds, resulting in poor recognition performance in complex noise scenarios and susceptibility to interference from irrelevant visual information. It achieves the technical effect of sparse inference with on-demand allocation of computing power while ensuring the accuracy of speech recognition in complex acoustic environments, significantly reducing redundant model calculations and terminal deployment power consumption, and significantly improving the model's adaptive adaptation ability and recognition robustness to different noise environments.

[0036] Figure 2 A flowchart illustrating another audio and video fusion method provided in this application embodiment includes the following steps: Step 201: Perform frame segmentation, windowing, and fast Fourier transform on the audio stream to obtain audio features.

[0037] In some embodiments, the acquired target audio stream is subjected to frame segmentation, windowing, and fast Fourier transform processing to extract the corresponding audio features.

[0038] The audio stream is a continuous time-domain speech signal with short-term stable acoustic characteristics. Frame processing divides the continuous audio stream into multiple continuous audio frames according to a preset frame length and frame shift. The frame length and frame shift can be adapted to the sampling rate of the speech signal. Typically, the frame length is selected as 20-30ms, and the frame shift is selected as half or one-third of the frame length to ensure the continuity of information between adjacent audio frames.

[0039] Each segmented audio frame undergoes windowing processing, employing window functions such as the Hamming window and Hanning window, adapted for speech signal processing. Multiplying the window function with the time-domain data of the audio frame suppresses signal jumps at both ends of the frame, reducing spectral leakage during subsequent spectral analysis. After windowing, a Fast Fourier Transform (FFT) is performed on each windowed audio frame to convert the time-domain audio signal to the frequency domain, obtaining the frequency domain representation of the corresponding audio frame. Based on the frequency domain data, high-dimensional audio feature sequences, such as the Mel spectrum and Mel cepstral coefficients, which characterize the acoustic properties of speech, are then extracted.

[0040] This step, through standardized framing, windowing, and fast Fourier transform processing, can accurately extract effective speech acoustic features from the audio stream, fully preserving the frequency domain characteristics and temporal continuity of the speech. This provides an accurate feature basis for subsequent audio quality analysis and noise identification, ensuring the effectiveness and stability of the audio features.

[0041] Step 202: Perform face detection and key point localization on the video stream to extract the region of interest for the lips, and extract lip motion features through a preset visual feature extraction network to obtain video features.

[0042] In some embodiments, face detection and key point localization are performed on the acquired target object video stream to extract the region of interest for the lips, and lip motion features are extracted through a preset visual feature extraction network to obtain the corresponding video features.

[0043] The video stream consists of consecutive temporal image frames. For each frame, face detection is first performed. Face detection algorithms identify the face region of the target object within the image frame, determining the face's position and bounding box. After face region detection, facial landmark localization is performed on the detected face regions, locating multiple key feature points of the face. Emphasis is placed on locating the contour landmarks of the lip region, including core lip features such as the cupid's bow, valley, and corners of the mouth. Based on the located lip landmarks, a region of interest (ROI) is determined, and a local image region containing only lip movement information is cropped, eliminating interference from other areas of the face and irrelevant background information. The consecutive temporal ROI images are then input into a pre-defined visual feature extraction network. This network can employ 3D convolutional neural networks, ResNet, or Vision Transformer architectures. Through multi-layer convolution and attention calculations, a temporal video feature sequence characterizing the continuous movement changes of the target object's lips is extracted.

[0044] This step accurately locates the effective lip area through face detection and key point localization, significantly filtering out interference from irrelevant visual information. Then, a dedicated visual feature extraction network fully captures the temporal motion features of the lips, providing effective visual features that are highly relevant to the speech content for subsequent multimodal fusion, ensuring the relevance and effectiveness of the visual features.

[0045] Step 203: Input the audio features into the pre-trained regression network and noise classification estimation network to obtain the audio clarity score, noise type and noise features corresponding to the audio features.

[0046] In some embodiments, the extracted audio features are input into a pre-trained regression network and a noise classification estimation network to obtain the audio clarity score, noise type, and noise features corresponding to the audio features. The extracted audio feature sequence is input in parallel into the pre-trained regression network and noise classification estimation network. The regression network is a deep neural network trained on a large amount of labeled audio data. Based on the input audio features, it can quantify and calculate an audio clarity score between 0 and 1. The higher the audio clarity score, the clearer the target speech signal in the current audio stream and the lower the degree of noise interference. The lower the score, the more severe the noise interference and the lower the proportion of effective information in the speech signal.

[0047] The noise classification and estimation network is a classification and feature extraction network pre-trained on audio datasets labeled with multiple types of noise. It can identify the types of noise present in the current audio environment based on the input audio features. The noise types include four major categories: competitive speech noise, non-steady-state noise, impulse noise, and steady-state noise. At the same time, it can extract the acoustic characteristics of the corresponding noise types and generate noise features that can characterize the noise interference intensity and time-frequency domain distribution characteristics.

[0048] This step, through a dedicated pre-trained network, can accurately quantify the clarity of audio signals, while simultaneously achieving precise identification of noise types and effective extraction of noise features. It comprehensively grasps the interference characteristics of the current acoustic environment, providing accurate judgment criteria for subsequent dynamic adjustment of fusion thresholds and fusion decisions, and ensuring the environmental adaptability of subsequent fusion strategies.

[0049] Step 204: Input the video features into the regression network to obtain a video sharpness score.

[0050] In some embodiments, the extracted video features are input into a pre-trained regression network to obtain a video sharpness score corresponding to the video features. The extracted temporal video feature sequence is input into the pre-trained regression network, which has been optimized through pre-training with a large amount of lip video feature data of varying quality. Based on the input video features, this network can analyze core interference factors affecting the effectiveness of visual features in the video stream, such as motion blur, occlusion, and lighting conditions in the lip region, and quantify the video sharpness score between 0 and 1. A higher video sharpness score indicates clear imaging of the lip region in the current video stream, good lighting conditions, no obvious occlusion, and complete and effective lip motion features. A lower score indicates problems such as blurring, occlusion, or abnormal lighting in the lip region, resulting in lower effectiveness and reliability of the visual features.

[0051] This step accurately quantifies the effectiveness of video features through a pre-trained regression network, obtaining a standardized video clarity score. This score can intuitively represent the signal quality of the visual modality, providing a reliable visual reference for subsequent cross-modal fusion decisions and avoiding interference from low-quality visual features in the speech recognition process.

[0052] Step 205: Input the multimodal data into a pre-trained multimodal encoder, which includes at least one cascaded encoder layer.

[0053] For a description of step 205, please refer to the description of step 102 in the above embodiment. This embodiment will not repeat the details further.

[0054] Step 206: Obtain the corresponding basic threshold from the preset threshold library according to the noise type, and adjust the basic threshold according to the noise intensity to obtain the fusion threshold. The noise intensity is determined according to the audio clarity score. The lower the audio clarity score, the higher the noise intensity.

[0055] In some embodiments, in each encoder layer of the multimodal encoder, a base threshold corresponding to the noise type identified in the multimodal data is first retrieved from a pre-built threshold library. The threshold library stores base thresholds adapted to the acoustic characteristics and speech interference mechanisms of various noise types. Differentiated base thresholds are set for competitive speech noise, non-steady-state noise, impulse noise, and steady-state noise. Lower base thresholds are set for noise types with stronger speech interference that are difficult to suppress using traditional audio processing, while medium-level base thresholds are set for steady-state noise that can be effectively suppressed by conventional noise reduction methods. Then, the noise intensity of the current acoustic environment is determined based on the audio clarity score. The audio clarity score ranges from 0 to 1; a lower score indicates more severe noise interference to the speech signal, and the corresponding noise intensity is higher, showing a negative correlation. Finally, based on the base threshold and noise intensity, the final fusion threshold is calculated using a preset formula:

[0056] in, This represents the adaptive fusion threshold of the i-th layer encoder. This represents the preset base threshold for the noise type corresponding to the i-th layer encoder. This represents the learnable scaling factor. This represents the audio clarity score, (1- () indicates noise intensity.

[0057] This step achieves precise adaptation of the fusion trigger threshold to the current acoustic environment by matching a differentiated basic threshold with the noise type and dynamically adjusting the fusion threshold in combination with the noise intensity, thus balancing recognition performance and computing power consumption.

[0058] Step 207: Calculate the fusion necessity weights based on the audio features, video features, and noise type of the current encoder layer.

[0059] In some embodiments, in each encoder layer of the multimodal encoder, the fusion necessity weights are calculated based on the audio features, video features, and noise types contained in the multimodal data input to the current encoder layer. The calculation process is implemented using a pre-trained multilayer perceptron. The multilayer perceptron first performs feature mapping and concatenation on the input audio and video features to extract complementary representations of the audio and video modalities. Then, it combines the encoded features corresponding to the noise types, processes them through fully connected layers and activation functions, and finally outputs fusion necessity weights with values ​​ranging from 0 to 1.

[0060] The necessity weight of fusion is used to quantify the necessity of introducing video features to assist audio speech recognition in the current environment. The higher the weight value, the more serious the noise interference of the current audio signal, and the stronger the need to supplement speech information through cross-modal fusion. The lower the weight value, the clearer and more reliable the current audio signal, and there is no need to introduce video features to supplement it.

[0061] This step combines audio and video dual-modal features with noise type to complete weight calculation, which can comprehensively and accurately quantify the actual needs of cross-modal fusion, providing an accurate and reliable decision-making basis for subsequent selective cross-modal fusion.

[0062] Step 208: If the fusion necessity weight is less than the fusion threshold, then the audio features of the current encoder layer are used as the output features of the encoder layer, and the cross-modal fusion calculation is skipped.

[0063] In some embodiments, after calculating the fusion necessity weight and dynamically adjusting the fusion threshold, the fusion necessity weight and the fusion threshold are compared numerically. If the fusion necessity weight is less than the fusion threshold, the system determines that the current acoustic environment is a quiet or high signal-to-noise ratio environment, and the effective speech information carried by the current audio features is complete and reliable, without the need to introduce video features for supplementation.

[0064] At this point, the audio features of the current encoder layer after self-attention processing are directly used as the final output features of the encoder layer. At the same time, the cross-modal attention fusion calculation of this layer is completely skipped, and cross-modal interaction, matrix operation and other related operations between audio features and video features are no longer performed.

[0065] This step achieves hard sparse computation for cross-modal fusion through threshold comparison. In scenarios where fusion is not required, redundant computation of the visual pathway is directly cut off, which greatly reduces the amount of computation and computing power consumption of the encoder layer, effectively reduces the inference latency of the model, and avoids interference of irrelevant visual information on clear audio features, thus ensuring the accuracy and processing efficiency of speech recognition in high signal-to-noise ratio environments.

[0066] Step 209: If the fusion necessity weight is greater than or equal to the fusion threshold, then perform cross-modal fusion and use the fusion result as the output feature of the encoder layer.

[0067] In some embodiments, after comparing the fusion necessity weight with the fusion threshold, if the fusion necessity weight is greater than or equal to the fusion threshold, the system determines that the current acoustic environment is a noisy environment, the current audio features are severely affected by noise, and the accuracy of speech recognition cannot be guaranteed by relying solely on single-modal audio features.

[0068] At this point, cross-modal fusion computation is triggered, and the audio features and video features of the current encoder layer after self-attention processing are subjected to cross-modal interaction and fusion processing. The complementary information between audio features and video features is captured through the cross-modal attention mechanism to complete the fusion and optimization of features. Finally, the fused features obtained by the fusion processing are used as the output features of the encoder layer and passed to the next level encoder layer.

[0069] This step triggers cross-modal fusion in a timely manner when the scenario requiring fusion is determined. It can make full use of the lip movement features in the video modality that are not affected by acoustic noise, and supplement the effective speech information in the audio features that are contaminated by noise. This effectively improves the robustness and integrity of speech features in complex noise environments and ensures the accuracy and stability of speech recognition in low signal-to-noise ratio environments.

[0070] Furthermore, based on step 209, if the fusion necessity weight is greater than or equal to the fusion threshold, cross-modal fusion is performed, and the fusion result is used as the output feature of the encoder layer, including: Using multiple parallel temporal offsets, the audio and video features of the current encoder layer are temporally aligned and cross-modal attention is calculated to obtain multiple scale attention outputs; The weight prediction network generates fusion weights corresponding to attention outputs at multiple scales, and then performs a weighted summation of the attention outputs at multiple scales based on their respective fusion weights to obtain multi-scale fusion features. The multi-scale fusion features are weighted and fused with the audio features of the current encoder layer to obtain the output features of the encoder layer.

[0071] In this embodiment, when cross-modal fusion is triggered by a fusion necessity weight greater than or equal to a fusion threshold, multiple parallel temporal offsets are first established. Each temporal offset covers different audio and video delay ranges to adapt to the physical asynchrony between audio and video streams in real-world scenarios. For each temporal offset, the audio and video features of the current encoder layer are temporally aligned, and the corresponding cross-modal attention is calculated to obtain multiple scale attention outputs. The calculation formula for cross-modal attention is as follows:

[0072] in, , These are the learnable weight matrices for audio and video, respectively. These represent the audio features of the i-th layer encoder at time step T and time step T, respectively. The video features, The projected feature dimension. Use a scaling factor to stabilize training.

[0073] Subsequently, the attention outputs at multiple scales are input into a pre-trained weight prediction network to generate fusion weights corresponding to each scale's attention output. Based on these fusion weights, the attention outputs at multiple scales are weighted and summed to obtain the multi-scale fusion feature. The calculation formula for the multi-scale fusion feature is as follows:

[0074] in, The learnable weight matrix for the video; The weight for the k-th scale is used to quantify the contribution of different time series offsets; For the i-th layer encoder at time step Video characteristics; For cross-modal attention.

[0075] Finally, the multi-scale fusion features are weighted and fused with the audio features of the current encoder layer using fusion coefficients calculated by a dynamic gating network to obtain the output features of the encoder layer.

[0076]

[0077] in, This is the final output of the i-th layer; Calculate the fusion coefficients for the dynamic gating network; It is a multi-scale fusion feature; For audio self-processing features, self-attention calculation is performed on the audio to model contextual information and output the self-processed features. .

[0078] This process adapts to the asynchronous problem of audio-visual recording through multi-scale temporal offset and adaptive weighted fusion, fully exploring the complementarity of bimodal features and effectively improving the robustness of feature encoding and the accuracy of speech recognition in complex environments.

[0079] Step 210: Input the features output from the last encoder layer into the decoder module. The decoder module is built on the Transformer architecture and is used to decode the input features to generate speech recognition results.

[0080] In some embodiments, after the multimodal encoder completes feature processing at all levels, the features output from the last encoder layer in the cascaded architecture are input into the corresponding decoder module. This decoder module, built on the Transformer architecture, performs joint pre-training and parameter optimization with the multimodal encoder. It can fully adapt to the feature dimension, feature distribution, and encoding logic of the encoder output, avoiding the impact of feature adaptation deviations on decoding accuracy. The core function of the decoder module is to perform sequence decoding processing on the input high-dimensional deep encoded features. It maps the feature sequence, which has been optimized layer by layer by the multimodal encoder and integrates complementary audio-visual information and contextual semantic information, into a text sequence representation corresponding to the speech content, providing core decoding processing support for the final speech recognition output.

[0081] This step establishes a mapping path from deep features to text content by using a dedicated decoder based on the Transformer architecture to receive the final output features of the encoder. Relying on the powerful sequence modeling capabilities of the Transformer architecture, it accurately captures the contextual semantic relationships in the feature sequence, ensuring the accuracy and stability of the subsequent decoding process.

[0082] Step 211: The features are decoded by the decoder module and converted into speech recognition results in text form, which are then output as the speech recognition results of the target object.

[0083] In some embodiments, after receiving the features output from the last layer of the multimodal encoder, the decoder module performs end-to-end decoding processing on these features. Through the self-attention mechanism and sequence mapping logic of the Transformer architecture, it performs frame-by-frame semantic parsing and content reconstruction of the input high-dimensional feature sequence, converting the deep feature sequence representing speech information word-by-word into the corresponding text sequence, generating a text-based speech recognition result. This text result completely reconstructs the speech content contained in the target object's audio stream, and finally, this text-based speech recognition result is output as the final speech recognition result corresponding to the target object.

[0084] This step, through standardized decoding processing by the decoder, achieves accurate conversion from high-dimensional features to readable text content. It can fully utilize the robust features output by the multimodal encoder to completely restore the speech content of the target object, effectively ensuring the accuracy and completeness of speech recognition results in complex acoustic environments, and completing the entire closed loop of audio and video fusion speech recognition.

[0085] Figure 3 This is a schematic diagram of the structure of an audio and video fusion device provided in an embodiment of this application, as shown below. Figure 3 As shown, it includes: generation module 301, input module 302, fusion module 303, and decoding module 304.

[0086] The generation module 301 is configured to acquire the audio stream and video stream of the target object, extract audio features and video features respectively, and analyze the audio features and video features to generate multimodal data. The multimodal data includes at least audio features, audio clarity score, noise type, noise features, video features, and video clarity score. Input module 302 is configured to input multimodal data into a pre-trained multimodal encoder, the multimodal encoder including at least one cascaded encoder layer; The fusion module 303 is configured to dynamically adjust the fusion threshold based on the multimodal data in each encoder layer of the multimodal encoder, and selectively perform cross-modal fusion based on the comparison result of the fusion threshold and the calculated fusion necessity weight to obtain the output features of the encoder layer. The decoding module 304 is configured to decode the features output by the last encoder layer in the multimodal encoder to obtain the speech recognition result of the target object.

[0087] In some examples of this embodiment, the generation module 301 is specifically configured to perform frame segmentation, windowing, and fast Fourier transform processing on the audio stream to obtain audio features; perform face detection and key point localization on the video stream to extract the region of interest for the lips, and extract lip motion features through a preset visual feature extraction network to obtain video features; input the audio features into a pre-trained regression network and a noise classification estimation network to obtain the audio clarity score, noise type, and noise features corresponding to the audio features; and input the video features into the regression network to obtain the video clarity score.

[0088] In some examples of this embodiment, the fusion module 303 is specifically configured to obtain the corresponding basic threshold from a preset threshold library according to the noise type, and adjust the basic threshold according to the noise intensity to obtain the fusion threshold. The noise intensity is determined according to the audio clarity score, and the lower the audio clarity score, the higher the noise intensity.

[0089] In some examples of this embodiment, the fusion module 303 is specifically configured to calculate the fusion necessity weight based on the audio features, video features, and noise type of the current encoder layer; if the fusion necessity weight is less than the fusion threshold, the audio features of the current encoder layer are used as the output features of the encoder layer, and the cross-modal fusion calculation is skipped; if the fusion necessity weight is greater than or equal to the fusion threshold, cross-modal fusion is performed, and the fusion result is used as the output features of the encoder layer.

[0090] In some examples of this embodiment, the fusion module 303 is specifically configured to use multiple parallel temporal offsets to temporally align the audio features and video features of the current encoder layer and calculate cross-modal attention to obtain multiple scale attention outputs; generate fusion weights corresponding to each of the multiple scale attention outputs through a weight prediction network, and perform weighted summation of the multiple scale attention outputs based on their respective fusion weights to obtain multi-scale fusion features; and perform weighted fusion of the multi-scale fusion features with the audio features of the current encoder layer to obtain the output features of the encoder layer.

[0091] In some examples of this embodiment, the decoding module 304 is specifically configured to input the features output by the last encoder layer into the decoder module. The decoder module is built on the Transformer architecture and is used to decode the input features to generate speech recognition results. The decoder module decodes the features and converts them into speech recognition results in text form, which are then output as the speech recognition results of the target object.

[0092] It should be noted that other corresponding descriptions of the functional units involved in the audio and video fusion device provided in this embodiment can be found in [reference]. Figure 1 , Figure 2 The corresponding descriptions in [the document] will not be repeated here.

[0093] Based on the above, Figure 1 , Figure 2 The embodiment illustrates an audio and video fusion method. Correspondingly, this embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method. Figure 1 , Figure 2 This illustrates an audio and video fusion method.

[0094] Based on the above, Figure 1 , Figure 2 The embodiment illustrates an audio and video fusion method. Correspondingly, this embodiment also provides a computer program product on which a computer program is stored. When executed by a processor, this computer program implements the above-described method. Figure 1 , Figure 2 This illustrates an audio and video fusion method.

[0095] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.

[0096] Based on the above, Figure 1 , Figure 2 An audio and video fusion method is shown, and Figure 3 To achieve the above objectives, the present application also provides an electronic device, such as a personal computer or a server, in the illustrated virtual device embodiment. This device includes a storage medium and a processor; the storage medium stores a computer program; the processor executes the computer program to implement the above-described virtual device. Figure 1 , Figure 2 This illustrates an audio and video fusion method.

[0097] In some embodiments, the aforementioned physical device may further include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, an input unit such as a keyboard, etc., and optionally, a USB interface, a card reader interface, etc. In some embodiments, the network interface may include a standard wired interface, a wireless interface (such as a Wi-Fi interface), etc.

[0098] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0099] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0100] The above are merely specific embodiments of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. An audio and video fusion method, characterized in that, include: The audio and video streams of the target object are acquired, audio features and video features are extracted respectively, and the audio features and video features are analyzed to generate multimodal data. The multimodal data includes at least audio features, audio clarity score, noise type, noise features, video features, and video clarity score. The multimodal data is input into a pre-trained multimodal encoder, which includes at least one cascaded encoder layer; In each encoder layer of the multimodal encoder, the fusion threshold is dynamically adjusted according to the multimodal data, and cross-modal fusion is selectively performed based on the comparison result of the fusion threshold and the calculated fusion necessity weight to obtain the output features of the encoder layer. The features output by the last encoder layer in the multimodal encoder are decoded to obtain the speech recognition result of the target object.

2. The audio and video fusion method according to claim 1, characterized in that, The process of acquiring the audio and video streams of the target object and extracting audio and video features respectively includes: The audio stream is processed by frame segmentation, windowing, and fast Fourier transform to obtain the audio features; Face detection and key point localization are performed on the video stream to extract the region of interest for the lips, and lip motion features are extracted through a preset visual feature extraction network to obtain the video features; The audio features are input into a pre-trained regression network and a noise classification estimation network to obtain the audio clarity score, the noise type, and the noise features corresponding to the audio features; The video features are input into the regression network to obtain the video clarity score.

3. The audio and video fusion method according to claim 1, characterized in that, The step of dynamically adjusting the fusion threshold based on the multimodal data in each encoder layer of the multimodal encoder includes: The corresponding basic threshold is obtained from a preset threshold library according to the noise type, and the basic threshold is adjusted according to the noise intensity to obtain the fusion threshold. The noise intensity is determined according to the audio clarity score. The lower the audio clarity score, the higher the noise intensity.

4. The audio and video fusion method according to claim 1, characterized in that, Based on the comparison result between the fusion threshold and the calculated fusion necessity weight, cross-modal fusion is selectively performed to obtain the output features of the encoder layer, including: The fusion necessity weight is calculated based on the audio features, video features, and noise type of the current encoder layer. If the fusion necessity weight is less than the fusion threshold, then the audio features of the current encoder layer are used as the output features of the encoder layer, and the cross-modal fusion calculation is skipped. If the fusion necessity weight is greater than or equal to the fusion threshold, cross-modal fusion is performed, and the fusion result is used as the output feature of the encoder layer.

5. The audio and video fusion method according to claim 4, characterized in that, If the fusion necessity weight is greater than or equal to the fusion threshold, then cross-modal fusion is performed, and the fusion result is used as the output feature of the encoder layer, including: Using multiple parallel temporal offsets, the audio and video features of the current encoder layer are temporally aligned and cross-modal attention is calculated to obtain multiple scale attention outputs; The weight prediction network generates fusion weights corresponding to the attention outputs at multiple scales, and the attention outputs at multiple scales are weighted and summed based on the corresponding fusion weights to obtain multi-scale fusion features. The multi-scale fusion features are weighted and fused with the audio features of the current encoder layer to obtain the output features of the encoder layer.

6. The audio and video fusion method according to claim 1, characterized in that, Decoding the features output by the last encoder layer in the multimodal encoder to obtain the speech recognition result of the target object includes: The features output from the last encoder layer are input to the decoder module, which is built on the Transformer architecture and is used to decode the input features to generate speech recognition results. The features are decoded by the decoder module and converted into a text-based speech recognition result, which is then output as the speech recognition result of the target object.

7. An audio and video fusion device, characterized in that, include: The generation module is configured to acquire the audio stream and video stream of the target object, extract audio features and video features respectively, and analyze the audio features and video features to generate multimodal data. The multimodal data includes at least audio features, audio clarity score, noise type, noise features, video features, and video clarity score. An input module is configured to input the multimodal data into a pre-trained multimodal encoder, the multimodal encoder comprising at least one cascaded encoder layer; The fusion module is configured to dynamically adjust the fusion threshold based on the multimodal data in each encoder layer of the multimodal encoder, and selectively perform cross-modal fusion based on the comparison result of the fusion threshold and the calculated fusion necessity weight to obtain the output features of the encoder layer. The decoding module is configured to decode the features output by the last encoder layer in the multimodal encoder to obtain the speech recognition result of the target object.

8. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the audio and video fusion method according to any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the audio and video fusion method according to any one of claims 1-6.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the audio and video fusion method according to any one of claims 1-6.

Images