An audio-video speech recognition method based on quality-aware interleaved mark fusion

Abstract

The application belongs to the technical field of speech recognition, and specifically discloses an audio-video speech recognition method based on quality perception interlaced marking fusion. The method builds a layered decoupled audio-video speech recognition model, and a single-modality time sequence stabilization module is designed in the model to model the projected audio features and video features in a bidirectional state space. Meanwhile, the application also designs a local time sequence enhancement module to further enhance the stabilized audio features and video features through convolution. In addition, the application also designs a quality perception interlaced marking fusion module to generate a frame-by-frame fusion weight through the joint of content correlation weight and modality reliability score, and to construct an interlaced marking sequence with explicit frame-level structure constraint through an interlaced marking construction module, and then to perform bidirectional controlled information injection on the double-modality information in the same time frame through a frame alignment cross-gating residual refinement module. The application can realize high-precision audio-video speech recognition in a complex environment.

Description

Technical Field

[0001] This invention belongs to the field of speech recognition technology, specifically relating to an audio / video speech recognition method based on quality-perceived interleaved marker fusion. Background Technology

[0002] Audio-visual speech recognition methods utilize both audio and video lip-sync signals to jointly model and recognize speech content. Compared to automatic speech recognition methods that rely solely on audio input, audio-visual speech recognition can improve accuracy and robustness in complex environments such as strong background noise, acoustic signal degradation, echo interference, speaker accent differences, confusion of similar phonemes, and multi-person interference. It leverages supplementary evidence from the visual modality, including lip movements, facial dynamics, and pronunciation timing information. Existing audio-visual speech recognition methods mainly fall into the following categories: early fusion methods, which directly concatenate audio and video features before processing them in a unified coding network; late fusion methods, which model the audio and video streams separately and then weight and merge the results from both modalities at the output stage; fusion methods based on cross-modal attention, which explicitly model the correspondence between audio and video through an attention mechanism; and audio-visual speech recognition methods based on self-supervised pre-training, which first learn a general representation using a large amount of unlabeled audio and video data and then fine-tune it on labeled speech and text data.

[0003] In recent years, with the development of Transformer, Conformer, self-supervised representation learning, and audio-video joint modeling technology, the performance of audio-video speech recognition has been significantly improved. However, existing solutions still have the following shortcomings: (1) Most methods tightly couple single-modal temporal modeling and cross-modal fusion in the same network, causing unstable features formed in the early stage of audio and video modes to directly enter the fusion stage, which easily propagates and amplifies interference such as noise, occlusion, jitter, and local blur; (2) Audio streams and video streams usually have slight temporal misalignment in real acquisition environments. Existing methods often lack explicit frame-level structural constraints, so when the two modalities are not fully synchronized, cross-modal interaction is prone to failure or degradation; (3) Existing fusion methods usually adopt simple splicing, direct addition, or global cross-attention methods. The former two It is difficult to fully characterize the complex relationships across modalities. The latter has high computational complexity, especially under long sequence conditions, which will significantly increase the number of parameters, memory usage and deployment costs; (4) Existing methods usually adopt static weighting or weak adaptive weighting for contributions of different modalities. They cannot dynamically suppress low-quality modalities based on the content correlation and modal reliability in different time frames. They are not robust enough in noise enhancement, video occlusion and modal degradation; (5) Existing self-supervised pre-training schemes mostly emphasize the pre-training framework itself, but there is still a lack of sufficient design on how to introduce a fusion structure scheme that is suitable for data quality fluctuation scenarios and has fine representation capabilities in the pre-training and downstream recognition process. Summary of the Invention

[0004] The purpose of this invention is to propose an audio / video speech recognition method based on quality-aware interleaved label fusion, so as to realize audio / video speech recognition and improve the accuracy, robustness and computational efficiency of audio / video speech recognition.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: The audio / video speech recognition method based on quality-aware interleaved label fusion includes the following steps: Step 1. Acquire audio and video data, preprocess the audio and video data, and construct a training dataset; Step 2. Construct a layered and decoupled audio / video speech recognition model, which includes: The audio feature extraction module is used to perform frame segmentation, spectral transformation, Mel filter bank mapping, and shallow convolution processing on the audio sequence to obtain audio frame-level features. The video feature extraction module is used to perform shallow 3D convolution and 2D lightweight convolution on a continuous sequence of lip images to obtain video frame-level features. The unified projection module is used to project audio frame-level features and video frame-level features onto the same dimension through linear transformation. The single-modal temporal stabilization module is used to perform bidirectional state-space modeling on the projected audio and video features to generate stabilized audio and video features. The local temporal enhancement module is used to further enhance the stabilized audio and video features through convolution; The quality-aware interleaved tag fusion module is used to jointly generate frame-by-frame fusion weights by combining content relevance weights and modality reliability scores, and to construct interleaved tag sequences with explicit frame-level structural constraints through the interleaved tag construction module to achieve cross-modal joint modeling. Then, the frame-aligned cross-gated residual refinement module injects bidirectional controlled information into the dual-modal information within the same time frame. The post-fusion context refinement module is used to further integrate features through a temporal coding network; And a decoding output module; Step 3. Train the hierarchical decoupled audio-visual speech recognition model based on the training dataset from Step 1, and use the trained model to perform speech recognition and output the text recognition results.

[0006] The present invention has the following advantages: As described above, this invention proposes an audio / video speech recognition method based on quality-aware interleaved label fusion. It constructs a hierarchical decoupled audio / video speech recognition model, separating single-modal temporal stabilization, cross-modal fusion, and post-fusion context refinement into different stages. This avoids unstable single-modal features directly entering the fusion stage, significantly reducing the propagation risk of noise, occlusion, and local perturbations during the fusion stage. Furthermore, this invention designs a single-modal temporal stabilization module, utilizing bidirectional state-space temporal modeling to achieve long-range context modeling. Compared to globally attention-driven schemes, this approach has lower computational complexity and better long-sequence deployment efficiency. Meanwhile, the quality-aware interleaved labeling fusion module designed in this invention, through joint weighting of content relevance weights and modal reliability branches, can adaptively adjust modal contributions according to the quality changes of the two modalities in different time frames, improving the model's robustness in complex environments. This invention also enhances the structural constraints of cross-modal interactions and mitigates performance degradation caused by slight audio-visual misalignment by explicitly encoding cross-modal adjacency relationships within the same frame through interleaved labeling construction units. The frame-aligned cross-gated residual refinement module further injects lightweight bidirectional controlled information into the dual-modal information within the same time frame after joint cross-modal modeling, reducing computational complexity while enhancing modal complementarity and reducing the additional computational overhead caused by the high-complexity global cross-modal attention structure. This invention combines mask prediction learning in the pre-training stage with subsequent supervised fine-tuning, enabling the model to fully utilize unlabeled audio-visual data and improving recognition capabilities in low-resource and complex environments. Overall, this invention balances recognition accuracy, computational efficiency, and adaptability to multimodal quality fluctuations, making it suitable for various application scenarios such as noisy speech recognition, in-vehicle speech, human-computer interaction, smart terminals, conference transcription, and assisted accessibility recognition. Attached Figure Description

[0007] Figure 1 This is a flowchart of the audio / video speech recognition method based on quality-aware interleaved marker fusion in an embodiment of the present invention; Figure 2 This is a diagram of the layered and decoupled audio / video speech recognition model architecture in an embodiment of the present invention; Figure 3 This is a flowchart of the single-modal timing stabilization module in an embodiment of the present invention; Figure 4 This is a flowchart of the quality-perceived interlaced label fusion module in an embodiment of the present invention; Figure 5 This is a flowchart of the weight combination recalibration module in an embodiment of the present invention; Figure 6 This is a flowchart of the frame alignment cross-gated residual refinement module in an embodiment of the present invention; Figure 7 This is a flowchart of the pre-training, fine-tuning, and inference decoding process in an embodiment of the present invention. Detailed Implementation

[0008] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Example 1 This embodiment 1 describes an audio / video speech recognition method based on quality-aware interleaved tag fusion. By constructing a hierarchical decoupled architecture of "single-modal temporal stabilization - quality-aware interleaved tag fusion - post-fusion context refinement", the stability of audio and video modal features is improved before fusion. During the fusion stage, a dynamic weighting mechanism jointly driven by content relevance and modal reliability is introduced. Structured cross-modal interaction is achieved through interleaved tag structure, joint cross-modal modeling, and frame alignment cross-gated residual refinement. Thus, the accuracy, robustness, and computational efficiency of audio / video speech recognition are improved under conditions of noise, occlusion, slight alignment errors, and dynamic changes in modal quality.

[0009] like Figure 1 As shown, the audio / video speech recognition method based on quality-aware interleaved marker fusion includes the following steps: Step 1. Obtain audio and video data, preprocess the audio and video data, and construct a training dataset.

[0010] Acquire audio and video speech samples, where each sample includes an audio sequence, a video sequence corresponding to the audio sequence time, and corresponding text annotations. The audio sequence is the original speech waveform signal, and the video sequence is a continuous sequence of video frames containing the speaker's facial or lip regions.

[0011] When preprocessing the video sequence, face detection is first performed on the original video frames to determine the speaker's facial region; then the lip region is located based on facial key points and cropped; subsequently, the cropped lip images are scaled to a uniform size, and the video sequence is sampled or resampled according to a set frame rate to make the video frame rate uniform at 25 frames / second, resulting in a continuous lip image sequence.

[0012] When preprocessing the audio sequence, the original audio signal is resampled to 16 kHz, and pre-emphasis, silence removal, or endpoint detection are performed as needed to reduce interference from invalid silence segments and non-speech segments.

[0013] Subsequently, time alignment is performed based on the timestamps or frame numbers of the audio and video sequences to ensure temporal correspondence. Finally, the audio and video sequences, along with their corresponding text annotations, after the aforementioned basic preprocessing, are combined to form the training dataset.

[0014] The preprocessing in step 1 is mainly used to unify the sampling rate, frame rate, spatial cropping, silence processing, and time alignment of the original audio and video data; the subsequent audio feature extraction module and video feature extraction module are used to further extract audio frame-level feature sequences and video frame-level feature sequences within the model, and the two have different processing purposes.

[0015] Step 2. Build a layered and decoupled audio / video speech recognition model, such as... Figure 2 As shown, the model includes an audio feature extraction module, a video feature extraction module, a unified projection module, a single-modal temporal stabilization module, a local temporal enhancement module, a quality-aware interleaved label fusion module (QIT), a post-fusion context refinement module, and a decoding output module.

[0016] The audio feature extraction module is used to perform frame segmentation, spectral transformation, Mel filter bank mapping, and shallow convolution processing on the audio sequence to obtain audio frame-level features. The video feature extraction module is used to perform shallow 3D convolution and 2D lightweight convolution on a continuous sequence of lip images to obtain video frame-level features. The unified projection module is used to project audio frame-level features and video frame-level features onto the same dimension through linear transformation. The single-modal temporal stabilization module is used to perform bidirectional state-space modeling on the projected audio and video features to generate stabilized audio and video features. The local temporal enhancement module is used to further enhance the stabilized audio and video features through convolution; The quality-aware interleaved tag fusion module is used to jointly generate frame-by-frame fusion weights by combining content relevance weights and modality reliability scores, and to construct interleaved tag sequences with explicit frame-level structural constraints through the interleaved tag construction module to achieve cross-modal joint modeling. Then, the frame-aligned cross-gated residual refinement module injects bidirectional controlled information into the dual-modal information within the same time frame. The post-fusion context refinement module is used to further integrate features through a temporal coding network.

[0017] The layered decoupled audio / video speech recognition model adopts a layered decoupled processing flow of "single-modal temporal stabilization—quality-aware interleaved label fusion—post-fusion context refinement". Specifically, it first performs temporal stabilization and local enhancement on each modality separately, then performs structured cross-modal fusion, and finally refines the context of the fused features. This processing flow differs from schemes that simply stack all modeling processes in the same stage, and can more effectively suppress the propagation of single-modal noise, improving the stability of subsequent fusion inputs.

[0018] Step 2.1. Input the audio sequence and video sequence into the audio feature extraction module and video feature extraction module respectively to obtain the audio frame-level feature sequence A and the video frame-level feature sequence V.

[0019] The audio feature extraction module includes an audio preprocessing unit, a spectral feature extraction unit, a shallow convolutional front-end, and a linear mapping front-end.

[0020] The audio preprocessing unit performs frame segmentation and windowing on the received audio sequence. The spectral feature extraction unit performs a short-time Fourier transform on the framed audio signal to obtain spectral features, and further obtains log-Mel spectral features through Mel filter bank mapping and logarithmic transform. The shallow convolutional front-end consists of a one-dimensional or two-dimensional convolutional layer, a normalization layer, and a nonlinear activation layer, and is used to extract local time-frequency patterns from the log-Mel spectral features. The linear mapping front-end maps the acoustic features output by the shallow convolutional front-end to a set audio feature dimension, thereby obtaining the audio frame-level feature sequence A.

[0021] The video feature extraction module includes a shallow 3D convolutional front-end and a 2D lightweight convolutional network.

[0022] The shallow 3D convolutional front-end consists of 3D convolutional layers, normalization layers, and nonlinear activation layers. It is used to simultaneously perform convolutional processing on continuous lip image sequences in both temporal and spatial dimensions to extract short-term lip movement information between adjacent video frames. The 2D lightweight convolutional network consists of depthwise separable convolutional layers or lightweight 2D convolutional layers. It is used to further extract spatial features from the frame-by-frame features output by the shallow 3D convolutional front-end. Specifically, it performs convolutional extraction on lip edges, lip contours, texture changes, and local spatial structure information in each time frame, and obtains the video frame-level feature sequence V through global pooling or flattening mapping.

[0023] The size of the audio frame-level feature sequence A is T×Da, and the size of the video frame-level feature sequence V is T×Dv; T is the number of time frames, Da is the audio feature dimension, and Dv is the video feature dimension.

[0024] Step 2.2. Input the audio frame-level feature sequence A and the video frame-level feature sequence V into the unified projection module for processing to obtain the projected audio feature sequence A1 and the projected video feature sequence V1.

[0025] The unified projection module includes an audio linear projection layer, a video linear projection layer, and a layer normalization layer.

[0026] The audio frame-level feature sequence A is mapped to the hidden dimension d through an audio linear projection layer, and then subjected to layer normalization to obtain the projected audio feature sequence A1.

[0027] The video frame-level feature sequence V is mapped to the hidden dimension d through a video linear projection layer, and then subjected to layer normalization to obtain the projected video feature sequence V1.

[0028] Both A1 and V1 have dimensions of T×d.

[0029] This step ensures that the features of the two modalities have a unified representation dimension and similar feature scale before subsequent cross-modal interactions.

[0030] Step 2.3. Input the projected audio feature sequence A1 and the projected video feature sequence V1 into the corresponding single-mode temporal stabilization module to obtain the stabilized audio feature sequence A2 and the stabilized video feature sequence V2.

[0031] like Figure 3 As shown, the single-modal time-series stabilization module includes a layer normalization unit, a forward state space modeling unit, a backward state space modeling unit, a bidirectional fusion unit, a residual connection unit, and a random deactivation unit.

[0032] For any modal input feature sequence X, the processing flow of the single-modal time series stabilization module is as follows: First, the input feature sequence X is normalized in the layer normalization unit to obtain the normalized feature sequence LN(X).

[0033] Then, LN(X) is input into the forward state space modeling unit for processing to obtain the forward output F(X); at the same time, LN(X) is input into the backward state space modeling unit in the reverse time dimension for processing to obtain the backward output B(X).

[0034] The forward state space modeling unit includes an input projection layer, a gating unit, and an output projection layer. Its processing flow is as follows: For the normalized feature sequence LN(X), the input projection layer first maps it to the intermediate features required for state-space modeling. Using state-space sequence modeling, the hidden state is recursively updated from frame 1 to frame T in chronological order to model the long-range dependency between the current frame and historical frames. The gating unit adopts a gating structure composed of a linear layer and a sigmoid activation function. It generates modulation weights with values ​​ranging from 0 to 1 based on the features of the current frame and uses the modulation weights to selectively enhance or suppress the state update result. The output projection layer maps the gated and modulated state update result back to the same hidden dimension d as the input feature sequence X to obtain the forward output F(X), thereby ensuring that F(X) can be added to the original input feature sequence X as a residual.

[0035] The backward state space modeling unit has the same structure as the forward state space modeling unit, including an input projection layer, a gating unit, and an output projection layer. The backward state space modeling unit first reverses the temporal order of LN(X) to obtain a reverse normalized feature sequence. Then, it maps the reverse normalized feature sequence to intermediate features required for state space modeling through the input projection layer. The hidden state is then recursively updated from frame T to frame 1 in reverse temporal order to model the long-range dependency between the current frame and future frames. The gating unit generates modulation weights based on the features of the current reverse time step, selectively enhancing or suppressing the state update result. The output projection layer maps the gated and modulated state update result back to the hidden dimension d. Finally, the output obtained from the reverse modeling is restored in its original temporal order to obtain the backward output B(X).

[0036] The forward output F(X) and the backward output B(X) are fed into the bidirectional fusion unit for fusion to obtain the bidirectional fusion result U(X). The specific fusion formula is as follows: U(X) = [F(X) + B(X)] / 2.

[0037] Subsequently, U(X) is added to the original input feature sequence X through a residual connection unit to obtain the residual fusion feature R(X), which is calculated using the following formula: R(X) = X + U(X).

[0038] Finally, the residual fusion feature R(X) is fed into a random deactivation unit for processing to obtain the stabilized feature sequence X1.

[0039] During the model training phase, the random deactivation unit randomly sets some feature elements to zero according to a preset probability p, and performs scale compensation on the retained features to reduce the model's overfitting to local features; during the inference phase, the random deactivation unit no longer randomly sets the elements to zero, but directly outputs the features after residual connection.

[0040] When X is the projected audio feature sequence A1, the output of the single-mode temporal stabilization module is the stabilized audio feature sequence A2.

[0041] When X is the projected video feature sequence V1, the output of the single-modal temporal stabilization module is the stabilized video feature sequence V2.

[0042] The single-modal temporal stabilization module, which combines "long-range bidirectional state-space modeling with short-term local temporal enhancement," enables both audio and video modalities to simultaneously acquire long-range contextual supplementation and suppress local perturbations before fusion. The bidirectional state-space design within the single-modal temporal stabilization module can inject long-range contextual information with low complexity while simultaneously utilizing past and future temporal dependencies, making it suitable for bidirectional representation modeling tasks in audio, video, and speech recognition.

[0043] This invention solves the problem of overly tight coupling between single-modal temporal modeling and cross-modal fusion in existing solutions, and avoids noise propagation caused by unstable audio and video representations entering the fusion stage too early.

[0044] Step 2.4. Input the stabilized audio feature sequence A2 and the stabilized video feature sequence V2 into the local temporal enhancement module to obtain the enhanced audio feature sequence A3 and the enhanced video feature sequence V3.

[0045] The local temporal enhancement module includes a one-dimensional convolutional layer, a layer normalization layer, a nonlinear activation layer, and residual connection units.

[0046] For any modality-stabilized feature sequence X1, the processing flow of the local temporal enhancement module is as follows: First, X1 is convolved along the time dimension to obtain the local context enhancement feature C(X1).

[0047] Then, C(X1) is subjected to layer normalization and nonlinear activation to obtain the enhanced feature E(X1).

[0048] Finally, E(X1) and X1 are added together by residuals to obtain the feature X2 after local temporal enhancement, which is calculated by the following formula: X2 = X1 + E(X1).

[0049] When X1 is the stabilized audio feature sequence A2, the output of the local temporal enhancement module is the enhanced audio feature sequence A3.

[0050] When X1 is the stabilized video feature sequence V2, the output of the local temporal enhancement module is the enhanced video feature sequence V3.

[0051] This step is used to supplement short-term contextual consistency, enhancing the model's adaptability to transient disturbances, local ambiguity, and slight temporal misalignment.

[0052] Step 2.5. Input the enhanced audio feature sequence A3 and the enhanced video feature sequence V3 into the QIT module for processing to obtain the fused feature sequence F.

[0053] like Figure 4 As shown, the QIT module includes a modality-aware router, a quality gating module, a weight combination recalibration module, an interleaving marker construction module, a joint transformer coding module, a deinterleaving recovery module, and a frame alignment cross-gating residual refinement module.

[0054] Step 2.5.1. Input the enhanced audio feature sequence A3 and the enhanced video feature sequence V3 into the modality-aware router for processing to obtain the content relevance weight sequence ra of the audio modality and the content relevance weight sequence rv of the video modality.

[0055] Specifically, for each time frame, output the audio content relevance weight ra(t) and the video content relevance weight rv(t); arrange the weights of all time frames in chronological order to obtain the audio content relevance weight sequence ra and the video content relevance weight sequence rv.

[0056] ra = [ra(1) ,…,ra(t),…, ra(T)]; rv = [rv(1) ,…,rv(t),…, rv(T)]; Where ra(t) and rv(t) represent the audio content relevance weight and video content relevance weight of the t-th time frame, respectively.

[0057] A modality-aware router is used to predict the priority of two modalities in a given time frame based on the content relevance of audio and video features. The modality-aware router consists of a linear layer, a layer normalization layer, and a non-linear activation layer; its specific process is as follows: At each time frame t, the enhanced audio feature A3(t) and the enhanced video feature V3(t) are concatenated to obtain the concatenated feature C1(t).

[0058] The splicing feature C1(t) is processed sequentially through a linear layer, a normalization layer, a nonlinear activation layer, and a linear layer to obtain a two-dimensional routing value; the two-dimensional routing values ​​correspond to the audio modality and the video modality, respectively.

[0059] Then, input the two-dimensional routing values ​​into the normalization function to obtain the content relevance weight ra(t) of the audio modality and the content relevance weight rv(t) of the video modality, which satisfy: ra(t) + rv(t) = 1.

[0060] A temperature coefficient is introduced to adjust the sensitivity of different modal weight allocations.

[0061] Let the two-dimensional routing values ​​output by the second linear layer be la(t) and lv(t), where la(t) corresponds to the audio modality and lv(t) corresponds to the video modality. The content relevance weights obtained through Softmax normalization are as follows: ra(t) = exp(la(t) / τr) / [exp(la(t) / τr) + exp(lv(t) / τr)]; rv(t) = exp(lv(t) / τr) / [exp(la(t) / τr) + exp(lv(t) / τr)]; Where τr is the routing temperature coefficient; when τr is large, the weight distribution is smoother; when τr is small, the weight distribution is more biased towards the mode corresponding to the larger routing value.

[0062] Step 2.5.2. Input the enhanced audio feature sequence A3 and the enhanced video feature sequence V3 into the quality gating module for processing to obtain the audio reliability score sequence qa and the video reliability score sequence qv.

[0063] qa=[qa(1),…,qa(t),…,qa(T)], qv=[qv(1),…,qv(t),…,qv(T)]; qa(t) and qv(t) represent the audio reliability score and video reliability score of the t-th time frame, respectively.

[0064] The quality gating module is used to estimate the reliability of the audio and video modalities in the current time frame.

[0065] The quality gating module includes an audio quality gating branch and a video quality gating branch. Both branches have the same structure, comprising a linear layer, a layer normalization layer, a non-linear activation layer, and a Sigmoid activation function. For audio feature A3(t), it is first mapped to an intermediate dimension via a linear layer, then processed by a layer normalization layer and a non-linear activation layer, followed by an audio reliability gating value la_q(t) output by a linear layer, and finally, the audio reliability score qa(t) is obtained by the Sigmoid activation function. For video feature V3(t), the same processing method is used to obtain the video reliability score qv(t). The formulas for calculating the audio reliability score qa(t) and the video reliability score qv(t) are as follows: qa(t) = Sigmoid(la_q(t) / τq); qv(t) = Sigmoid(lv_q(t) / τq); Where lv_q(t) represents the video reliability gating value; τq is the quality gating temperature coefficient, used to adjust the smoothness of the reliability score; when τq is large, the reliability score changes more smoothly; when τq is small, the reliability score is closer to 0 or 1.

[0066] To improve training stability, a gradient decoupling mechanism can be configured between the quality gating results and the backbone encoding process.

[0067] The gradient decoupling mechanism refers to performing stopping gradient processing on the audio features A3(t) and video features V3(t) of the input quality gating module during the training phase. This allows them to participate in reliability score calculation during forward propagation, but blocks the gradient of the quality gating branch from being transmitted back to the preceding backbone encoding module during back propagation. Stopping gradient processing is a commonly used gradient control method in neural network training to reduce excessive perturbation of the backbone feature representation by auxiliary branches. Through this method, the quality gating module can estimate modal reliability based on the current features, while reducing the impact of the quality estimation process on the backbone encoding part, thereby improving training stability.

[0068] Step 2.5.3. Combine the content relevance weights output by the modality-aware router with the reliability scores output by the quality gating module to obtain a fused weight sequence. Then, use the fused weight sequence to recalibrate the enhanced audio feature sequence A3 and the enhanced video feature sequence V3 frame by frame to obtain the recalibrated audio feature sequence A4 and the recalibrated video feature sequence V4.

[0069] like Figure 5 As shown, the weight combination and recalibration module includes a weight combination unit, a weight normalization unit, and a frame-by-frame recalibration unit. For any time frame t, the processing flow of the weight combination and recalibration module is as follows: First, the weight combination unit combines the audio content relevance weight ra(t) output by the modality-aware router with the audio reliability score qa(t) output by the quality gating module to obtain the unnormalized audio weight ua(t), which is calculated using the following formula: ua(t) = ra(t) × qa(t); Simultaneously, the video content relevance weight rv(t) output by the modality-aware router is combined with the video reliability score qv(t) output by the quality gating module to obtain the unnormalized video weight uv(t), the calculation formula of which is: uv(t) = rv(t) × qv(t); Then, ua(t) and uv(t) are fed into the weight normalization unit for normalization processing to obtain the final audio fusion weight wa(t) and video fusion weight wv(t), the calculation formula of which is: wa(t) = ua(t) / [ua(t) + uv(t) + ε]; wv(t) = uv(t) / [ua(t) + uv(t) + ε]; Where ε is a minimal constant to prevent the denominator from being zero.

[0070] Finally, the frame-by-frame recalibration unit uses wa(t) and wv(t) to recalibrate the audio feature A3(t) and video feature V3(t) frame by frame, obtaining the recalibrated audio feature A4(t) and the recalibrated video feature V4(t). The calculation formula is as follows: A4(t) = wa(t) × A3(t); V4(t) = wv(t) × V3(t).

[0071] Frame-by-frame recalibration refers to multiplicatively modulating the audio and video features of corresponding time frames using frame-by-frame fusion weights, so that high-reliability or high-correlation modal features are preserved or enhanced, while low-reliability or low-correlation modal features are suppressed.

[0072] This step suppresses low-quality or low-relevance modalities before they enter the subsequent interleaving marker construction and joint modeling, thereby reducing the risk of noise, occlusion, or modal degradation information propagating during cross-modal fusion. This invention thus enables the simultaneous consideration of content relevance and modal reliability, and dynamically adjusts the contributions of audio and video modalities based on modal quality changes in each time frame.

[0073] Step 2.5.4. The interleaving marker construction module constructs an interleaving marker sequence Z from the recalibrated audio feature sequence A4 and the recalibrated video feature sequence V4 according to their time alignment relationship.

[0074] At the t-th time frame, V4(t) and A4(t) are arranged adjacently to obtain the interleaved label sequence Z = [V4(1), A4(1), V4(2), A4(2),..., V4(t), A4(t),..., V4(T), A4(T)]; where V4(t) represents the video feature of the t-th time frame in the recalibrated video feature sequence V4, and A4(t) represents the audio feature of the t-th time frame in the recalibrated audio feature sequence A4, t=1,2,...,T.

[0075] This interleaving method places the two modal features of the same time frame directly in adjacent positions, explicitly encoding the cross-modal adjacency relationship within the same frame, thus providing clear frame-level structural constraints for the subsequent joint modeling stage.

[0076] This invention solves the problem of existing fusion methods lacking explicit frame-level structural constraints, and improves the system's robustness to slight temporal misalignment between audio and video streams.

[0077] Step 2.5.5. Input the interleaved marker sequence Z into the joint transformer encoding module for processing to obtain the joint context output sequence J.

[0078] The joint converter coding module includes a position coding unit, a multi-head self-attention unit, a feedforward network unit, a residual connection unit, and a layer normalization unit.

[0079] The position coding unit is used to add time position information to the interleaved marker sequence Z to obtain the position-coded sequence Zp, which enables the model to distinguish the positional relationship between different time frames and different markers in the sequence.

[0080] The multi-head self-attention unit adopts the conventional multi-head self-attention processing method in transformer networks. This unit first linearly maps the position-encoded sequence Zp into query vector, key vector, and value vector respectively. Then, it calculates the correlation between different time frames and different modal labels in multiple attention heads respectively, and concatenates and linearly maps the outputs of each attention head to obtain the self-attention output features.

[0081] The residual connection unit is used to add the input and output of the multi-head self-attention unit to preserve the original feature information and stabilize feature propagation; the layer normalization unit is used to normalize the features after the residuals are added to maintain the stability of the feature scale.

[0082] The feedforward network unit consists of a linear layer, a nonlinear activation layer, and another linear layer connected sequentially. It performs a position-wise nonlinear transformation on the features at each location to enhance feature representation. After processing by the feedforward network unit, the sequence passes through a residual connection unit and a layer normalization unit to obtain the joint context output sequence J.

[0083] Since the input sequence Z is organized in such a way that video and audio tags are adjacent within the same time frame, the joint transformer coding module can learn cross-temporal and cross-modal dependencies simultaneously when performing sequence modeling, and complete the structured interaction between audio and video modalities under explicit frame-level alignment constraints.

[0084] Step 2.5.6. The deinterlacing recovery module re-splits the joint context output sequence J according to the original arrangement before interlacing, to obtain the audio output feature sequence Ja and the video output feature sequence Jv.

[0085] Ja=[Ja(1) ,…,Ja(t) ,…, Ja(T)], Jv=[Jv(1) ,…,Jv(t) ,…, Jv(T)]; Where Ja(t) and Jv(t) represent the audio output features and video output features of the t-th time frame, respectively.

[0086] The deinterleaving recovery module restores the original dual-path structure according to the odd / even position or fixed index rules.

[0087] After this step, the fused audio branch features and video branch features can be obtained separately, so as to carry out the next step of frame-by-frame refinement interaction.

[0088] Step 2.5.7. The frame-aligned cross-gated residual refinement module performs bidirectional controlled information injection and residual refinement processing on the audio output feature sequence Ja and the video output feature sequence Jv within the same time frame to obtain the refined audio feature sequence A5 and the refined video feature sequence V5.

[0089] The frame-aligned cross-gated residual refinement module performs lightweight bidirectional controlled information injection on audio and video output feature sequences within the same time frame without introducing highly complex global cross-modal attention. Bidirectional controlled information injection refers to performing information injection in both the video-to-audio and audio-to-video directions within the same time frame, controlling the injection strength of the cross-modal messages using corresponding gating coefficients. Specifically, bidirectional controlled information injection means: on the one hand, mapping video output features to messages injected into the audio branch, controlling the injection strength using audio-side gating coefficients; on the other hand, mapping audio output features to messages injected into the video branch, controlling the injection strength using video-side gating coefficients.

[0090] like Figure 6 As shown, the frame-aligned cross-gated residual refinement module includes an audio-to-video projection subunit, a video-to-audio projection subunit, an audio-side gating subunit, a video-side gating subunit, a video branch injection control unit, an audio branch injection control unit, a video residual addition and normalization unit, and an audio residual addition and normalization unit. For any time frame t, the processing flow of the frame-aligned cross-gated residual refinement module is as follows: First, the video output feature Jv(t) is input into the video-to-audio projection subunit for processing, resulting in the message vector Mva(t) injected into the audio branch. The video-to-audio projection subunit includes a first transmodal linear layer, a nonlinear activation layer, and a second transmodal linear layer, used to map the video output features to the same feature space as the audio output features. Specifically, the first transmodal linear layer first maps the video output feature Jv(t) to an intermediate feature space, the nonlinear activation layer performs a nonlinear transformation on the intermediate features, and the second transmodal linear layer then maps it to the same feature dimension as the audio output feature Ja(t), thus obtaining the video-to-audio message vector Mva(t).

[0091] The audio output feature Ja(t) is input into the audio-to-video projection subunit for processing, resulting in the message vector Mav(t) injected into the video branch. The audio-to-video projection subunit comprises a third cross-modal linear layer, a non-linear activation layer, and a fourth cross-modal linear layer, used to map the audio output features to the same feature space as the video output features. Specifically, the third cross-modal linear layer first maps the audio output feature Ja(t) to an intermediate feature space, the non-linear activation layer performs a non-linear transformation on the intermediate features, and the fourth cross-modal linear layer then maps it to the same feature dimension as the video output feature Jv(t), thus obtaining the audio-to-video message vector Mav(t).

[0092] Then, Ja(t) and Jv(t) are concatenated along their feature dimensions to obtain the concatenated feature [Ja(t), Jv(t)]. This concatenated feature is then input into the audio-side gating subunit. After processing by a multilayer perceptron and a sigmoid activation function, the audio-side gating coefficient ga(t) is obtained. The audio-side gating coefficient ga(t) is used to control the strength of the video message vector Mva(t) injected into the audio branch.

[0093] Simultaneously, Jv(t) and Ja(t) are concatenated along their feature dimensions to obtain the concatenated feature [Jv(t), Ja(t)]. This concatenated feature is then input into the video-side gating subunit, and after processing by a multilayer perceptron and a sigmoid activation function, the video-side gating coefficient gv(t) is obtained. The video-side gating coefficient gv(t) is used to control the intensity of the audio message vector Mav(t) injected into the video branch.

[0094] Subsequently, the video branch injection control unit controls the strength of the audio message vector Mav(t) injected into the video branch according to the video-side gating coefficient gv(t) and the scaling factor s; the audio branch injection control unit controls the strength of the video message vector Mva(t) injected into the audio branch according to the audio-side gating coefficient ga(t) and the scaling factor s. Subsequently, the video residual summation and normalization unit performs residual summation and layer normalization on the controlled injected audio message and the video output feature Jv(t) to obtain the refined video feature V5(t); the audio residual summation and normalization unit performs residual summation and layer normalization on the controlled injected video message and the audio output feature Ja(t) to obtain the refined audio feature A5(t).

[0095] The scaling factor s is used to adjust the overall influence of cross-modal injection information in residual updates; the audio-side gating coefficient ga(t) is used to control the injection intensity in the video-to-audio direction, and the video-side gating coefficient gv(t) is used to control the injection intensity in the audio-to-video direction.

[0096] Subsequently, the cross-modal message injection intensity is controlled by the gating coefficient and scaling factor s, and after residual summation and layer normalization, the refined audio feature A5(t) and the refined video feature V5(t) are obtained, and their calculation formulas are as follows: A5(t) = LN[Ja(t) + s × ga(t) × Mva(t)]; V5(t) = LN[Jv(t) + s × gv(t) × Mav(t)]; Where LN[ ] represents the layer normalization operation, used to normalize the feature vector within the brackets along the feature dimension. s represents the scaling factor, used to adjust the overall strength of cross-modal message injection. ga(t) controls the strength of the video message vector Mva(t) injected into the audio branch; gv(t) controls the strength of the audio message vector Mav(t) injected into the video branch.

[0097] The audio reliability score qa(t) and video reliability score qv(t) have been used in the weight combination recalibration module to generate frame-by-frame fusion weights, and the enhanced audio and video features have been quality modulated. Therefore, in the frame alignment cross-gating residual refinement stage, qa(t) and qv(t) are no longer introduced repeatedly. Instead, the cross-modal message injection strength is controlled by the audio-side gating coefficient ga(t), the video-side gating coefficient gv(t), and the scaling factor s.

[0098] The above process enables controlled injection of video information into the audio branch and controlled injection of audio information into the video branch within the same time frame.

[0099] Bidirectional controlled information injection within the same time frame refers to the bidirectional information exchange within the t-th time frame, utilizing only the audio output feature Ja(t) and video output feature Jv(t) corresponding to that time frame, without performing cross-frame injection on features from other time frames. Specifically, video-to-audio information injection involves injecting the video message vector Mva(t) into the audio branch after modulation by the audio-side gating coefficient ga(t) and scaling factor s; audio-to-video information injection involves injecting the audio message vector Mav(t) into the video branch after modulation by the video-side gating coefficient gv(t) and scaling factor s. Because the injection strength is controlled by the gating coefficient and scaling factor, it is called controlled information injection.

[0100] Through the above processing, the frame-aligned cross-gated residual refinement module can perform lightweight, controlled bidirectional supplementation of bimodal information within the same time frame after joint cross-modal modeling, thereby enhancing the complementarity between audio and video modalities. Furthermore, since this module only performs cross-modal injection within the same time frame and does not require constructing a global cross-modal attention matrix, it has lower computational complexity compared to a global cross-modal attention structure and better aligns with the fusion objective of audio and video within the same frame.

[0101] The frame-aligned cross-gated residual refinement module further injects lightweight, controlled bidirectional information into the dual-modal information within the same time frame after joint cross-modal modeling, thereby enhancing modal complementarity while reducing computational complexity.

[0102] This step restricts cross-modal interactions to occur within the same time frame, resulting in lower computational complexity compared to a global cross-modal attention structure, and better aligns with the fusion objective of audio and video within the same frame.

[0103] This invention solves the problems of high computational complexity and high engineering deployment cost of existing global cross-modal interaction structures, while reducing computational overhead while ensuring fusion effect.

[0104] Step 2.5.8. Concatenate the refined audio feature sequence A5 and the refined video feature sequence V5 according to the feature dimension to obtain the fused feature sequence F output by the QIT module, so as to retain the complementary information in the refined audio features and refined video features.

[0105] The QIT module generates frame-by-frame fusion weights by jointly generating content-related routing and modal reliability estimation, and implements cross-modal joint modeling with explicit frame-level structural constraints by combining interleaved labeling. At the same time, the frame-aligned cross-gated residual refinement module injects bidirectional controlled information into audio and video features within the same time frame, thereby enhancing the complementary expressive ability between the two modalities.

[0106] Step 2.6. Input the fused feature sequence F into the fusion context refinement module to obtain the final fused feature sequence Y.

[0107] Among them, the fused feature sequence F is the frame-level fused feature sequence output by the QIT module, and the final fused feature sequence Y is the frame-level context feature sequence used for subsequent text decoding.

[0108] The post-fusion context refinement module uses a convolutionally enhanced temporal encoder to process the fused feature sequence F output by the quality-aware interleaved label fusion module. The convolutionally enhanced temporal encoder includes an attention layer, a temporal convolutional layer, a feedforward network layer, a residual connection layer, and a layer normalization layer. Specifically, the attention layer uses multi-head self-attention to model the dependencies between different time frames in the fused feature sequence F to obtain long-range semantic associations; the temporal convolutional layer uses one-dimensional temporal convolution to extract local temporal changes between adjacent time frames along the temporal dimension; the feedforward network layer consists of a linear layer, a non-linear activation layer, and another linear layer connected sequentially, used to perform non-linear transformations on the features of each time frame to enhance feature expressiveness; the residual connection layer adds the current layer input to the current layer output to preserve original information and stabilize feature propagation; the layer normalization layer normalizes the features after residual addition to maintain feature scale stability. After the above processing, the final fused feature sequence Y is obtained.

[0109] Since the QIT module has completed cross-modal interaction under explicit alignment constraints, the post-fusion context refinement module no longer undertakes the main alignment and fusion tasks, but is used to further integrate long-range semantic dependencies, multi-scale temporal dynamics, and contextual consistency after fusion.

[0110] The aforementioned structured fusion mechanism is embedded into the mask prediction-supervised pre-training-supervised fine-tuning-decoding inference process, enabling the model to learn robust multimodal representations for modal quality fluctuations and minor alignment errors during the pre-training stage. The structured fusion mechanism includes content relevance weight calculation, modal reliability estimation, frame-by-frame recalibration, interleaving marker construction, joint transformer encoding, deinterleaving recovery, and frame alignment cross-gated residual refinement.

[0111] The decoding output module employs a transformer decoder. The transformer decoder is a commonly used sequence decoding structure in the field of speech recognition, used to generate text recognition results based on the final fused feature sequence Y. During the training phase, the decoding output module calculates the cross-entropy loss based on the predicted text and the ground truth text annotations, and can combine it with the connection-temporal classification loss to jointly optimize the model. During the inference phase, the decoding output module progressively generates text sequences based on the final fused feature sequence Y, and uses a beam search strategy to output the final recognition result.

[0112] Step 3. Train the hierarchical decoupled audio-visual speech recognition model based on the training dataset from Step 1, and use the trained model to perform speech recognition and output the text recognition results.

[0113] like Figure 7 As shown, the pre-training, fine-tuning, and inference decoding process of this invention is as follows: Step 3.1. Perform self-supervised pre-training and select a self-supervised pre-training method based on mask prediction.

[0114] In the pre-training phase, raw audio and video data that is unlabeled or only has audio and video samples without text annotations are first collected, and the raw audio and video data are preprocessed in accordance with the method described in step 1 to obtain time-aligned audio sequences and video sequences.

[0115] Then, frame-level discrete pseudo-label sequences are constructed from the preprocessed audio and video sequences. Specifically, clustering can be used to generate discrete pseudo-labels corresponding to time frames from the audio representation, video representation, or audio-video joint representation, so that each predicted position has a corresponding training target.

[0116] Subsequently, a masking process is applied to the preprocessed audio and video sequences to mask audio information, video information, or both at certain time points.

[0117] The masked audio and video sequences are input into a layered decoupled audio-video-speech recognition model for encoding. The sequences are then processed sequentially through an audio feature extraction module, a video feature extraction module, a unified projection module, a single-modal temporal stabilization module, a local temporal enhancement module, a QIT module, and a post-fusion context refinement module to obtain the output features.

[0118] Finally, the output features are input into the classification prediction head to predict the frame-level discrete pseudo-labels corresponding to the occluded positions, and the model parameters are updated using the cross-entropy loss between the prediction results and the frame-level discrete pseudo-labels to obtain the pre-trained model parameters.

[0119] In one implementation, the pre-training stage can progressively optimize the quality of frame-level discrete pseudo-labels through multiple rounds of clustering-pre-training iterations. Through this pre-training method, the model can learn the collaborative relationships, alignment relationships, and robust representation capabilities between audio and video on a large amount of unlabeled audio and video corpus, enabling the audio and video representations obtained in the pre-training stage to better serve subsequent speech recognition tasks in scenarios with fluctuating quality.

[0120] Step 3.2. After completing the self-supervised pre-training, perform supervised fine-tuning.

[0121] In the fine-tuning stage, audio and video training samples with text annotations are first obtained, and data preprocessing is performed in accordance with the method described in step 1 to obtain audio sequences and video sequences corresponding to the text annotations.

[0122] Then, the model parameters obtained in the pre-training phase are loaded, and the classification prediction head used in the pre-training phase is replaced with the text recognition decoding module.

[0123] Subsequently, the preprocessed audio and video training samples are input into the hierarchical decoupled audio and video speech recognition model for encoding. The samples are then processed sequentially through the audio feature extraction module, video feature extraction module, unified projection module, single-modal temporal stabilization module, local temporal enhancement module, QIT module, and post-fusion context refinement module to obtain the final fused feature sequence Y.

[0124] The final fused feature sequence Y is input into the text recognition decoding output module, which outputs the target character sequence, sub-word sequence, or word sequence. The supervised loss is calculated based on the prediction results and the ground truth text annotations, and the network parameters are updated through backpropagation. The supervised loss can be a weighted sum of the connection-time classification loss and the cross-entropy loss.

[0125] Through supervised fine-tuning, the model can further adapt the robust audio and video representations learned in the pre-training stage to specific text recognition tasks, thereby obtaining a fully trained model.

[0126] Step 3.3. Decoding and reasoning.

[0127] In the inference and decoding stage, the trained model obtained in the fine-tuning stage is first loaded; then the audio and video data to be identified, namely the audio sequence and video sequence, are input, and data preprocessing is performed in accordance with the method described in step 1.

[0128] Subsequently, the preprocessed audio and video data to be recognized is input into the hierarchical decoupled audio and video speech recognition model for forward encoding. It is then processed sequentially through the audio feature extraction module, video feature extraction module, unified projection module, single-modal temporal stabilization module, local temporal enhancement module, QIT module, and post-fusion context refinement module to obtain the final fused feature sequence Y.

[0129] Finally, the decoding output module generates a text sequence step by step based on the final fused feature sequence Y. During the decoding stage, a beam search strategy can be used to generate multiple candidate text sequences, and the candidate text with the highest score is taken as the final recognition result.

[0130] By adopting the above decoding method, the multimodal robust representation learned in the pre-training and fine-tuning stages can be fully utilized to obtain more stable recognition results under noise, occlusion and slight misalignment conditions.

[0131] This invention is not a simple splicing fusion, but rather improves audio and video speech recognition performance in complex environments through a collaborative design of single-modal stabilization, dynamic quality-aware reweighting, staggered structured interaction, and controlled refinement within the same frame. This invention addresses the performance degradation of existing speech recognition methods in complex environments such as noise enhancement, visual modality degradation, and partial occlusion. While maintaining recognition accuracy, this invention improves the structural rationality of multimodal fusion, its adaptability to modal quality fluctuations, its tolerance to slight misalignments, and its engineering deployment efficiency. It is suitable for robust speech content recognition in noisy environments, channel degradation environments, lip-syncing environments, or environments with dynamic changes in audio and video modal quality.

[0132] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. An audio / video speech recognition method based on quality-perceived interleaved marker fusion, characterized in that, Includes the following steps: Step 1. Acquire audio and video data, preprocess the audio and video data, and construct a training dataset; Step 2. Construct a layered and decoupled audio / video speech recognition model, which includes: The audio feature extraction module is used to perform frame segmentation, spectral transformation, Mel filter bank mapping, and shallow convolution processing on the audio sequence to obtain audio frame-level features. The video feature extraction module is used to perform shallow 3D convolution and 2D lightweight convolution on a continuous sequence of lip images to obtain video frame-level features. The unified projection module is used to project audio frame-level features and video frame-level features onto the same dimension through linear transformation. The single-modal temporal stabilization module is used to perform bidirectional state-space modeling on the projected audio and video features to generate stabilized audio and video features. The local temporal enhancement module is used to further enhance the stabilized audio and video features through convolution; The quality-aware interleaved tag fusion module is used to jointly generate frame-by-frame fusion weights by combining content relevance weights and modality reliability scores, and to construct interleaved tag sequences with explicit frame-level structural constraints through the interleaved tag construction module to achieve cross-modal joint modeling. Then, the frame-aligned cross-gated residual refinement module injects bidirectional controlled information into the dual-modal information within the same time frame. The post-fusion context refinement module is used to further integrate features through a temporal coding network; And a decoding output module; Step 3. Train the hierarchical decoupled audio-visual speech recognition model based on the training dataset from Step 1, and use the trained model to perform speech recognition and output the text recognition results.

2. The audio / video speech recognition method based on quality-perceived interleaved marker fusion according to claim 1, characterized in that, In step 2, the audio feature extraction module includes an audio preprocessing unit, a spectral feature extraction unit, a shallow convolutional front-end, and a linear mapping front-end; The audio preprocessing unit is used to perform frame segmentation and windowing processing on the received audio sequence; The spectral feature extraction unit is used to perform short-time Fourier transform on the framed audio signal to obtain spectral features, and further obtain log-Mel spectral features through Mel filter bank mapping and logarithmic transform; The shallow convolutional front end is used to extract local time-frequency patterns from the log-Mel spectrum features; the linear mapping front end is used to map the acoustic features output by the shallow convolutional front end into an audio frame-level feature sequence A of a set dimension. The video feature extraction module includes a shallow 3D convolutional front-end and a 2D lightweight convolutional network; Shallow 3D convolutional front-ends are used to simultaneously perform convolution processing on consecutive lip image sequences in both temporal and spatial dimensions to extract short-term lip movement information between adjacent video frames. Two-dimensional lightweight convolutional networks are used to further extract spatial features from the frame-by-frame features output by the three-dimensional convolutional front-end through convolution, and obtain the video frame-level feature sequence V through global pooling or flattening mapping; The size of the audio frame-level feature sequence A is T×Da, and the size of the video frame-level feature sequence V is T×Dv; T is the number of time frames, Da is the audio feature dimension, and Dv is the video feature dimension.

3. The audio / video speech recognition method based on quality-perceived interleaved marker fusion according to claim 1, characterized in that, In step 2, the unified projection module includes an audio linear projection layer, a video linear projection layer, and a layer normalization layer. The audio frame-level feature sequence A is mapped to the hidden dimension d through an audio linear projection layer, and then subjected to layer normalization to obtain the projected audio feature sequence A1. The video frame-level feature sequence V is mapped to the hidden dimension d through a video linear projection layer, and then subjected to layer normalization to obtain the projected video feature sequence V1; the dimensions of A1 and V1 are both T×d.

4. The audio / video speech recognition method based on quality-perceived interleaved marker fusion according to claim 1, characterized in that, In step 2, for any modal input feature sequence X, the processing flow of the single-modal temporal stabilization module is as follows: First, the input feature sequence X is subjected to layer normalization to obtain the normalized feature sequence LN(X); Then, LN(X) is input into the forward state space modeling unit for processing to obtain the forward output F(X); at the same time, LN(X) is input into the backward state space modeling unit in reverse time dimension for processing to obtain the backward output B(X); The forward state space modeling unit includes an input projection layer, a state update unit, a gating unit, and an output projection layer. Its processing flow is as follows: For the normalized feature sequence LN(X), the input projection layer first maps it to the intermediate features required for state space modeling; the state update unit recursively updates the hidden state from frame 1 to frame T in chronological order to model the long-range dependency between the current frame and historical frames; the gating unit generates modulation weights based on the features of the current frame and selectively enhances or suppresses the state update result; the output projection layer maps the state update result back to the same hidden dimension d as the input feature sequence X to obtain the forward output F(X); The backward state space modeling unit has the same structure as the forward state space modeling unit, but its input is LN(X) arranged in reverse time dimension, and it performs state updates in reverse time order. After obtaining the reverse output, restore it according to the original time sequence to obtain the backward output B(X); The forward output F(X) and the backward output B(X) are fed into the bidirectional fusion unit for fusion to obtain the bidirectional fusion result U(X). The specific fusion formula is: U(X) = [F(X) + B(X)] / 2; Subsequently, U(X) is added to the original input feature sequence X through the residual connection unit to obtain the residual fusion feature R(X), which is calculated as: R(X) = X + U(X); Finally, the residual fusion feature R(X) is fed into a random deactivation unit to obtain the stabilized feature sequence X1. When X is the projected audio feature sequence A1, the output of the single-mode temporal stabilization module is the stabilized audio feature sequence A2; When X is the projected video feature sequence V1, the output of the single-modal temporal stabilization module is the stabilized video feature sequence V2.

5. The audio / video speech recognition method based on quality-perceived interleaved marker fusion according to claim 1, characterized in that, In step 2, for any modality stabilization feature sequence X1, the processing flow of the local temporal enhancement module is as follows: First, X1 is convolved along the time dimension in one dimension to obtain the local context enhancement feature C(X1); Then, C(X1) is subjected to layer normalization and nonlinear activation to obtain the enhanced feature E(X1); Finally, E(X1) and X1 are added together by residuals to obtain the feature X2 after local temporal enhancement, which is calculated by the following formula: X2 = X1 + E(X1); When X1 is the stabilized audio feature sequence A2, the output of the local temporal enhancement module is the enhanced audio feature sequence A3; When X1 is the stabilized video feature sequence V2, the output of the local temporal enhancement module is the enhanced video feature sequence V3.

6. The audio / video speech recognition method based on quality-perceived interleaved marker fusion according to claim 1, characterized in that, In step 2, the quality-aware interleaved label fusion module includes a modality-aware router, a quality gating module, a weight combination recalibration module, an interleaved label construction module, a joint transformer coding module, a deinterleaving recovery module, and a frame alignment cross-gating residual refinement module. The modality-aware router includes a linear layer, a layer normalization layer, and a nonlinear activation layer; it processes the enhanced audio feature sequence A3 and the enhanced video feature sequence V3 frame by frame to obtain the content relevance weight sequence ra of the audio modality and the content relevance weight sequence rv of the video modality. The quality gating module includes a linear layer, a layer normalization layer, and a nonlinear activation layer; it processes the enhanced audio feature sequence A3 and the enhanced video feature sequence V3 frame by frame to obtain the audio reliability score sequence qa and the video reliability score sequence qv. The weight combination recalibration module combines the ra and rv output by the modality sensing router with the qa and qv output by the quality gating module to obtain the fusion weight. Then, the fusion weight is used to recalibrate the enhanced audio features and enhanced video features frame by frame to obtain the recalibrated audio feature sequence A4 and the recalibrated video feature sequence V4. The interleaved marker construction module interleaves A4 and V4 according to time frames to obtain the interleaved marker sequence Z; specifically, in the t-th time frame, V4(t) and A4(t) are arranged adjacently to obtain the interleaved marker sequence Z = [V4(1), A4(1), V4(2), A4(2),..., V4(t), A4(t),..., V4(T), A4(T)]; where V4(t) represents the video feature of the t-th time frame in the recalibrated video feature sequence V4, and A4(t) represents the audio feature of the t-th time frame in the recalibrated audio feature sequence A4, t=1,2,...,T; The joint transformer encoding module processes the interleaved label sequence Z to obtain the joint context output sequence J. Specifically, the joint transformer encoding module includes a position encoding unit, a multi-head self-attention unit, a feedforward network unit, a residual connection unit, and a layer normalization unit. The position encoding unit is used to add temporal position information, the multi-head self-attention unit is used to model cross-time and cross-modal relationships, the feedforward network unit is used to enhance feature representation, and the residual connection unit and the layer normalization unit are used to stabilize feature propagation. The deinterlacing recovery module re-splits the joint context output sequence J into the audio output feature sequence Ja and the video output feature sequence Jv according to the original interlacing arrangement; The frame-aligned cross-gated residual refinement module performs bidirectional controlled information injection and residual refinement processing on the audio output feature sequence Ja and the video output feature sequence Jv within the same time frame, resulting in the refined audio feature sequence A5 and the refined video feature sequence V5. By concatenating A5 and V5 along their feature dimensions, the fused feature sequence F output by the QIT module is obtained.

7. The audio / video speech recognition method based on quality-perceived interleaved marker fusion according to claim 6, characterized in that, For any time frame t, the processing flow of the modality-aware router is as follows: First, the enhanced audio feature A3(t) and the enhanced video feature V3(t) are concatenated along the feature dimension to obtain the concatenated feature C1(t); Then, the spliced ​​feature C1(t) is processed sequentially through a linear layer, a layer normalization layer, a nonlinear activation layer, and a linear layer to obtain a two-dimensional routing value. Finally, the two-dimensional routing values ​​are input into the normalization function to obtain the content relevance weight ra(t) of the audio modality and the content relevance weight rv(t) of the video modality, which satisfy: ra(t) + rv(t) = 1.

8. The audio / video speech recognition method based on quality-perceived interleaved marker fusion according to claim 6, characterized in that, In step 2, for any time frame t, the processing flow of the weight combination recalibration module is as follows: First, the audio content relevance weight ra(t) output by the modality-aware router is combined with the audio reliability score qa(t) output by the quality gating module to obtain the unnormalized audio weight ua(t), which is calculated using the following formula: ua(t) = ra(t) × qa(t); Simultaneously, the video content relevance weight rv(t) output by the modality-aware router is combined with the video reliability score qv(t) output by the quality gating module to obtain the unnormalized video weight uv(t), the calculation formula of which is: uv(t) = rv(t) × qv(t); Then, normalize ua(t) and uv(t) to obtain the final audio fusion weight wa(t) and video fusion weight wv(t), which are calculated using the following formula: wa(t) = ua(t) / [ua(t) + uv(t) + ε]; wv(t) = uv(t) / [ua(t) + uv(t) + ε]; Where ε is a minimal constant to prevent the denominator from being zero; Finally, using wa(t) and wv(t), the enhanced audio feature A3(t) and the enhanced video feature V3(t) are recalibrated frame by frame to obtain the recalibrated audio feature A4(t) and the recalibrated video feature V4(t), respectively. The calculation formula is as follows: A4(t) = wa(t) × A3(t); V4(t) = wv(t) × V3(t).

9. The audio / video speech recognition method based on quality-perceived interleaved marker fusion according to claim 6, characterized in that, In step 2, for any time frame t, the processing flow of the frame alignment cross-gated residual refinement module is as follows: First, the video output feature Jv(t) is input into the video and processed by the audio projection subunit to obtain the message vector Mva(t) injected into the audio branch; The audio output feature Ja(t) is input into the audio and processed by the video projection subunit to obtain the message vector Mav(t) injected into the video branch; Then, Ja(t) and Jv(t) are concatenated according to the feature dimension to obtain the concatenated feature [Ja(t), Jv(t)], and the concatenated feature is input into the audio side gating subunit. After processing by the multilayer perceptron and the Sigmoid activation function, the audio side gating coefficient ga(t) is obtained. Meanwhile, Jv(t) and Ja(t) are concatenated according to the feature dimension to obtain the concatenated feature [Jv(t), Ja(t)], and the concatenated feature is input into the video side gating subunit. After processing by the multilayer perceptron and the Sigmoid activation function, the video side gating coefficient gv(t) is obtained. Subsequently, the cross-modal message injection intensity is controlled by the gating coefficient and scaling factor s, and after residual summation and layer normalization, the refined audio feature A5(t) and the refined video feature V5(t) are obtained, and their calculation formulas are as follows: A5(t) = LN[Ja(t) + s × ga(t) × Mva(t)]; V5(t) = LN[Jv(t) + s × gv(t) × Mav(t)]; Where LN[ ] represents the layer normalization operation.

10. The audio / video speech recognition method based on quality-perceived interleaved marker fusion according to claim 1, characterized in that, In step 2, the post-fusion context refinement module uses a convolutional enhanced temporal encoder to further process the fusion feature sequence F output by the quality-aware interleaved label fusion module to obtain the final fusion feature sequence Y. The convolution-enhanced temporal encoder includes an attention layer, a temporal convolutional layer, a feedforward network layer, a residual connection layer, and a layer normalization layer. The attention layer is used to model the long-range dependencies between different time frames in the fused feature sequence F. The temporal convolutional layer is used to extract the local temporal dynamics between adjacent time frames. The feedforward network layer is used to enhance the feature representation capability of each time frame. The residual connection layer and the layer normalization layer are used to stabilize feature propagation and model training.

Images