Speech recognition method and system based on multi-modal fusion and adaptive noise modeling, and storage medium

Abstract

The invention discloses a speech recognition method and system based on multi-modal fusion and adaptive noise modeling, and a storage medium, and belongs to the technical field of speech recognition. The method comprises the following steps: S1, synchronously acquiring multi-modal data; s2, performing voice preprocessing, and extracting an acoustic feature sequence; s3, extracting a visual lip shape feature sequence; s4, generating a low-dimensional noise vector; s5, performing cross-modal time alignment to obtain a multi-modal feature sequence corresponding to time; s6, unifying the semantic embedding space; s7, dynamically calculating and fusing the multi-modal features in the unified embedding space through an adaptive weighted fusion network to obtain a fusion feature vector; s8, noise self-adaptive denoising is carried out; s9, acoustic modeling is carried out through the end-to-end acoustic model, and a probability sequence is output; s10, decoding the probability sequence by the language model to obtain an initial text sequence; and S11, performing post-processing on the initial text sequence. The method can be adaptive to environmental noise, dynamic fusion between modes is realized, and high robustness is kept.

Description

Technical Field

[0001] This invention belongs to the field of speech recognition technology, specifically relating to a speech recognition method, system, and storage medium based on multimodal fusion and adaptive noise modeling. Background Technology

[0002] Automatic speech recognition technology has been widely applied in human-computer interaction, smart devices, meeting transcription, and other scenarios. Existing automatic speech recognition technologies are mainly divided into two categories: one is single-modal audio feature modeling methods, and the other is multi-modal fusion speech recognition methods.

[0003] Single-modal audio feature modeling methods fall into two categories: First, Hidden Markov Model-Deep Neural Network (HMM-DNN) systems based on Mel-Frequency Cepstral Coefficients (MFCC) features. These systems leverage the combined effects of acoustic and language models to achieve speech-to-text mapping. Second, end-to-end Automatic Speech Recognition (ASR) systems, such as Recurrent Neural Network-Connected Temporal Classification (RNN-CTC) systems or Transformer-based ASR systems. These systems directly model speech and text sequences, effectively improving the flexibility of speech recognition. For speech recognition in noisy environments, methods such as spectral subtraction, Wiener filtering, or adding noise data for enhancement are mainly used to suppress noise. For example, Chinese patent CN119206608A discloses a system and method for safety detection by maintenance personnel, which mentions the use of single-modal audio feature modeling. However, the single-modal method is highly dependent on the audio signal, and its recognition performance will be significantly reduced in low signal-to-noise ratio (SNR) or complex noise environments.

[0004] Multimodal fusion speech recognition methods primarily fuse acoustic features (e.g., Mel-frequency cepstral coefficients, short-time Fourier transform features) with visual lip-shape features or environmental noise features. The fused features are then input into an end-to-end automatic speech recognition (ASR) model for recognition. Compared to the single-modal methods mentioned above, this multimodal fusion method improves recognition accuracy in noisy environments to some extent. However, this multimodal fusion speech recognition method has a simple feature fusion mechanism, lacks an adaptive weight adjustment mechanism, and has insufficient modeling ability for noise types. This leads to a decline in speech recognition performance in complex noisy environments. Therefore, in practical applications, the aforementioned multimodal fusion speech recognition method still has the following significant drawbacks: (1) Insufficient noise robustness; lack of adaptive modeling capability for noise types, fixed processing strategy, unable to be dynamically adjusted; (2) Multimodal information fusion is insufficient; simple feature splicing or fixed weight fusion is often used, which cannot be dynamically adjusted according to the environment and modal reliability. (3) Poor adaptability to complex environments; model training relies on fixed scene data and lacks online adaptive mechanisms, resulting in large performance fluctuations under different or sudden noise environments. Summary of the Invention

[0005] To address the problems in related technologies, this invention proposes a speech recognition method, system, and storage medium based on multimodal fusion and adaptive noise modeling, in order to overcome the aforementioned technical problems in existing related technologies. This enables the speech recognition system of this invention to adapt to environmental noise, achieve intelligent dynamic fusion between modalities, and maintain high robustness.

[0006] The technical solution of this invention is implemented as follows: a speech recognition method based on multimodal fusion and adaptive noise modeling, comprising the following steps: Step S1: Perform synchronous acquisition of multimodal data and obtain synchronous multimodal data stream signals; the multimodal data stream signals include audio stream signals, video lip-sync stream signals, and environmental noise signals; Step S2: Perform speech preprocessing on the audio stream signal to extract multidimensional acoustic feature sequences; Step S3: Process the video lip flow signal to extract a visual lip feature sequence containing static shape, dynamic optical flow and three-dimensional geometric information; Step S4: Based on the environmental noise signal, perform noise analysis, classification, and generate a low-dimensional noise vector characterizing the noise type and intensity; Step S5: Perform cross-modal temporal alignment on the acoustic feature sequence, visual lip shape feature sequence, and low-dimensional noise vector to obtain the temporally corresponding multimodal feature sequence; Step S6: Map the temporally aligned modal features to a unified semantic embedding space; Step S7: Based on the low-dimensional noise vector and the real-time confidence of each modality feature, the multimodal features in the unified embedding space are dynamically calculated and fused through an adaptive weighted fusion network to obtain the fused feature vector; Step S8: Use the noise adaptive denoising network guided by the low-dimensional noise vector to enhance the acoustic features or fused features to obtain denoised and enhanced features; Step S9: Using the fused feature vector and / or the denoising enhancement feature as input, perform acoustic modeling through an end-to-end acoustic model, and output a probability sequence corresponding to the language unit; Step S10: Introduce a language model to decode the probability sequence and use the BeamSearch algorithm to obtain the initial text sequence; Step S11: Perform post-processing on the initial text sequence, including punctuation restoration, proper noun correction, and text cleaning, to output the final speech recognition text.

[0007] Furthermore, in step S1, a hardware clock synchronization protocol is used to provide a unified clock reference for all acquisition sensors, and software timestamp calibration and timing drift compensation (such as periodic frame insertion / deletion) are performed on devices that cannot be hardware synchronized to obtain the time-synchronized multimodal data stream signal. Furthermore, the original timestamps for the three signals—audio stream, video lip-sync stream, and ambient noise—are set as follows: ; in, These represent the time indices in the speech signal, visual lip-sync video signal, and ambient noise signal, respectively, and are used to identify the positions of different sampling points or time frames in the corresponding modal. Then, a time remapping function is used to align the times of each mode on a unified time axis. The formula for calculating the time remapping function is as follows: ; in, This is the device offset, and It is obtained by fitting with the minimum mean square error; where... It is a generic index variable, and its specific values ​​include These correspond to the audio frame index, video frame index, and noise frame index, respectively. It is a time remapping function; Specific values ​​include ; It should be noted that in order to handle clock drift during long-term synchronous acquisition, this invention introduces periodic drift detection. By statistically analyzing the rate of change of timestamp differences among the multimodal data, frame insertion or deletion compensation is automatically performed. After step S1, a strictly synchronized multimodal data stream can be obtained, laying the foundation for multimodal time alignment in the subsequent step S5.

[0008] Preferably, the hardware clock synchronization protocol is the PTP precision clock synchronization protocol; Furthermore, the acquisition sensors include, but are not limited to, camera sensors, microphone sensors, and noise sensors.

[0009] In step S2, the audio stream signal often contains background noise, device noise, and speech silence segments in actual acquisition environments. Therefore, systematic speech preprocessing is required before entering the deep model. This invention constructs a speech preprocessing process including pre-emphasis filtering, Voice Activity Detection (VAD), endpoint detection, frame windowing, frequency domain transformation, and multi-feature extraction. The specific speech preprocessing process is as follows: First, the pre-emphasis filtering is used to enhance the high-frequency energy of the audio stream signal, giving plosives, fricatives, etc., a clearer frequency structure. The calculation expression is as follows: ; in, This is the pre-weighting factor, used to control the high-frequency enhancement intensity; its value is typically 0 < <1, which is preferably 0.95 in this embodiment; n is the discrete time index, used to represent the sampling position of the speech signal in the time series; This represents the sample value of the original speech signal at the discrete-time index n; This represents the sampled value of the original speech signal at the previous sampling time (n-1); This represents the output value of the speech signal obtained at time index n after the pre-emphasis filtering; It should be emphasized that step S2 can effectively enhance the high-frequency energy of the audio stream signal, providing a clearer frequency structure for subsequent MFCC feature construction; The audio stream signal is then subjected to the Voice Activity Detection (VAD) to automatically remove silent segments of speech, thereby reducing computational load and increasing the density of training data. The VAD process combines short-time energy, zero-crossing rate, and dynamic threshold to automatically identify speech boundaries. The endpoint detection further precisely delineates the range of speech segments, reducing the impact of background noise. In the frequency domain processing of step S2, the audio stream signal is divided into frames and windowed according to a window length of 25ms and a frame shift of 10ms. The short-time fourier transform (STFT) is used to map the time signal to the time-frequency plane to obtain various frequency representations such as spectrogram and power spectrum. The multidimensional acoustic feature sequence includes MFCC, pitch, energy spectrum, and formalant; mean and variance normalization is performed on the acoustic feature sequence to make it have more stable distribution characteristics in cross-speaker and cross-device scenarios; finally, the output multidimensional acoustic feature sequence will serve as the core input for mapping each modality feature to the unified semantic embedding space in subsequent step S6 and for acoustic modeling in step S9.

[0010] In step S3, optical flow features are first calculated to extract the motion changes of the lips between frames. A three-dimensional facial geometry restoration algorithm (such as 3D Morphable Model) is introduced to map the 2D lip image onto a 3D mesh to obtain a three-dimensional motion vector. Finally, the visual lip shape feature sequence is obtained, which provides compensatory visual information for the adaptive weighted fusion network in the subsequent step S7.

[0011] Further, in step S4, noise signals are first extracted from the non-speech segments of the audio stream signal, and the spectral characteristics of the noise signals are calculated for noise type determination; wherein, the spectral characteristics include spectral quality characteristics, spectral attenuation rate characteristics, and spectral flatness characteristics; This invention concatenates statistical features such as spectral quality, spectral attenuation rate, and spectral flatness, and generates a noise embedding vector of uniform dimension through linear mapping or a lightweight neural network encoder, expressed as follows: ; in, , These represent the mean vector and variance vector of the noise spectral characteristics in the time dimension, respectively. This represents a noise coding function used to map multidimensional noise statistical features to a fixed-dimensional noise vector space. The signal-to-noise ratio of the noise signal is then calculated using the following estimation formula: ; in, The average power of the speech segment of the audio stream signal; The average power of the noise segment of the audio stream signal; Finally, the noise type and the signal-to-noise ratio information are mapped to the low-dimensional noise vector through a noise coding network.

[0012] Further, in step S5, the visual lip shape feature sequence is mapped onto the reference time axis using interpolation, based on the time axis of the acoustic feature sequence, so that it is aligned with the target timestamp. For example, when the audio is 100 FPS and the video is 25 FPS, the present invention inserts intermediate frames or repeats keyframes into the visual frames to align them with the target timestamp. The alignment effect is further refined using the Dynamic Time Warping (DTW) algorithm. This algorithm can automatically find the optimal matching path between two sequences, and is especially suitable for situations where lip movements and speech rhythms are out of sync. The calculation formula is as follows: ; in, The audio frames of the acoustic feature sequence Visual frames with the visual lip shape feature sequence The similarity; this invention utilizes the above-mentioned minimum cumulative distance path to achieve optimal alignment of modal sequences; After processing in step S5, a multimodal feature sequence corresponding to a unified time is obtained, which provides a consistent time basis for the mapping of each modality feature in step S6 and the fusion of multimodal features in step S7. It should be noted that in existing multimodal fusion speech recognition systems, the sampling frequencies and frame periods of audio features, visual lip features, and environmental noise features are different, which can lead to "temporal misalignment" between modalities. If these features are directly fed into the fusion network module, they will cause serious information shift. Therefore, in step S5 of this invention, a cross-modal time alignment mechanism is designed to achieve strict correspondence between multimodal feature sequences in the time dimension through strategies such as frame-level alignment, interpolation alignment, and synchronization beacon calibration.

[0013] In step S6, since audio features, visual lip shape features, and noise features have different dimensions, statistical distributions, and information densities, they need to be mapped to a unified embedding space for subsequent fusion. The core objective of step S6 is to construct a "cross-modal semantically consistent embedding representation" so that different modalities are comparable and fusionable in the embedding space. First, an independent encoding network is built for each modality, configuring an audio encoder (CNN + Transformer), a visual encoder (3D CNN + LSTM), and a noise feature encoder (MLP or lightweight attention network). It should be noted that CNN stands for Convolutional Neural Network, and LSTM stands for Long Short-Term Memory. Each encoder outputs modal features of different dimensions, therefore a unified mapping is required, specifically: An audio coding network is constructed for the audio features using the audio encoder; a visual coding network is constructed for the visual lip shape features using the visual encoder; and a noise coding network is constructed for the environmental noise features using the noise feature encoder. Then, using a linear projection structure and a nonlinear activation structure, all the above modes are mapped to a d-dimensional unified space. The calculation formula is as follows: ; in, For modal primitive features, It is a linear projection matrix. To unify the embedding vector; Indicates the first Embedding Mapping Function corresponding to each modality. Indicates the first Bias Vector corresponding to each mode. Finally, the audio, visual lip shape, and noise modal embedding features output in step S6 will be used as input to the adaptive weighted fusion network in step S7 to achieve close cross-modal integration.

[0014] Further, in step S7, the adaptive weighted fusion network dynamically generates and allocates fusion weights for each modality based on the low-dimensional noise vector and the real-time calculated audio intelligibility index and lip-sync detection confidence. The fusion feature vector is a weighted sum of the embedded features of each modality. Traditional multimodal methods typically employ simple concatenation or fixed weighting strategies, but these cannot adapt to scenarios involving noise variations or fluctuations in the quality of the current modality. The adaptive weighting mechanism proposed in this invention can dynamically calculate the fusion weights based on the modal quality of the current frame. ; The current frame refers to the "multimodal frame under a unified time index" that has been time-aligned in step S5; The modality scoring network is driven by the low-dimensional noise vector and the detection confidence of each modality. For modal weights, To unify embedding features; This represents a modality index, used to distinguish different modality types; The summation index is used to normalize the calculation of all modalities involved in the fusion. This mechanism ensures that the weight of visual features is increased in high-noise environments, while the weight of audio features is increased in low-light or camera-occluded environments, forming a dynamic fusion system with environmental adaptability. The formula for calculating the fused output is: ; The fused vectors have multimodal complementarity and can maintain a high amount of information under various complex conditions such as noisy environments, long-distance speech scenarios, and low-light environments; In addition, this invention designs a "modal confidence estimation module" that assigns a real-time confidence score to each modality by calculating indicators such as audio signal clarity, lip-sync detection stability, and noise intensity, making the fusion more intelligent. Ultimately, the fused features output in step S7 will serve as the input to the acoustic model in the subsequent step S9, which is crucial for improving system performance.

[0015] Furthermore, in step S8, the noise adaptive denoising network is a deep network with U-Net or a denoising autoencoder as its backbone. It uses the low-dimensional noise vector as a conditional input and dynamically adjusts the internal parameters of the network through the feature modulation layer to achieve adaptive denoising for specific noise types and intensities. In step S8, the present invention further proposes a "noise adaptive denoising network"; its core idea is that different noise types destroy speech features in different ways, and a denoising model with fixed parameters cannot achieve the best results. Therefore, it is necessary to adjust the parameters of the denoising network autonomously according to the noise vector. First, this invention constructs a deep denoising model based on the U-Net or a denoising autoencoder (DAE), and guides the deep network to perform parameter adjustment through the low-dimensional noise vector; the parameters of the U-Net are... Noise embedding is Then, a hyperparameter-tuned network is used to generate the dynamic offset, and the calculation formula is: ; in, For lightweight parameter prediction networks, These are the adaptive denoising parameters under the current noise conditions. Through this mechanism, the denoising network can perform lightweight enhancement at high signal-to-noise ratios and strong noise suppression at extremely low signal-to-noise ratios, ensuring that the output speech always maintains a high signal-to-noise ratio. In addition, the present invention also designed a "visual assistance enhancement module" that uses lip-shape dynamic information to infer the energy change trend of speech when the audio is severely damaged, thereby improving the noise reduction effect; Finally, the enhanced speech features (spectral or Mel features) output in step S8 will serve as the core input for acoustic modeling in step S9, enabling the system to maintain stable recognition performance even in extreme noise environments.

[0016] Furthermore, in step S9, the end-to-end acoustic model is a Transformer or Conformer structure, and the encoder of the end-to-end acoustic model includes a cross-modal attention mechanism, so that during the acoustic modeling process, audio features are used to focus on the corresponding visual lip features. Based on the fused features output in step S7 and the enhanced speech features obtained in step S8, an end-to-end acoustic modeling module is designed in step S9 to map the fused multimodal information into language units (such as phonemes, characters, words, or sub-word units). Unlike traditional HMM-DNN speech recognition systems, this invention adopts end-to-end Transformer, Conformer, or LSTM-CTC structures to reduce intermediate steps, lower complexity, and greatly improve modeling capabilities. It should be noted that DNN stands for Deep Neural Network. First, this invention employs a deep encoder structure, taking the fused vector sequence as input; the deep encoder includes a multi-head attention mechanism to capture long-term dependencies, enabling the model to obtain pronunciation patterns and semantic transition information from the temporal context; in speech, many syllables span multiple time frames, and the correspondence between visual lip shapes and audio has time-delay characteristics, so long-distance modeling capability is crucial. To improve the robustness of the model in non-stationary noise environments, this invention introduces a "cross-modal attention" structure, which encodes audio features, visual features and noise features separately, and then uses a shared attention mechanism to perform cross-modal interaction, so that the audio model can prioritize visual dynamics in frames affected by noise, thus avoiding misjudgment. The output sequence of the end-to-end acoustic model is usually of a different length than the text sequence, therefore, it is necessary to use CTC (Connectionist Temporal Classification) or attention-based sequence-to-sequence decoding methods. For the case of using CTC, the loss function is calculated using the following formula: ; in, To fuse feature sequences, The target text label sequence; CTC enables the end-to-end acoustic model to handle inputs and outputs of different lengths by summing all valid alignment paths.

[0017] For the Transformer / Conformer structure, this invention also designs a "speech-visual consistency regularization term" to improve the stability of the model in multimodal collaborative learning by constraining the attention weights of the two modalities to be consistent in the pronunciation transition segment.

[0018] Further, in step S10, the decoding process combines the probability sequence output by the end-to-end acoustic model with the prior probability provided by the language model, and uses the beam search algorithm for searching. Specifically, candidate paths with high confidence in the visual lip shape features are given a score reward, which includes the following process: After completing the acoustic modeling, the present invention performs text decoding in step S10, that is, maps the probability sequence output by the acoustic model to the initial text sequence; since the sequence output by the acoustic model usually contains ambiguity, homophony interference and grammatical incompleteness, it is necessary to introduce a language model for global semantic constraints. This invention supports multiple types of language models, including N-gram statistical language models, RNN-LM, Transformer-LM, and pre-trained language models (such as GPT-like models); the language model is used to estimate the rationality of word sequences, thereby eliminating low-probability language segments during the decoding process; It should be noted that RNN stands for Recurrent Neural Network. During the decoding process, the present invention uses the beam search algorithm for searching; the beam search algorithm retains several candidate paths with the highest scores at each step, making the search process computationally manageable; if the acoustic model outputs a probability of The language model gives the sentence probability as follows: Therefore, the present invention uses the following joint scoring function: ; Where λ is the language model weight, used to balance the probabilities of acoustics and language; β is the length penalty coefficient; The sequence length; The candidate text sequence is the output sequence generated by the acoustic model during the decoding process, consisting of several discrete symbols (characters, words, or sub-word units). This joint scoring function formula enables the system to achieve a balance between "acoustic model confidence" and "linguistic rationality," thereby avoiding misjudgments caused by noise in the model. Furthermore, this invention incorporates a "multimodal consistency constraint" into the beam search algorithm to improve the score of sequences with high confidence in visual information among candidate paths. In this way, even when the audio is extremely damaged, the system can still generate stable text with the help of visual information (lip movement features), which is particularly suitable for scenarios with extremely complex noise, such as industrial environments and the interior of moving vehicles. In the decoding stage, this invention can also introduce a "context-adaptive language model" to automatically enhance specific terms and expressions based on the application field, such as medical, legal, or specific corporate terms, so that the recognition results are more in line with the actual scenario. Finally, step S10 outputs a high-confidence initial text sequence as a preliminary version of the final speech recognition result, which will be further processed in natural language level formatting in step S11. In step S11, although the output sequence after text decoding has strong semantic rationality under the constraints of the language model, there may still be problems such as missing punctuation, capitalization errors, and inaccurate recognition of proper nouns. Therefore, in step S11, the present invention implements a complete text post-processing mechanism to make the output result closer to the natural language text format. First, this invention designs a punctuation recovery model; since the end-to-end acoustic model typically does not generate punctuation marks, a deep sequence annotation model (such as BiLSTM-CRF or Transformer) is used to automatically add punctuation marks such as commas, periods, question marks, and pauses to the text; the end-to-end acoustic model recovers natural language writing habits by learning the semantic structure, word relationships, intonation patterns, and other information of the initial text sequence. Secondly, the post-processing output module includes a proper name recognition unit, which automatically identifies and corrects personal names, place names, organization names, and professional terms in the text. For example, "Zhang San" may be misidentified as "Changshan" due to the acoustic model. This invention automatically corrects this through contextual features and entity dictionary matching. Subsequently, this invention performs text cleaning processing, including deleting redundant and repeated words, removing colloquial filler words, and correcting typos. For example, filler words such as "um," "ah," and "that" often appear in real speech, and this system can automatically determine whether they need to be removed based on the context. To support diverse application scenarios, this invention also includes a domain dictionary enhancement mechanism, enabling the system to achieve higher accuracy in terminology recognition within specific industries. For example, in medical scenarios, terms such as "liver function" and "troponin" will be prioritized for recognition. Finally, step S11 outputs the final text after punctuation restoration, semantic enhancement, and domain correction, which is presented to the user as the final speech recognition text.

[0019] A speech recognition system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it applies a speech recognition method based on multimodal fusion and adaptive noise modeling as described above. The speech recognition system also includes a multimodal data acquisition module, a speech preprocessing module, a visual lip shape feature extraction module, a noise modeling module, a multimodal alignment module, a unified embedding space mapping module, a fusion network module, a noise adaptive denoising module, an end-to-end acoustic modeling module, a decoding module, and a post-processing output module. The multimodal data acquisition module is used for synchronous acquisition of multimodal data; The speech preprocessing module is used to preprocess the audio stream; The visual lip shape feature extraction module is used to process the video lip shape stream; The noise modeling module is used to perform noise analysis, classification, and generate low-dimensional noise vectors that characterize the type and intensity of noise. The multimodal alignment module is used to perform cross-modal temporal alignment of acoustic feature sequences, visual lip shape feature sequences, and low-dimensional noise vectors to obtain temporally corresponding multimodal feature sequences. The unified embedding space mapping module is used to map the time-aligned modal features to the unified semantic embedding space; The fusion network module dynamically calculates and fuses the multimodal features in the unified embedding space mapping module through an adaptive weighted fusion network to obtain a fused feature vector; The noise adaptive denoising module uses a noise adaptive denoising network to enhance acoustic features or fused features to obtain denoised and enhanced features. The end-to-end acoustic modeling module takes the fused feature vector and / or the denoising and enhancement features as input, performs acoustic modeling, and outputs a probability sequence corresponding to the language unit. The decoding module decodes the probability sequence; The post-processing output module is used to perform punctuation restoration, proper noun correction and text cleaning on the initial text sequence obtained by the beam search algorithm, and output the final recognized text.

[0020] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition method based on multimodal fusion and adaptive noise modeling as described above.

[0021] The beneficial effects of this invention are: (1) In view of the problem that the accuracy of existing technologies decreases in low signal-to-noise ratio and complex noise environment, the present invention realizes dynamic identification and denoising of different types of noise through an adaptive noise modeling module, thereby improving the robustness of the identification system in various noise environments.

[0022] (2) In view of the problems of simple feature fusion and fixed weight in existing technologies, this invention designs an adaptive multimodal feature fusion mechanism that can dynamically adjust the weight of each modality according to environmental noise and modal reliability, thereby improving the overall recognition performance.

[0023] (3) In view of the problem that existing technologies cannot adapt to environmental changes online, the present invention constructs an environmental perception and dynamic adjustment mechanism, so that the recognition system can maintain a stable recognition effect under different noise types and sudden environments.

[0024] (4) From data synchronization and acquisition, feature extraction, alignment, fusion, modeling to post-processing, this invention provides a complete and deployable solution with high practicality and scalability. Attached Figure Description

[0025] Figure 1 This is a flowchart of the method steps of the speech recognition method of the present invention; Figure 2 This is a schematic diagram of the module connection of the speech recognition system of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0028] like Figure 1 As shown, this embodiment provides a speech recognition method based on multimodal fusion and adaptive noise modeling, including the following steps: Step S1: Perform synchronous acquisition of multimodal data and obtain synchronous multimodal data stream signals; the multimodal data stream signals include audio stream signals, video lip-sync stream signals, and environmental noise signals; Step S2: Perform speech preprocessing on the audio stream signal to extract multidimensional acoustic feature sequences; Step S3: Process the video lip flow signal to extract a visual lip feature sequence containing static shape, dynamic optical flow and three-dimensional geometric information; Step S4: Based on the environmental noise signal, perform noise analysis, classification, and generate a low-dimensional noise vector characterizing the noise type and intensity; Step S5: Perform cross-modal temporal alignment on the acoustic feature sequence, visual lip shape feature sequence, and low-dimensional noise vector to obtain the temporally corresponding multimodal feature sequence; Step S6: Map the temporally aligned modal features to a unified semantic embedding space; Step S7: Based on the low-dimensional noise vector and the real-time confidence of each modality feature, the multimodal features in the unified embedding space are dynamically calculated and fused through an adaptive weighted fusion network to obtain the fused feature vector; Step S8: Use the noise adaptive denoising network guided by the low-dimensional noise vector to enhance the acoustic features or fused features to obtain denoised and enhanced features; Step S9: Using the fused feature vector and / or the denoising enhancement feature as input, perform acoustic modeling through an end-to-end acoustic model, and output a probability sequence corresponding to the language unit; Step S10: Introduce a language model to decode the probability sequence and use the BeamSearch algorithm to obtain the initial text sequence; Step S11: Perform post-processing on the initial text sequence, including punctuation restoration, proper noun correction, and text cleaning, to output the final speech recognition text.

[0029] Specifically, in step S1, a hardware clock synchronization protocol is used to provide a unified clock reference for all acquisition sensors, and software timestamp calibration and timing drift compensation (such as periodic frame insertion / deletion) are performed on devices that cannot be synchronized by hardware to obtain the time-synchronized multimodal data stream signal; thereby ensuring that the audio, video lip-sync and environmental noise data streams are synchronized at the millisecond level, laying the foundation for subsequent processing; More specifically, the original timestamps for the three signals—audio stream, video lip-sync stream, and ambient noise—are set as follows: ; in, These represent the time indices in the speech signal, visual lip-sync video signal, and ambient noise signal, respectively, and are used to identify the positions of different sampling points or time frames in the corresponding modal. Then, a time remapping function is used to align the times of each mode on a unified time axis. The formula for calculating the time remapping function is as follows: ; in, This is the device offset, and It is obtained by fitting with the minimum mean square error; where... It is a generic index variable, and its specific values ​​include These correspond to the audio frame index, video frame index, and noise frame index, respectively. It is a time remapping function; Specific values ​​include ; It should be noted that in order to handle clock drift during long-term synchronous acquisition, this embodiment introduces periodic drift detection. By statistically analyzing the rate of change of timestamp differences between the multimodal data, frame insertion or deletion compensation is automatically performed. After step S1, a strictly synchronized multimodal data stream can be obtained, laying the foundation for multimodal time alignment in the subsequent step S5.

[0030] Preferably, the hardware clock synchronization protocol is the PTP precision clock synchronization protocol; Specifically, the acquisition sensors include, but are not limited to, camera sensors, microphone sensors, and noise sensors.

[0031] In step S2, the audio stream signal often contains background noise, device noise, and speech silence segments in actual acquisition environments. Therefore, systematic speech preprocessing is required before it enters the deep model. This embodiment constructs a speech preprocessing process including pre-emphasis filtering, Voice Activity Detection (VAD), endpoint detection, frame windowing, frequency domain transformation, and multi-feature extraction. The specific speech preprocessing process is as follows: First, the pre-emphasis filtering is used to enhance the high-frequency energy of the audio stream signal, giving plosives, fricatives, etc., a clearer frequency structure. The calculation expression is as follows: ; in, This is the pre-weighting factor, used to control the high-frequency enhancement intensity; its value is typically 0 < <1, which is preferably 0.95 in this embodiment; n is the discrete time index, used to represent the sampling position of the speech signal in the time series; This represents the sample value of the original speech signal at the discrete-time index n; This represents the sampled value of the original speech signal at the previous sampling time (n-1); This represents the output value of the speech signal obtained at time index n after the pre-emphasis filtering.

[0032] It should be emphasized that step S2 can effectively enhance the high-frequency energy of the audio stream signal, providing a clearer frequency structure for subsequent MFCC feature construction; The audio stream signal is then subjected to Voice Activity Detection (VAD) to automatically remove silent segments of speech, thereby reducing computational load and increasing the density of training data. The VAD process combines short-time energy, zero-crossing rate, and dynamic threshold to automatically identify speech boundaries. Endpoint detection further precisely delineates the range of speech segments, reducing the impact of background noise. It should be noted that Voice Activity Detection (VAD) is a processing method used to distinguish between speech segments and non-speech segments. Its core objective is to automatically determine whether each time slice in a continuous speech stream contains valid speech components, thereby eliminating silent segments and pure noise segments. In this embodiment, VAD is mainly based on the short-time statistical characteristics of the speech signal for determination. Specifically, the system first performs short-time framing processing on the preprocessed speech signal, calculates multiple discrimination indicators within each frame, and comprehensively determines whether the frame is a speech frame based on these indicators. It should also be noted that: short-time energy is used to measure the energy intensity of a speech signal within a short time window; generally speaking, the energy of a speech frame is significantly higher than that of a silent or background noise frame; in this embodiment, the short-time energy value of the frame is obtained by accumulating or squaring the amplitude of the sampling points in each frame, and is used as an important basis for judging speech activity; The zero-crossing rate describes the number of times a signal crosses zero amplitude per unit time, reflecting the frequency variation characteristics of the signal. Under normal circumstances, the zero-crossing rate of a clear speech signal is significantly different from that of a pure noise signal. This embodiment uses the zero-crossing rate as an auxiliary discrimination indicator to avoid misjudgment in low-energy speech or sudden noise scenarios by relying solely on the energy threshold. Moreover, in order to adapt to the changes in speech characteristics under different speakers and different environmental noise intensities, this embodiment does not use a fixed threshold, but introduces a dynamic threshold mechanism. This mechanism adaptively adjusts the judgment threshold according to the statistical characteristics (such as average energy, energy variance, etc.) over a period of time, so that the VAD can maintain stable performance under strong noise or weak speech conditions. In the actual judgment process, this embodiment combines the results of short-time energy, zero-crossing rate and dynamic threshold to jointly judge each frame. Only when the preset voice activity conditions are met is the frame marked as a voice frame; otherwise, it is judged as a non-voice frame. After completing speech activity detection, this embodiment further performs endpoint detection processing to accurately locate the start and end points of speech segments. Unlike VAD, which is mainly used to distinguish between speech and non-speech, endpoint detection focuses on accurately delineating the effective speech segment range within the speech region, thereby reducing the impact of residual noise and invalid transition segments on subsequent feature extraction.

[0033] In the frequency domain processing of step S2, the audio stream signal is divided into frames and windowed according to a window length of 25ms and a frame shift of 10ms. The short-time fourier transform (STFT) is used to map the time signal to the time-frequency plane to obtain various frequency representations such as spectrogram and power spectrum. After completing the time-domain-based speech activity detection and endpoint detection, the system has obtained continuous and effective speech signal segments. Based on this, this embodiment further performs frequency domain analysis processing on the speech signal to extract spectral features that can characterize the acoustic properties of speech. Specifically, in this embodiment, the filtered speech signal is processed by framing and windowing according to a fixed window length and frame shift, wherein the preferred window length is 25ms and the preferred frame shift is 10ms; through framing operation, the non-stationary speech signal can be approximated as a stationary signal in a short period of time, thereby meeting the prerequisite for frequency domain analysis. For each windowed speech signal, this embodiment uses Short-Time Fourier Transform (STFT) to map the time-domain signal to the time-frequency plane; through STFT processing, the spectral distribution of the speech signal at different time positions can be obtained; The results of STFT are usually represented in the form of a spectrogram, with time on the horizontal axis and frequency on the vertical axis. The color or amplitude represents the energy intensity of the corresponding frequency component. Based on this, this embodiment can further calculate the power spectrum, that is, perform a square operation on the spectral amplitude to reflect the energy distribution characteristics of the speech signal in the frequency domain. The obtained spectrograms and power spectrograms provide a basic frequency domain representation for subsequent Mel filtering, MFCC feature extraction, and deep acoustic modeling; The multidimensional acoustic feature sequence includes MFCC, pitch, energy spectrum, and formalant; mean and variance normalization is performed on the acoustic feature sequence to make it have more stable distribution characteristics in cross-speaker and cross-device scenarios; finally, the output multidimensional acoustic feature sequence will serve as the core input for mapping each modality feature to the unified semantic embedding space in subsequent step S6 and for acoustic modeling in step S9.

[0034] In step S3, a face detection model based on a deep convolutional network or Vision Transformer is used to locate the face region. Then, a key point detection model (such as a 68-point or 106-point model) is used to obtain the coordinates of the key points of the mouth, and the lip ROI region is automatically cropped accordingly. This process can adapt to changes in facial pose, lighting, and camera position, so that the lip region always remains clear and structurally consistent. In terms of dynamic modeling, this embodiment first calculates optical flow features to extract the motion changes of the lips between frames, especially in the transition phase between consonants and vowels, which has obvious dynamic features; In addition, this embodiment introduces a three-dimensional facial geometry restoration algorithm (such as 3D Morphable Model) to map the 2D lip image onto a 3D mesh, resulting in a more stable three-dimensional motion vector and avoiding lip jitter caused by video noise; Finally, the visual lip shape feature sequence is obtained, which provides compensatory visual information for the adaptive weighted fusion network in the subsequent step S7.

[0035] Specifically, in step S4, noise signals are first extracted from the non-speech segments of the audio stream signal, and the spectral characteristics of the noise signals are calculated for noise type determination; wherein, the spectral characteristics include spectral quality characteristics, spectral attenuation rate characteristics, and spectral flatness characteristics; This embodiment concatenates statistical features such as spectral quality, spectral attenuation rate, and spectral flatness, and generates a noise embedding vector of uniform dimension through linear mapping or a lightweight neural network encoder, which is expressed as follows: ; in, , These represent the mean vector and variance vector of the noise spectral characteristics in the time dimension, respectively. This represents a noise coding function used to map multidimensional noise statistical features to a fixed-dimensional noise vector space. The signal-to-noise ratio of the noise signal is then calculated using the following estimation formula: ; in, The average power of the speech segment of the audio stream signal; The average power of the noise segment of the audio stream signal; Finally, the noise type and the signal-to-noise ratio information are mapped to the low-dimensional noise vector through a noise coding network to guide the multimodal weighting in step S7 and the adaptive denoising network parameter adjustment in step S8. The low-dimensional noise vector can effectively express the differences in different noise environments, enabling the system to maintain robustness in dynamic noise.

[0036] Specifically, in step S5, the visual lip shape feature sequence is mapped onto the reference time axis using interpolation, based on the time axis of the acoustic feature sequence, so that it is aligned with the target timestamp. For example, when the audio is 100 FPS and the video is 25 FPS, this embodiment will insert intermediate frames or repeat keyframes into the visual frames to align them with the target timestamp. Specifically, in step S5, the frame-level timestamps corresponding to the acoustic feature sequences constructed in step S2 are used as a unified alignment reference time axis, and the set of timestamps of the acoustic feature sequences is defined as the set of target timestamps; the feature values ​​of the visual lip shape feature sequences on their original timestamps are mapped to the set of target timestamps through interpolation, thereby achieving alignment between the visual modality and the acoustic modality in the time dimension. The alignment effect is further refined using the Dynamic Time Warping (DTW) algorithm. This algorithm can automatically find the optimal matching path between two sequences, and is especially suitable for situations where lip movements and speech rhythms are out of sync. The calculation formula is as follows: ; in, The audio frames of the acoustic feature sequence Visual frames with the visual lip shape feature sequence The similarity; this embodiment utilizes the above-mentioned minimum cumulative distance path to achieve optimal alignment of modal sequences; In addition, this embodiment also designs an alignment mechanism based on "synchronization beacons", that is, recording the flashing of light signals in the video stream and recording instantaneous pulses in the audio stream, which serves as "cross-modal anchor points" to automatically correct micro-hour timing drift during the acquisition phase; this mechanism is particularly important during long-term recording, as it can ensure that lip features and speech features remain synchronized at all times. After processing in step S5, a multimodal feature sequence corresponding to a unified time is obtained, which provides a consistent time basis for the mapping of each modality feature in step S6 and the fusion of multimodal features in step S7. It should be noted that in existing multimodal fusion speech recognition systems, the different sampling frequencies and inconsistent frame periods of audio features, visual lip features, and environmental noise features can lead to "temporal misalignment" between modalities. If these features are directly fed into the fusion network module, they will cause serious information shifts. Therefore, this embodiment designs the cross-modal time alignment mechanism in step S5. Through strategies such as frame-level alignment, interpolation alignment, and synchronization beacon calibration, the multimodal feature sequences achieve strict correspondence in the time dimension.

[0037] In step S6, since audio features, visual lip shape features, and noise features have different dimensions, statistical distributions, and information densities, they need to be mapped to a unified embedding space for subsequent fusion. The core objective of step S6 is to construct a "cross-modal semantically consistent embedding representation" so that different modalities are comparable and fusionable in the embedding space. First, an independent encoding network is built for each modality, configuring an audio encoder (CNN + Transformer), a visual encoder (3D CNN + LSTM), and a noise feature encoder (MLP or lightweight attention network). It should be noted that CNN stands for Convolutional Neural Network, and LSTM stands for Long Short-Term Memory. Each encoder outputs modal features of different dimensions, therefore a unified mapping is required, specifically: An audio coding network is constructed for the audio features using the audio encoder; a visual coding network is constructed for the visual lip shape features using the visual encoder; and a noise coding network is constructed for the environmental noise features using the noise feature encoder. Then, using a linear projection structure and a nonlinear activation structure, all the above modes are mapped to a d-dimensional unified space. The calculation formula is as follows: ; in, For modal primitive features, It is a linear projection matrix. To unify the embedding vector; Indicates the first Embedding Mapping Function corresponding to each modality. Indicates the first Bias Vector corresponding to each mode. This embodiment further optimizes the semantic alignment between different modal embeddings by using multimodal consistency loss (such as contrast loss), so that features such as audio, visual lip shape and noise in the same time slice can maintain geometric proximity in a unified space; In addition, this embodiment also designs a "modal compression and expansion mechanism"; for visual lip shape feature sequences with high dimensionality and a lot of redundancy, the system will automatically perform dimensionality reduction; while for noise features with low dimensionality, the system performs dimensionality expansion operation to enable them to carry more semantic information. Finally, the audio, visual lip shape, and noise modal embedding features output in step S6 will be used as input to the adaptive weighted fusion network in step S7 to achieve close cross-modal integration.

[0038] Specifically, in step S7, the adaptive weighted fusion network dynamically generates and allocates fusion weights for each modality based on the low-dimensional noise vector and the real-time calculated audio intelligibility index and lip-sync detection confidence. The fusion feature vector is a weighted sum of the embedded features of each modality. Step S7 is one of the core innovations of this embodiment. This step uses an adaptive weighting mechanism to dynamically adjust the fusion ratio of audio, visual, and noise modalities based on the noise environment, modal confidence, and feature quality, so that the recognition system can automatically select the optimal modal combination under different noise conditions. Traditional multimodal methods typically employ simple concatenation or fixed weighting strategies, but these cannot adapt to scenarios involving noise variations or fluctuations in the quality of the current modality. The adaptive weighting mechanism proposed in this embodiment can dynamically calculate the fusion weights based on the modal quality of the current frame. ; The current frame refers to the "multimodal frame under a unified time index" that has been time-aligned in step S5; The modality scoring network is driven by the low-dimensional noise vector and the detection confidence of each modality. For modal weights, To unify embedding features; This represents a modality index, used to distinguish different modality types; The summation index is used to normalize the calculation of all modalities involved in the fusion. This mechanism ensures that the weight of visual features is increased in high-noise environments, while the weight of audio features is increased in low-light or camera-occluded environments, forming a dynamic fusion system with environmental adaptability. The formula for calculating the fused output is: ; The fused vectors have multimodal complementarity and can maintain a high amount of information under various complex conditions such as noisy environments, long-distance speech scenarios, and low-light environments; In addition, this embodiment designs a "modal confidence estimation module", which calculates indicators such as audio signal clarity, lip detection stability, and noise intensity to assign a real-time confidence score to each modality, making the fusion more intelligent; Ultimately, the fused features output in step S7 will serve as the input to the acoustic model in the subsequent step S9, which is crucial for improving system performance.

[0039] Specifically, in step S8, the noise adaptive denoising network is a deep network with U-Net or a denoising autoencoder as its backbone. It uses the low-dimensional noise vector as a conditional input and dynamically adjusts the internal parameters of the network through the feature modulation layer to achieve adaptive denoising for specific noise types and intensities. In step S8 of this embodiment, a "noise adaptive denoising network" is further proposed. Its core idea is that different types of noise destroy speech features in different ways, and a denoising model with fixed parameters cannot achieve the best results. Therefore, it is necessary to adjust the parameters of the denoising network autonomously according to the noise vector. First, this embodiment constructs a deep denoising model based on the U-Net or the denoising autoencoder (DAE), and guides the deep network to perform parameter adjustment through the low-dimensional noise vector; the parameters of U-Net are... Noise embedding is Then, a hyperparameter-tuned network is used to generate the dynamic offset, and the calculation formula is: ; in, For lightweight parameter prediction networks, These are the adaptive denoising parameters under the current noise conditions. Through this mechanism, the denoising network can perform lightweight enhancement at high signal-to-noise ratios and strong noise suppression at extremely low signal-to-noise ratios, ensuring that the output speech always maintains a high signal-to-noise ratio. In addition, this embodiment also includes a "visual assistance enhancement module" that uses lip-shape dynamic information to infer the energy change trend of speech when the audio is severely damaged, thereby improving the noise reduction effect. Finally, the enhanced speech features (spectral or Mel features) output in step S8 will serve as the core input for acoustic modeling in step S9, enabling the system to maintain stable recognition performance even in extreme noise environments.

[0040] Specifically, in step S9, the end-to-end acoustic model is a Transformer or Conformer structure, and the encoder of the end-to-end acoustic model includes a cross-modal attention mechanism, so that during the acoustic modeling process, audio features are used to focus on the corresponding visual lip features. Based on the fused features output in step S7 and the enhanced speech features obtained in step S8, an end-to-end acoustic modeling module is designed in step S9 to map the fused multimodal information into language units (such as phonemes, characters, words, or sub-word units). Unlike traditional HMM-DNN speech recognition systems, this embodiment adopts end-to-end Transformer, Conformer, or LSTM-CTC structures to reduce intermediate steps, lower complexity, and greatly improve modeling capabilities. It should be noted that DNN stands for Deep Neural Network. First, this embodiment adopts a deep encoder structure and takes the fused vector sequence as input. The deep encoder includes a multi-head attention mechanism to capture long-term dependencies, enabling the model to obtain pronunciation patterns and semantic transition information from the temporal context. In speech, many syllables span multiple time frames, and the correspondence between visual lip shapes and audio has time-delay characteristics, so long-distance modeling capability is crucial. To improve the robustness of the model in non-stationary noise environments, this embodiment introduces a "cross-modal attention" structure, which encodes audio features, visual features and noise features separately, and then uses a shared attention mechanism to perform cross-modal interaction, so that the audio model can prioritize visual dynamics in frames affected by noise and avoid misjudgment. The output sequence of the end-to-end acoustic model is usually of a different length than the text sequence, therefore, it is necessary to use CTC (Connectionist Temporal Classification) or attention-based sequence-to-sequence decoding methods. For the case of using CTC, the loss function is calculated using the following formula: ; in, To fuse feature sequences, The target text label sequence; CTC enables the end-to-end acoustic model to handle inputs and outputs of different lengths by summing all valid alignment paths.

[0041] For the Transformer / Conformer structure, this embodiment also designs a "speech-visual consistency regularization term" to improve the stability of the model in multimodal collaborative learning by constraining the attention weights of the two modalities to be consistent in the pronunciation transition segment.

[0042] Using the above method, the acoustic model in this embodiment can maintain robust speech-to-text modeling capabilities in various complex scenarios such as noisy environments, low-light environments, fast speech rates, and occlusion conditions, providing highly reliable candidate sequences for language model decoding in the subsequent step S10.

[0043] Specifically, in step S10, the decoding process combines the probability sequence output by the end-to-end acoustic model with the prior probability provided by the language model, and uses the beam search algorithm for searching. Candidate paths with high confidence in the visual lip shape features are given a score reward, specifically including the following process: After completing the acoustic modeling, this embodiment performs text decoding in step S10, that is, mapping the probability sequence output by the acoustic model to the initial text sequence; since the sequence output by the acoustic model usually contains ambiguity, homophony interference and grammatical incompleteness, it is necessary to introduce a language model for global semantic constraints; This embodiment supports multiple types of language models, including N-gram statistical language models, RNN-LM, Transformer-LM, and pre-trained language models (such as GPT-like models); the language model is used to estimate the rationality of word sequences, thereby eliminating low-probability language segments during the decoding process; It should be noted that RNN stands for Recurrent Neural Network. During the decoding process, this embodiment uses the beam search algorithm for searching; the beam search algorithm retains several candidate paths with the highest scores at each step, making the search process computationally manageable; if the acoustic model outputs a probability of The language model gives the sentence probability as follows: Therefore, this embodiment uses the following joint scoring function: ; Where λ is the language model weight, used to balance the probabilities of acoustics and language; β is the length penalty coefficient; The sequence length; The candidate text sequence is the output sequence generated by the acoustic model during the decoding process, consisting of several discrete symbols (characters, words, or sub-word units). This joint scoring function formula enables the system to achieve a balance between "acoustic model confidence" and "linguistic rationality," thereby avoiding misjudgments caused by noise in the model. Furthermore, this embodiment incorporates a "multimodal consistency constraint" into the beam search algorithm to improve the score of sequences with high confidence in visual information among candidate paths. In this way, even when the audio is extremely damaged, the system can still generate stable text with the help of visual information (lip movement features), which is particularly suitable for scenarios with extremely complex noise, such as industrial environments and the interior of moving vehicles. In the decoding stage, this embodiment can also introduce a "context-adaptive language model" to automatically enhance specific terms and expressions based on the application domain, such as medical, legal, or specific corporate terms, so that the recognition results are more in line with the actual scenario. Finally, step S10 outputs a high-confidence initial text sequence as a preliminary version of the final speech recognition result, which will be further processed in natural language level formatting in step S11. In step S11, although the output sequence after text decoding has strong semantic rationality under the constraints of the language model, there may still be problems such as missing punctuation, capitalization errors, and inaccurate recognition of proper nouns. Therefore, this embodiment implements a complete text post-processing mechanism in step S11 to make the output result closer to the natural language text format. First, this embodiment designs a punctuation recovery model; since the end-to-end acoustic model usually does not generate punctuation marks, a deep sequence annotation model (such as BiLSTM-CRF or Transformer) is used to automatically add punctuation marks such as commas, periods, question marks, and pauses to the text; the end-to-end acoustic model recovers natural language writing habits by learning the semantic structure, word relationships, intonation patterns, and other information of the initial text sequence. Secondly, the post-processing output module includes a proper name recognition unit, which automatically identifies and corrects personal names, place names, organization names, and professional terms in the text. For example, "Zhang San" may be misidentified as "Changshan" due to the acoustic model. In this embodiment, it is automatically corrected by matching context features and entity dictionary. Subsequently, this embodiment performs text cleaning processing, including deleting redundant and repeated words, removing colloquial filler words, and correcting typos. For example, filler words such as "um," "ah," and "that" often appear in real speech, and this system can automatically determine whether they need to be removed based on the context. To support diverse application scenarios, this embodiment also includes a domain dictionary enhancement mechanism, enabling the system to achieve higher terminology recognition accuracy in specific industries. For example, in medical scenarios, terms such as "liver function" and "troponin" will be prioritized for recognition. Finally, step S11 outputs the final text after punctuation restoration, semantic enhancement, and domain correction, which is presented to the user as the final speech recognition text.

[0044] like Figure 2 As shown, this embodiment also provides a speech recognition system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it applies a speech recognition method based on multimodal fusion and adaptive noise modeling as described above. The speech recognition system also includes a multimodal data acquisition module, a speech preprocessing module, a visual lip shape feature extraction module, a noise modeling module, a multimodal alignment module, a unified embedding space mapping module, a fusion network module, a noise adaptive denoising module, an end-to-end acoustic modeling module, a decoding module, and a post-processing output module. The multimodal data acquisition module is used for synchronous acquisition of multimodal data; The speech preprocessing module is used to preprocess the audio stream; The visual lip shape feature extraction module is used to process the video lip shape stream; The noise modeling module is used to perform noise analysis, classification, and generate low-dimensional noise vectors that characterize the type and intensity of noise. The multimodal alignment module is used to perform cross-modal temporal alignment of acoustic feature sequences, visual lip shape feature sequences, and low-dimensional noise vectors to obtain temporally corresponding multimodal feature sequences. The unified embedding space mapping module is used to map the time-aligned modal features to the unified semantic embedding space; The fusion network module dynamically calculates and fuses the multimodal features in the unified embedding space mapping module through an adaptive weighted fusion network to obtain a fused feature vector; The noise adaptive denoising module uses a noise adaptive denoising network to enhance acoustic features or fused features to obtain denoised and enhanced features. The end-to-end acoustic modeling module takes the fused feature vector and / or the denoising and enhancement features as input, performs acoustic modeling, and outputs a probability sequence corresponding to the language unit. The decoding module decodes the probability sequence; The post-processing output module is used to perform punctuation restoration, proper noun correction and text cleaning on the initial text sequence obtained by the beam search algorithm, and output the final recognized text.

[0045] The aforementioned speech recognition method and system can operate stably in different computing environments, possessing high versatility and scalability, meeting the requirements of industrial-grade speech recognition systems. Recognition accuracy is significantly improved in low signal-to-noise ratio and complex environments. The speech recognition system can dynamically adjust feature fusion weights and denoising strategies based on environmental noise, enhancing recognition stability. The speech recognition system adopts a multi-modal design, reducing the need for manual parameter adjustment, exhibiting strong adaptability, and can be applied in multiple scenarios.

[0046] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition method based on multimodal fusion and adaptive noise modeling as described above.

[0047] First, each module in the speech recognition system described in this embodiment can be deployed and updated independently, so as to facilitate the expansion and maintenance of the entire system; Secondly, the speech recognition system can be deployed on GPU servers to support large-scale computing, or it can be deployed on low-power edge devices through model compression, quantization, pruning and other technologies. For example, in environments such as smart glasses, in-vehicle devices, robots, and human-computer interaction terminals, this embodiment can achieve real-time multimodal speech recognition.

[0048] Furthermore, all steps in this embodiment can be implemented by computer program instructions and stored in a computer-readable storage medium, such as ROM, RAM, hard disk, flash drive, etc. Users can invoke the functions of this embodiment by loading the instructions in the storage medium.

[0049] In addition, this embodiment also supports an online update mechanism, which allows the acoustic model, visual model, noise model and language model to evolve continuously with the scene; the system can automatically collect new data and complete incremental training of the model in the background to achieve long-term adaptation and maintain high-performance output.

[0050] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the present invention should also fall within the protection scope of the claims of the present invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the present invention.

Claims

1. A speech recognition method based on multimodal fusion and adaptive noise modeling, characterized in that, The method includes the following steps: Step S1: Perform synchronous acquisition of multimodal data and obtain synchronous multimodal data stream signals; the multimodal data stream signals include audio stream signals, video lip-sync stream signals, and environmental noise signals; Step S2: Perform speech preprocessing on the audio stream signal to extract multidimensional acoustic feature sequences; Step S3: Process the video lip flow signal to extract a visual lip feature sequence containing static shape, dynamic optical flow and three-dimensional geometric information; Step S4: Based on the environmental noise signal, perform noise analysis, classification, and generate a low-dimensional noise vector characterizing the noise type and intensity; Step S5: Perform cross-modal temporal alignment on the acoustic feature sequence, visual lip shape feature sequence, and low-dimensional noise vector to obtain the temporally corresponding multimodal feature sequence; Step S6: Map the temporally aligned modal features to a unified semantic embedding space; Step S7: Based on the low-dimensional noise vector and the real-time confidence of each modality feature, the multimodal features in the unified embedding space are dynamically calculated and fused through an adaptive weighted fusion network to obtain the fused feature vector; Step S8: Use the noise adaptive denoising network guided by the low-dimensional noise vector to enhance the acoustic features or fused features to obtain denoised and enhanced features; Step S9: Using the fused feature vector and / or the denoising enhancement feature as input, perform acoustic modeling through an end-to-end acoustic model, and output a probability sequence corresponding to the language unit; Step S10: Introduce a language model to decode the probability sequence and use a beam search algorithm to obtain the initial text sequence; Step S11: Perform post-processing on the initial text sequence, including punctuation restoration, proper noun correction, and text cleaning, to output the final speech recognition text.

2. The speech recognition method based on multimodal fusion and adaptive noise modeling according to claim 1, characterized in that, In step S1, a hardware clock synchronization protocol is used to provide a unified clock reference for all acquisition sensors, and software timestamp calibration and timing drift compensation are performed on devices that cannot be hardware synchronized in order to obtain the time-synchronized multimodal data stream signal. In step S2, pre-emphasis filtering is first used to enhance the high-frequency energy of the audio stream signal. The calculation expression is as follows: ; in, The pre-emphasis coefficient is n; n is the discrete-time index. This represents the sample value of the original speech signal at the discrete-time index n; This represents the sampled value of the original speech signal at the previous sampling time (n-1); This represents the speech signal output value obtained at time index n after the pre-emphasis filtering; In step S3, optical flow features are first calculated to extract the motion changes of the lips between frames, and a three-dimensional facial geometry restoration algorithm is introduced to map the 2D lip image onto a 3D mesh to obtain a three-dimensional motion vector; finally, the visual lip shape feature sequence is obtained.

3. The speech recognition method based on multimodal fusion and adaptive noise modeling according to claim 1, characterized in that, In step S4, noise signals are first extracted from the non-speech segments of the audio stream signal, and the spectral characteristics of the noise signals are calculated for noise type determination; wherein, the spectral characteristics include spectral quality characteristics, spectral attenuation rate characteristics, and spectral flatness characteristics; The spectral features are concatenated and a noise embedding vector of uniform dimension is generated through a linear mapping or a lightweight neural network encoder, as shown below: ； in, , These represent the mean vector and variance vector of the noise spectral characteristics in the time dimension, respectively; Represents the noise coding function; The signal-to-noise ratio of the noise signal is then calculated using the following estimation formula: ; in, The average power of the speech segment of the audio stream signal; The average power of the noise segment of the audio stream signal; Finally, the noise type and the signal-to-noise ratio information are mapped to the low-dimensional noise vector through a noise coding network.

4. The speech recognition method based on multimodal fusion and adaptive noise modeling according to claim 1, characterized in that, In step S5, the visual lip shape feature sequence is mapped onto the reference time axis using interpolation, based on the time axis of the acoustic feature sequence, so that it is aligned with the target timestamp. The alignment effect is further refined using a dynamic time warping algorithm, calculated as follows: ； in, The audio frames of the acoustic feature sequence Visual frames with the visual lip shape feature sequence Similarity; After processing in step S5, a multimodal feature sequence corresponding to a unified time is obtained; In step S6, an audio coding network is constructed for the audio features using an audio encoder; a visual coding network is constructed for the visual lip shape features using a visual encoder; and a noise coding network is constructed for the environmental noise features using a noise feature encoder. Then, using a linear projection structure and a nonlinear activation structure, all the above modes are mapped to a d-dimensional unified space. The calculation formula is as follows: ； in, For modal primitive features, It is a linear projection matrix. To unify the embedding vector; Indicates the first The embedding mapping function corresponding to each mode; Indicates the first The bias vector corresponding to each mode; Step S6 ultimately outputs three modal embedding features: audio, visual lip shape, and noise.

5. The speech recognition method based on multimodal fusion and adaptive noise modeling according to claim 4, characterized in that, In step S7, the adaptive weighted fusion network dynamically generates and assigns fusion weights for each modality based on the low-dimensional noise vector and the real-time calculated audio intelligibility index and lip-sync detection confidence. The fusion feature vector is a weighted sum of the embedded features of each modality. The fusion weights are dynamically calculated based on the modal quality of the current frame using an adaptive weighting mechanism. ； The current frame refers to the multimodal frame under the unified time index after time alignment has been completed in step S5; The modality scoring network is driven by the low-dimensional noise vector and the detection confidence of each modality. For modal weights, To unify embedding features; Indicates modal index; For summation index; The formula for calculating the fused output is: 。 6. A speech recognition method based on multimodal fusion and adaptive noise modeling according to claim 1 or 3, characterized in that, In step S8, the noise adaptive denoising network is a deep network with U-Net or denoising autoencoder as its backbone. It uses the low-dimensional noise vector as a conditional input and dynamically adjusts the internal parameters of the network through the feature modulation layer to achieve adaptive denoising for specific noise types and intensities. First, a deep denoising model is constructed based on the U-Net or a denoising autoencoder. The low-dimensional noise vector guides the deep network to perform parameter adjustments. The parameters of the U-Net are... Noise embedding is Then, a hyperparameter-tuned network is used to generate the dynamic offset, and the calculation formula is: ； in, For lightweight parameter prediction networks, These are the adaptive denoising parameters under the current noise conditions.

7. The speech recognition method based on multimodal fusion and adaptive noise modeling according to claim 1, characterized in that, In step S9, the end-to-end acoustic model is a Transformer or Conformer structure, and the encoder of the end-to-end acoustic model includes a cross-modal attention mechanism, so that during the acoustic modeling process, audio features are used to focus on the corresponding visual lip features.

8. The speech recognition method based on multimodal fusion and adaptive noise modeling according to claim 1, characterized in that, In step S10, the decoding process combines the probability sequence output by the end-to-end acoustic model with the prior probability provided by the language model, and uses the beam search algorithm for searching. Candidate paths with high confidence in the visual lip shape features are given a score reward, specifically including the following process: During the decoding process, the beam search algorithm is used for searching; if the acoustic model outputs a probability of... The language model gives the sentence probability as follows: Then the following joint scoring function is used: ； Where λ is the language model weight, used to balance the probabilities of acoustics and language; β is the length penalty coefficient; The sequence length; Represents a sequence of candidate texts; In step S11, punctuation is automatically added to the text using a deep sequence annotation model; the end-to-end acoustic model recovers language writing habits by learning the semantic structure, word relationships, and intonation patterns of the initial text sequence.

9. A speech recognition system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it applies a speech recognition method based on multimodal fusion and adaptive noise modeling as described in any one of claims 1 to 8; The speech recognition system also includes a multimodal data acquisition module, a speech preprocessing module, a visual lip shape feature extraction module, a noise modeling module, a multimodal alignment module, a unified embedding space mapping module, a fusion network module, a noise adaptive denoising module, an end-to-end acoustic modeling module, a decoding module, and a post-processing output module. The multimodal data acquisition module is used for synchronous acquisition of multimodal data; The speech preprocessing module is used to preprocess the audio stream; The visual lip shape feature extraction module is used to process the video lip shape stream; The noise modeling module is used to perform noise analysis, classification, and generate low-dimensional noise vectors that characterize the type and intensity of noise. The multimodal alignment module is used to perform cross-modal temporal alignment of acoustic feature sequences, visual lip shape feature sequences, and low-dimensional noise vectors to obtain temporally corresponding multimodal feature sequences. The unified embedding space mapping module is used to map the time-aligned modal features to the unified semantic embedding space; The fusion network module dynamically calculates and fuses the multimodal features in the unified embedding space mapping module through an adaptive weighted fusion network to obtain a fused feature vector; The noise adaptive denoising module uses a noise adaptive denoising network to enhance acoustic features or fused features to obtain denoised and enhanced features. The end-to-end acoustic modeling module takes the fused feature vector and / or the denoising and enhancement features as input, performs acoustic modeling, and outputs a probability sequence corresponding to the language unit. The decoding module decodes the probability sequence; The post-processing output module is used to perform punctuation restoration, proper noun correction and text cleaning on the initial text sequence obtained by the beam search algorithm, and output the final recognized text.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements a method based on multimodal fusion and adaptive noise modeling as described in any one of claims 1 to 8.

Images