Speech Recognition Methods and Systems Based on Multimodal Distortion Compensation

Abstract

This application relates to a speech recognition method and system based on multimodal distortion compensation. Regarding the reliability of recognition in multimodal distortion environments, the method relies on a weakly supervised dynamic contribution learning network and a collaborative compensation update mechanism to solve the problems of insufficient fusion flexibility and modal value imbalance in traditional systems. The dynamic contribution learning network uses the prediction loss of each modality as a weak supervision signal, enabling real-time evaluation of sample-level modal reliability and dynamic adjustment of modal weights. This ensures that reliable modalities dominate the fusion decision, overcoming the insufficient adaptability of fixed-weighting schemes in dynamic distortion scenarios. The collaborative compensation update mechanism allocates sample weights according to the CTC loss of each modality, guiding audio modalities, video modalities, and cross-modal mutual attention to difficult-to-recognize samples, achieving cross-modal knowledge transfer, avoiding convergence bias caused by single-modal distortion, and significantly improving the recognition stability of the system in multimodal distortion environments.

Description

Technical Field

[0001] This invention belongs to the field of multimodal speech recognition technology, and relates to a speech recognition method and system based on multimodal distortion compensation. Background Technology

[0002] The rapid development of voice interaction technology in various fields such as in-vehicle systems and remote conferencing has gradually exposed the significant limitations of single-audio-modal voice recognition technology. In complex acoustic environments, strong noise interference (such as noise from demolition equipment, engine sounds, mechanical roars, etc.) can lead to a significant decrease in voice recognition accuracy, with the false recognition rate even exceeding 50% in some scenarios, seriously affecting the reliable transmission of voice commands and the stability of human-machine collaboration.

[0003] To overcome the aforementioned bottlenecks, Audio-Visual Speech Recognition (AVSR) technology has emerged. This technology leverages both audio and video modal information, taking full advantage of the fact that visual cues such as lip movements are unaffected by the acoustic environment. It compensates for speech signal degradation with visual features, thus achieving a complementary recognition mechanism of "audio completing vision and vision verifying audio." This approach effectively improves the robustness of speech recognition in complex scenarios and has become an important research direction in the field of voice interaction in recent years.

[0004] Current mainstream AVSR methods mainly fall into three categories: feature concatenation fusion, fixed-weight fusion, and Transformer-based attention fusion. The first two methods typically fuse audio and video information through early feature concatenation or later fixed weighting. While structurally simple, they lack fusion flexibility and struggle to adapt to modal quality differences between samples. When audio is damaged, the system cannot dynamically adjust modal weights, often resulting in a significant drop in recognition performance. While Transformer-based attention fusion can capture cross-modal dependencies to some extent, its attention mechanism lacks the ability to distinguish intra-sample modal reliability, easily misclassifying noisy features as valid signals, thus affecting recognition accuracy and stability.

[0005] Furthermore, most existing AVSR systems exhibit an "audio-dominated" bias, treating the visual modality merely as an auxiliary signal when audio is distorted, failing to construct a fusion mechanism for equal interaction between the two modalities. When audio completely fails, the system cannot fully rely on visual information to independently complete recognition, leading to a sharp drop in recognition performance in extreme environments. For example, in industrial production sites, strong mechanical noise can completely drown out audio signals, while the visual modality, due to its low weight, fails to play an effective role and cannot recognize key commands through lip movements, potentially causing operational errors or safety risks. The root cause of this bias lies in the lack of a real-time evaluation mechanism for modal reliability. The system cannot dynamically adjust its feature fusion strategy based on the current environment and modal quality, thus weakening the potential advantages of multimodal complementarity.

[0006] In summary, existing audiovisual fusion speech recognition technologies generally suffer from rigid fusion strategies, modality-dependent biases, and a lack of real-time reliability assessment under multi-scenario and multimodal dynamic distortion conditions, making it difficult to meet the demand for highly robust speech recognition in complex environments such as emergency rescue. This technical deficiency directly limits the practical application of multimodal speech recognition, which is precisely the research basis and motivation for improvement proposed in this invention. Summary of the Invention

[0007] To address the issue of insufficient audiovisual fusion speech recognition performance in various complex application scenarios such as emergency rescue and in-vehicle intelligent interaction due to multimodal distortion (including audio noise interference, video occlusion and abnormal lighting, and modal synchronization failure), this invention proposes a speech recognition method and system based on multimodal distortion compensation. This method enables real-time evaluation and adaptive fusion of audio and visual modal reliability, thereby significantly improving the recognition accuracy and robustness of the system in complex acoustic and visual environments. It ensures the accurate transmission of voice commands and the continuous stability of business processes, and reduces security risks and task delays caused by recognition errors or information loss.

[0008] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0009] On the one hand, a speech recognition method based on multimodal distortion compensation is provided, the method comprising the following steps: The acquired audio signal and the corresponding facial video signal are preprocessed to obtain the audio sequence and the corresponding facial image sequence.

[0010] A trained speech recognition model is used to process audio sequences and corresponding facial image sequences to obtain speech recognition results. The speech recognition model includes a feature extraction module, a multimodal learner, a dynamic contribution evaluation module, a collaborative fusion module, and a decoder. The feature extraction module extracts audio modal features and video modal features from the audio sequences and corresponding facial image sequences. The multimodal learner uses a single-modal learner to model the audio and video modal features separately, preserving the semantic details and scene features of each modality. A cross-modal learner captures the temporal and spatial correlation between the speech signal and lip movements from the output features of the single-modal learner. The dynamic contribution evaluation module obtains corresponding prediction results based on the single-modal and cross-modal learning results, and uses the prediction loss of each modality as a weak supervision signal to convert the prediction loss into modality confidence. The collaborative fusion module fuses the prediction results of each modality learner based on the modality confidence to obtain fused features. The decoder decodes the fused features to obtain the speech recognition results.

[0011] On the other hand, a speech recognition system based on multimodal distortion compensation is also provided, the device comprising: The multimodal signal preprocessing module is used to preprocess the acquired audio signal and the corresponding facial video signal to obtain the audio sequence and the corresponding facial image sequence.

[0012] The speech recognition module based on multimodal distortion compensation processes audio sequences and corresponding facial image sequences using a trained speech recognition model to obtain speech recognition results. The speech recognition model includes a feature extraction module, a multimodal learner, a dynamic contribution evaluation module, a collaborative fusion module, and a decoder. The feature extraction module extracts audio and video modal features from the audio and corresponding facial image sequences. The multimodal learner uses a single-modal learner to model the audio and video modal features separately, preserving the semantic details and scene features of each modality. A cross-modal learner captures the temporal and spatial correlation between the speech signal and lip movements from the output features of the single-modal learner. The dynamic contribution evaluation module obtains corresponding prediction results based on the single-modal and cross-modal learning results, and uses the prediction loss of each modality as a weak supervision signal to convert the prediction loss into modal confidence. The collaborative fusion module fuses the prediction results of each modality learner based on the modal confidence to obtain fused features. The decoder decodes the fused features to obtain the speech recognition result.

[0013] One of the above technical solutions has the following advantages and beneficial effects: The aforementioned speech recognition method and system based on multimodal distortion compensation addresses the issues of insufficient fusion flexibility and modal value imbalance in traditional systems by relying on a weakly supervised dynamic contribution learning network and a collaborative compensation update mechanism to improve recognition reliability in multimodal distortion environments. The dynamic contribution learning network uses the prediction loss of each modality as a weak supervision signal, enabling real-time evaluation of sample-level modal reliability and dynamic adjustment of modal weights. This ensures that reliable modalities dominate the fusion decision, overcoming the insufficient adaptability of fixed-weighting schemes in dynamic distortion scenarios. The collaborative compensation update mechanism allocates sample weights based on the CTC loss of each modality, guiding audio modalities, video modalities, and cross-modal interactions to focus on difficult-to-recognize samples. This achieves cross-modal knowledge transfer, avoids convergence bias caused by single-modal distortion, and significantly improves the system's recognition stability in multimodal distortion environments. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating a speech recognition method based on multimodal distortion compensation in one embodiment. Figure 2 Here is a flowchart of a speech recognition model in one embodiment; Figure 3 This is a schematic diagram of a single-modal learner structure in one embodiment; Figure 4 This is a schematic diagram of the DCLN network structure in one embodiment; Figure 5 This is a DCLN training optimization process in one embodiment; Figure 6 This is a schematic diagram of the data acquisition and processing flow in one embodiment. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0018] It should be noted that, in this document, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The presentation of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments. The term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, including such combinations.

[0019] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0020] In one embodiment, such as Figure 1 As shown, a speech recognition method based on multimodal distortion compensation is provided, which may include the following processing steps 1 and 2: Step 1: Preprocess the acquired audio signal and the corresponding facial video signal to obtain the audio sequence and the corresponding facial image sequence.

[0021] Specifically, audio and video acquisition devices are used to collect audio and video data covering multimodal distortion features. The dataset needs to cover typical interference conditions, including audio noise, video occlusion, abnormal lighting, and modal synchronization distortion, and also include voice commands from various scenarios such as emergency rescue, vehicle use, industry, and meetings, ensuring the representativeness and diversity of the sample content. Strict audio-visual synchronization is maintained during acquisition, and the recorded data undergoes preliminary screening to remove silent, non-lip-gesture, and invalid segments. Based on the duration characteristics of the voice commands (approximately 1–5 seconds), valid samples are divided into fixed-length subsequences (approximately 3 seconds each), and audio and lip-gesture video information are bound to each sample. This step ensures consistent input sample lengths, facilitating model training and testing, while also meeting the terminal's computing power constraints.

[0022] Data preprocessing: FFmpeg was used to separate the segmented fixed-length subsequences into audio and video sequences, resulting in silent video and audio sequences. For the silent video, image frames were extracted at 30fps, and each frame was processed using the OpenFace open-source face detection tool to identify and extract the face regions in each frame, obtaining the video clip of the target speaker's face when speaking. The audio sequences underwent denoising, filtering, and time alignment.

[0023] Face detection can also use YOLOv5, MTCNN, RetinaFace, etc. as alternatives.

[0024] Step 2: The trained speech recognition model is used to process the audio sequence and the corresponding facial image sequence to obtain the speech recognition result. The speech recognition model includes: a feature extraction module, a multimodal learner, a dynamic contribution evaluation module, a collaborative fusion module, and a decoder. The feature extraction module is used to extract audio modal features and video modal features from the audio sequence and the corresponding facial image sequence. The multimodal learner is used to model the audio modal features and video modal features separately using a single modal learner, preserving the semantic details and scene features of each modality. The cross-modal learner captures the temporal and spatial correlation between the speech signal and lip movements from the output features of the single modal learner. The dynamic contribution evaluation module is used to obtain the corresponding prediction results based on the single modal learning results and the cross-modal learning results, and uses the prediction loss of each modality as a weak supervision signal to convert the prediction loss into modal confidence. The collaborative fusion module is used to fuse the prediction results of each modal learner based on the modal confidence to obtain fused features. The decoder is used to decode the fused features to obtain the speech recognition result.

[0025] Specifically, to overcome the shortcomings of existing technologies in terms of fusion flexibility and modal balance, this invention proposes an audiovisual fusion architecture with a dynamic contribution evaluation module. This architecture uses the model's own prediction loss as a weak supervision signal, eliminating the need for an additional distortion estimation module and quantifying the reliability of each modality at the sample level. The system can automatically adjust the fusion ratio of audio and video features according to the modal state in different scenarios. For example, when strong noise or demolition sounds cause audio distortion at an emergency rescue scene, the system automatically determines a decrease in audio reliability based on the loss difference and dynamically increases the visual modality weights to maintain recognition stability. This mechanism effectively overcomes the performance collapse limitation of fixed-weighting schemes under simultaneous dual-modal distortion, achieving true dynamic adaptive fusion.

[0026] In terms of algorithmic structure, this invention constructs a dual-track learning architecture consisting of a Unimodal Learner (UML) and a Cross-modal Learner (CML). The Unimodal Learner models features of audio and video data separately, preserving the semantic details and scene features of each modality; the Cross-modal Learner, on this basis, captures the temporal and spatial correlation between speech signals and lip movements, strengthening the complementary expression of cross-modal information. The system achieves weakly supervised evaluation of modality reliability through a Dynamic Contribution Learning Network (DCLN), transforming prediction loss into modality confidence and realizing sample-level adaptive weighting; simultaneously, a Collaborative Compensatory Update (CCU) mechanism is introduced to guide complementary optimization among different learners through error-driven sample reweighting, enabling audio and visual modalities to mutually correct erroneous predictions and enhance overall recognition stability and generalization performance.

[0027] Through the above innovative design, this invention can maintain a high level of recognition accuracy and robustness under various complex acoustic and visual conditions, and overcomes key problems in the prior art such as rigid fusion, modality dependency bias and lack of reliability assessment, providing reliable technical support for the practical application of multimodal speech recognition systems in emergency response, vehicle, industrial and remote interaction fields.

[0028] The flowchart of the speech recognition model is as follows Figure 2 As shown.

[0029] The aforementioned speech recognition method based on multimodal distortion compensation addresses the issues of insufficient fusion flexibility and modal value imbalance in traditional systems by relying on a weakly supervised dynamic contribution learning network and a collaborative compensation update mechanism to improve recognition reliability in multimodal distortion environments. The dynamic contribution learning network uses the prediction loss of each modality as a weak supervision signal, enabling real-time evaluation of sample-level modal reliability and dynamic adjustment of modal weights. This ensures that reliable modalities dominate the fusion decision, overcoming the insufficient adaptability of fixed-weighting schemes in dynamic distortion scenarios. The collaborative compensation update mechanism allocates sample weights based on the CTC loss of each modality, guiding audio modalities, video modalities, and cross-modal interactions to focus on difficult-to-recognize samples. This achieves cross-modal knowledge transfer, avoids convergence bias caused by single-modal distortion, and significantly improves the system's recognition stability in multimodal distortion environments.

[0030] This method is widely applicable to various scenarios, including emergency rescue, in-vehicle intelligent interaction, industrial production scheduling, and remote conferencing / online education. Addressing common issues in these scenarios such as audio noise interference, video occlusion / blurring, and modal synchronization distortion, it achieves accurate fusion and distortion compensation of multimodal information through dynamic contribution evaluation and collaborative compensation mechanisms. This ensures real-time and accurate recognition of voice commands, meeting the needs of different fields for highly robust voice interaction. Specific application scenarios include: Emergency rescue scenarios: Addressing the challenges of strong audio interference and video feature extraction in extreme environments, ensuring the transmission of critical rescue commands, and improving collaborative efficiency. For example, during earthquake rubble search and rescue, it can reduce the impact of strong noise from hydraulic demolition equipment on audio recognition, while overcoming video blurring caused by dense smoke, ensuring the accurate transmission of commands such as "life signal location"; in building fire interior attack scenarios, it can overcome the limitation of dust and gas masks covering the lips, preventing the loss of critical commands such as "transporting tourniquets".

[0031] In-vehicle intelligent interaction scenarios: This addresses audio and video distortion issues in the in-vehicle environment, reduces the misrecognition rate of voice commands during driving, and ensures driving safety. For example, at high speeds, it can mitigate the interference of wind noise from the windows on audio, while also alleviating video overexposure caused by backlighting, preventing commands like "navigate to a building" from being misinterpreted as other commands. In congested traffic, it can reduce interference from passenger conversations and the impact of the steering wheel obscuring the lips, ensuring accurate recognition of commands such as "adjust the air conditioning temperature."

[0032] Industrial production scheduling scenarios: Adapting to workshop machinery noise and visual obstructions to ensure accurate recognition of scheduling instructions and avoid equipment malfunctions and personnel misoperation. For example, when machinery is running, it can filter out the interference of high-decibel noise in the workshop on audio, while solving the problem of safety helmets covering the lips, ensuring that "stop for maintenance" instructions are accurately transmitted; when pneumatic tools are in operation, it can reduce the impact of tool noise, overcome the problem of camera blurring caused by dust, and prevent misinterpretation of instructions such as "adjust equipment parameters".

[0033] In remote conferencing / online education scenarios: Addressing environmental interference and network-induced modal distortion improves transcription accuracy and optimizes the interactive experience. For example, during home meetings, it can mitigate audio interference from keyboard clicks and alleviate video frame loss caused by network lag, ensuring accurate transcription of content such as "project deadline confirmation." During online classes, it can reduce the impact of neighbor noise, resolve lip feature blurring caused by backlighting, and prevent the omission of information such as "knowledge point markings."

[0034] In one embodiment, step 2 specifically includes: Step 201: Input the audio sequence and the corresponding facial image sequence into the feature extraction module to obtain audio modal features and video modal features.

[0035] Step 202: Input the audio modal features and video modal features into the multimodal learner to obtain the semantic details and scene features of the corresponding modal and the temporal and spatial correlation features between the audio signal and lip movements.

[0036] Specifically, a dual-track learning and dynamic compensation network integrating "single-modal + cross-modal" was constructed to achieve deep synergy between modality learning, reliability assessment, and cross-modal complementarity. The dual-track learning layer uses a Conformer encoder to construct audio and video UML, accurately captures single-modal features through Inter-CTC intermediate supervision, and utilizes CML feature concatenation and nonlinear mapping to mine cross-modal correlations without the need for pre-suppressing damaged modalities.

[0037] Step 203: Input the temporal and spatial correlation features, semantic details and scene features of the audio modality and video module into the dynamic contribution evaluation module to obtain the prediction results across modalities and each single modality and the corresponding final contribution weight.

[0038] Specifically, the dynamic contribution evaluation layer relies on the weakly supervised DCLN to generate sample-level modal contribution weights using the predicted loss as a weakly supervised signal. After normalization, reliable modality-driven decision-making is achieved. The collaborative compensation layer assigns sample weights based on the CTC loss of each modality, guiding different learners to focus on samples that are difficult for each other to identify, thus promoting cross-modal knowledge transfer. This network, through hierarchical collaboration of "dual-track learning—dynamic evaluation—collaborative compensation," solves the problems of modal information loss, poor weight adaptability, and insufficient cross-modal complementarity in traditional networks, forming the core protection point at the algorithm architecture level.

[0039] Step 204: Input the prediction results of audio modality, video modality and cross-modality and the corresponding final contribution weights into the collaborative fusion module for weighted summation to obtain the fusion features.

[0040] Step 205: Input the fused features into the decoder to obtain the speech recognition result.

[0041] In one embodiment, the feature extraction module includes an audio feature extraction module and a video feature extraction module; step 201 includes: inputting an audio sequence into the audio feature extraction module, converting the audio sequence into a Mel spectrogram, processing the Mel spectrogram using a two-dimensional convolutional layer and a linear transformation layer to obtain audio module features; inputting a facial image sequence into the video feature extraction module, using a three-dimensional convolutional neural network to extract temporal and spatial information from the facial image sequence, processing the extracted temporal and spatial information using a ResNet network, and performing global average pooling on the processing results to obtain video modal features.

[0042] Specifically, based on the preprocessed facial image sequence, a combination of a 3D Convolutional Neural Network (3D CNN) and a ResNet network is used for spatiotemporal feature extraction. The image sequence first passes through a 3D convolutional layer to simultaneously capture temporal and spatial information from the video sequence. The convolutional kernel size is 5×7×7, with a temporal dimension of 5 and a spatial dimension of 7×7. Subsequently, the features are input into a 2D residual network, ResNet-18, which consists of multiple residual blocks. Each residual block contains convolutional layers and batch normalization layers to enhance feature stability and expressive power. Finally, global average pooling is applied at the end of the ResNet-18 to reduce the spatial dimension of the features, resulting in a fixed-length feature vector used to represent the global spatiotemporal features of the video sequence.

[0043] The preprocessed audio signal (i.e., the audio sequence) is converted into a Mel-spectrum. This process is achieved through a Short-Time Fourier Transform (STFT) with a window length of 20 milliseconds and a frame shift of 10 milliseconds. An 80-dimensional Mel-scale Log Filter Bank is then applied to transform the STFT results to simulate the auditory response characteristics of the human ear to different frequencies. The resulting Mel-spectrum is processed by a two-dimensional convolutional layer (Conv2d) to extract local time-frequency features and generate feature frame signals with a time resolution of 20 milliseconds. Subsequently, a linear transformation layer maps the features to a high-dimensional space to enhance the separability and representational power of the features, providing input for the subsequent modality learner.

[0044] In one embodiment, the multimodal learner includes: an audio single-modal learner, a video single-modal learner, and a cross-modal learner; the audio single-modal learner consists of three Conformer encoders, one Inter-CTC residual module, two Conformer encoders, and two Inter-CTC residual modules connected in sequence; the audio single-modal learner and the video single-modal learner have the same structure; step 202 includes: inputting audio modal features into the audio single-modal learner to obtain semantic details and scene features of the audio modality; inputting video modal features into the video single-modal learner to obtain semantic details and scene features of the video modality; inputting the semantic details and scene features of the audio modality and the semantic details and scene features of the video modality into the cross-modal learner to obtain the temporal and spatial correlation features between the audio signal and lip movements.

[0045] Specifically, to fully utilize the multimodal information from audio and video and improve the robustness of audiovisual speech recognition in complex distortion scenarios, this module integrates an audio unimodal learner, a video unimodal learner, and a cross-modal learner. Through a collaborative mechanism of "unimodal feature preservation and cross-modal dependency modeling," it effectively reduces the interference of distorted modalities on the recognition results and suppresses the risk of model overfitting. The audio unimodal learner and the video unimodal learner are constructed independently for the audio and video modalities, respectively, with the network structure as follows: Figure 3 As shown, both methods employ a Conformer encoder, combining the local feature modeling capability of convolutional neural networks with the long-range dependency modeling advantage of Transformers to encode their respective modal features with high precision. An Inter-CTC intermediate layer is introduced during the encoding process to alleviate the independence limitations of the traditional CTC assumption and reduce the risk of keyword omission under distortion conditions. Finally, a single-modal prediction result is output via a linear layer.

[0046] A cross-modal learner is used to capture cross-modal interaction relationships. The network first receives feature outputs from audio and video unimodal learners, concatenates them along the feature dimension to generate an initial cross-modal feature vector; subsequently, it undergoes two linear transformations and Tanh nonlinear activation function mapping to enhance the expressive power of the cross-modal features, finally outputting the fused cross-modal prediction result. In this process, the system does not pre-suppress the feature signals of the damaged modality, but instead delegates the modality reliability assessment to the subsequent DCLN module, thereby ensuring that it can fully utilize its recognition function when any modality is available. The processing procedure is as follows:

[0047]

[0048]

[0049]

[0050] in a 'v' represents the audio modality, and 'v' represents the video modality. , These are audio and video features, , , These are the prediction results from the audio unimodal learner, the video unimodal learner, and the cross-modal learner, respectively.

[0051] In one embodiment, the dynamic contribution evaluation module includes: three prediction networks and three dynamic contribution learning networks; the dynamic contribution learning networks are used to estimate the reliability of each modality sample in real time using the prediction loss as a weak supervision signal; step 203 includes: inputting the temporal and spatial correlation features, the semantic details and scene features of the audio modality and the video modality into the corresponding prediction networks respectively to obtain cross-modal prediction results, audio modal prediction results, and pre-video modal prediction results; inputting the cross-modal prediction results, audio modal prediction results, and pre-video modal prediction results into the corresponding dynamic contribution learning networks respectively to obtain the final contribution weights of the cross-modal, audio modal, and video modal.

[0052] In one embodiment, the dynamic contribution learning network includes three sequentially connected fully connected modules and a Sigmoid activation function; the fully connected module includes a fully connected layer, a ReLU activation function, a batch normalization layer, and a Dropout layer.

[0053] Specifically, to address the problem that traditional fusion strategies cannot adapt to sample-level modality distortion, this module uses a weakly supervised DCLN as its core, with the network structure as follows: Figure 4 As shown. Each modality learner branch is equipped with a dedicated DCLN, which consists of three fully connected layers. The hidden layers use ReLU activation, combined with batch normalization and dropout mechanisms, and finally output the initial contribution probability in the [0,1] interval through Sigmoid activation. The specific processing flow is as follows: a. Initial contribution probability generation: The prediction results of the audio single-modal learner, video single-modal learner and cross-modal learner are input into the corresponding DCLN respectively. The initial contribution probability of each modality is generated through feature transformation and Sigmoid activation, which initially reflects the reliability of the sample-level modality.

[0054]

[0055] b. Contribution score normalization: using temperature control (temperature parameter) The Softmax function normalizes the initial contribution probabilities to obtain the final contribution weights for each mode. This makes the total weights equal to 1, while enhancing the distinguishability of the differences in contributions between different modes.

[0056]

[0057] c. Weakly supervised signal generation and DCLN optimization: such as Figure 5 As shown, the CTC loss of each modality learner is first calculated. The lower the loss, the more reliable the modality prediction; subsequently, it is converted into pseudo-labels. To reflect modal confidence.

[0058] In one embodiment, step 205 includes: transforming the fused features through a linear layer and then processing them through a Softmax layer to obtain the recognition probability of the Chinese character unit; inputting the recognition probability into the CTC classifier to obtain the classification result of the Chinese character sequence.

[0059] Specifically, the CTC classifier in the decoder can be replaced with an attention-based Seq2Seq structure to achieve the same functionality.

[0060] In one embodiment, a custom-designed headset that simultaneously supports audio and video dual-modal synchronous acquisition is used to acquire audio signals and corresponding facial video signals. The custom-designed headset is obtained by integrating a high-sensitivity active noise reduction module at the audio acquisition end of the headset and configuring a miniature anti-interference camera at the video acquisition end to focus on the core area of ​​the lips.

[0061] Specifically, this invention also includes a scene-adaptive multimodal acquisition device. Through hardware and software co-optimization, it improves the acquisition quality and modal balance of audio and video data from the source. For different application environments, the system can be configured with miniature cameras and high-sensitivity noise-reducing microphone arrays to adapt to typical distortion scenarios such as dust mask obstruction, engine vibration, and industrial noise. This integrated design enables the system to maintain high-quality input even in complex external environments, ensuring stable output of algorithm performance from the data source level.

[0062] To address the issue of poor audio and video data acquisition quality in multimodal distortion scenarios, the headphones were customized and modified, such as... Figure 6 As shown: The audio acquisition end integrates a high-sensitivity active noise reduction module, which can accurately filter strong noise interference and retain clear voice signals; the video acquisition end is equipped with a miniature anti-interference camera that focuses on the core lip area, possessing low-light adaptation and anti-blur capabilities to avoid loss of lip features due to occlusion or abnormal lighting. The device design balances portability and stability, with a weight controlled to ≤150g to ensure adaptability to various usage scenarios. It also supports simultaneous audio and video dual-modal acquisition; when one modality is interfered with, the other modality can serve as core auxiliary information, ensuring the integrity and effectiveness of data acquisition.

[0063] To address distortion issues caused by audio noise, video occlusion, and abnormal lighting in various scenarios, an integrated headphone acquisition device combining a high-sensitivity noise-reducing audio module and an anti-interference lip camera was designed. The audio end effectively resists strong noise interference, while the video end focuses on the core lip area and adapts to low-light environments, while maintaining portability to meet the needs of multiple application scenarios. The preprocessing stage includes audio-visual separation, lip key area cropping (removing occlusion and unifying resolution), and multi-dimensional distortion simulation (adding noise and simulating video blur), reducing the impact of distortion from the data source. This solves the problems of poor distortion tolerance, low data quality, and weak adaptability to preprocessing scenarios in traditional acquisition devices, forming the core protection points at both the hardware and data processing levels.

[0064] Regarding its anti-interference capabilities in complex scenarios, the system has undergone targeted optimization in the data acquisition and preprocessing stages. The customized acquisition equipment integrates a high-sensitivity active noise-canceling frequency module and an anti-interference lip camera, reducing the impact of environmental factors such as strong noise, obstruction, and low light on data acquisition. In the preprocessing stage, operations such as audiovisual separation, key lip region extraction, and distortion simulation improve the quality of multimodal data and enhance the model's adaptability to different distortion scenarios. The system does not require parameter adjustments for a single scenario and is applicable to various application scenarios such as emergency rescue, vehicle-mounted systems, industrial applications, and remote conferencing, solving the problem of traditional systems failing in multiple scenarios.

[0065] In terms of technology implementation and practicality, the system adopts an integrated design of "software algorithm + hardware adaptation," balancing performance and deployability. Lightweight algorithms (such as the Conformer encoder and CTC decoder) ensure real-time voice command recognition, avoiding the high latency issues of traditional complex models. Customized acquisition equipment balances portability and data quality, reducing distortion from the source, alleviating the pressure on algorithm compensation, and eliminating the need for additional complex peripherals, thus lowering the deployment threshold. The overall solution achieves reliable and easily deployable voice interaction in multimodal distortion scenarios, providing a feasible solution for practical applications.

[0066] Through a collaborative design encompassing "customized acquisition hardware (lightweight and highly distortion-resistant) + multimodal collaborative network (DCLN+CCU) + efficient CTC decoding (automatic timing alignment)," end-to-end distortion compensation is achieved from data acquisition, feature learning, fusion decision-making to decoding output. At the hardware level, high-quality acquisition of dual-modal data is ensured across multiple scenarios; at the algorithm level, dynamic distortion compensation is achieved through network structure innovation; and at the decoding level, it adapts to different speech rate instructions.

[0067] The speech recognition method based on multimodal distortion compensation achieves significant technical results by addressing key issues in speech recognition under multimodal distortion scenarios through software and hardware co-design and core algorithm innovation.

[0068] In one embodiment, the process of constructing the total loss function during the training of a speech recognition model based on multimodal distortion compensation includes: firstly, calculating the CTC loss of each modality learner. Transform CTC loss into pseudo-labels To reflect modal confidence;

[0069]

[0070] Where y represents the actual transcribed text. This represents the predicted text for each modality. These are pseudo-tags; For calibration parameters, For the CTC loss of each modality learner, Utilizing weak supervision loss As an optimization objective, the parameters of the dynamically contributing learning network are updated through backpropagation; the weakly supervised loss is:

[0071] in, The weights predicted by DCLN.

[0072] After the feature extraction module parameters stabilize, the feature extraction module is frozen; based on the CTC loss of each modality learner... Each modality learner is assigned sample weights, which are positively correlated with the prediction errors of other learners for that sample; UML-A sample weights. Determined by the minimum loss principle of UML-V and CML, this approach focuses on samples that are difficult for video modality and cross-modality learners to recognize. UML-V sample weights Determined by the minimum loss of UML-A and CML, this approach focuses on samples that are difficult for audio modal and cross-modal learners to recognize. CML sample weights Determined by the minimum loss of UML-V and UML-A, the focus is on samples that are difficult for video and audio modal learners to recognize. The sample weights are obtained by performing Softmax normalization. And build a collaborative compensation update loss This is used to optimize the learners for each modality, making them pay more attention to difficult cross-modality samples during training; specifically:

[0073]

[0074] Using CTC loss function This enables the model to automatically learn the correspondence between the fused sequence and the Chinese character sequence; where the CTC loss function... The expression is:

[0075] in, Given an input sequence x, the probability of outputting a Chinese character sequence label y is calculated. The summation is performed on all possible paths, including those with inserted whitespace labels.

[0076] The total loss of the speech recognition model is determined based on the weak supervision loss, collaborative compensation update loss, and CTC loss function; the expression for the total loss of the speech recognition model is:

[0077] in For the total loss, For the weight size, and .

[0078] Specifically, the training sample preparation process includes: using FFmpeg to separate the segmented fixed-length subsequences into audio and video sequences, obtaining silent video and audio sequences; for the silent video, image frames are extracted at a frame rate of 30fps, and each frame is processed using the OpenFace open-source face detection tool to identify and extract the face regions in each frame, obtaining the video clip of the target speaker's face when speaking. For the audio sequence, mature speech transcription tools, such as iFlytek's speech service, are used to transcribe the audio content into text, and manual verification is performed to ensure the accuracy of the transcription results, ultimately obtaining the Chinese character tag sequence. Given that the recognition accuracy of this invention reaches the Chinese character level, there is no need to perform further word segmentation processing on the transcribed text. In addition, some silent video and audio data are subjected to interference such as occlusion and outdoor noise to simulate the multimodal data distortion caused by harsh environments in real command and control, and used as training and testing data.

[0079] During training, weakly supervised loss is utilized. As an optimization objective, the DCLN parameters are updated via backpropagation. After the parameters of the audio and video front-end feature extraction network stabilize, they are frozen to ensure the reliability of pseudo-label generation and achieve accurate estimation of modality contribution without additional annotation.

[0080] Collaborative Fusion Module: Based on Normalized Modal Contribution Weights Prediction results for each modality learner Weighted summation is performed to obtain the final identification and prediction result P. This mechanism achieves dynamic fusion at the sample level, enabling reliable modalities to dominate the decision-making process, thereby improving the identification accuracy in distorted scenarios.

[0081] To address the issues of modality learners easily converging to biased representations and failing to fully utilize cross-modal complementary information, an error-driven sample reweighting strategy is introduced. This strategy guides each learner to focus on samples that other learners struggle to correctly identify, enabling cross-modal knowledge transfer and complementary learning, and enhancing recognition robustness in multimodal distortion scenarios. The specific implementation process is as follows: Based on the calculated CTC loss Each modality learner is assigned sample weights, which are positively correlated with the prediction errors of other learners for that sample. UML-A Sample Weights Determined by the minimum loss principle of UML-V and CML, this approach focuses on samples that are difficult for video modality and cross-modality learners to recognize. ; UML-V sample weights Determined by the minimum loss of UML-A and CML, this approach focuses on samples that are difficult for audio modal and cross-modal learners to recognize. ; CML Sample Weights Determined by the minimum loss of UML-V and UML-A, the focus is on samples that are difficult for video and audio modal learners to recognize. ; The sample weights are obtained by performing Softmax normalization. And build a collaborative compensation update loss This is used to optimize the learners of each modality, making them pay more attention to difficult samples across modalities during training, thereby improving the robustness and accuracy of the overall system.

[0082] In the decoder, the fused features are first transformed through a linear layer, then passed through a softmax layer to obtain the recognition probability distribution of Chinese character units. These probability distributions are then fed into a Connected Temporal Classifier (CTC) to produce the classification results for the Chinese character sequence. The CTC loss function... This allows the model to automatically learn the correspondence between the fused sequence and the Chinese character sequence without having to maintain strict time alignment between them. The calculation of this loss function is based on the assumption of conditional independence, and the specific calculation method is as described in the CTC loss function above. As shown in the expression.

[0083] Therefore, the total loss is designed as described in the total loss expression of the speech recognition model above.

[0084] It should be understood that, although the above Figure 1The steps are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated in this document, there is no strict order in which these steps are executed; they can be performed in other orders. Furthermore, the above... Figure 1 At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0085] In one embodiment, a speech recognition system based on multimodal distortion compensation is also provided, the device comprising: The multimodal signal preprocessing module is used to preprocess the acquired audio signal and the corresponding facial video signal to obtain the audio sequence and the corresponding facial image sequence.

[0086] The speech recognition module based on multimodal distortion compensation processes audio sequences and corresponding facial image sequences using a trained speech recognition model to obtain speech recognition results. The speech recognition model includes a feature extraction module, a multimodal learner, a dynamic contribution evaluation module, a collaborative fusion module, and a decoder. The feature extraction module extracts audio and video modal features from the audio and corresponding facial image sequences. The multimodal learner uses a single-modal learner to model the audio and video modal features separately, preserving the semantic details and scene features of each modality. A cross-modal learner captures the temporal and spatial correlation between the speech signal and lip movements from the output features of the single-modal learner. The dynamic contribution evaluation module obtains corresponding prediction results based on the single-modal and cross-modal learning results, and uses the prediction loss of each modality as a weak supervision signal to convert the prediction loss into modal confidence. The collaborative fusion module fuses the prediction results of each modality learner based on the modal confidence to obtain fused features. The decoder decodes the fused features to obtain the speech recognition result.

[0087] In one embodiment, the speech recognition module based on multimodal distortion compensation is further configured to input an audio sequence and a corresponding facial image sequence into a feature extraction module to obtain audio modal features and video modal features; input the audio modal features and video modal features into a multimodal learner to obtain semantic details and scene features of the corresponding modality and temporal and spatial correlation features between audio signals and lip movements; input the temporal and spatial correlation features, semantic details and scene features of the audio modality and video module into a dynamic contribution evaluation module to obtain cross-modal and single-modal prediction results and corresponding final contribution weights; input the prediction results of the audio modality, video modality and cross-modality and corresponding final contribution weights into a collaborative fusion module for weighted summation to obtain fused features; and input the fused features into a decoder to obtain the speech recognition result.

[0088] In one embodiment, the feature extraction module includes an audio feature extraction module and a video feature extraction module; the speech recognition module based on multimodal distortion compensation is further configured to input an audio sequence into the audio feature extraction module, convert the audio sequence into a Mel spectrogram, process the Mel spectrogram using a two-dimensional convolutional layer and a linear transformation layer to obtain audio module features; input a facial image sequence into the video feature extraction module, use a three-dimensional convolutional neural network to extract temporal and spatial information from the facial image sequence, process the extracted temporal and spatial information using a ResNet network, and process the processing result through global average pooling to obtain video modal features.

[0089] In one embodiment, the multimodal learner includes: an audio single-modal learner, a video single-modal learner, and a cross-modal learner; the audio single-modal learner consists of three Conformer encoders, one Inter-CTC residual module, two Conformer encoders, and two Inter-CTC residual modules connected in sequence; the audio single-modal learner and the video single-modal learner have the same structure; the speech recognition module based on multimodal distortion compensation is further used to input audio modal features into the audio single-modal learner to obtain semantic details and scene features of the audio modality; input video modal features into the video single-modal learner to obtain semantic details and scene features of the video modality; and input the semantic details and scene features of the audio modality and the semantic details and scene features of the video modality into the cross-modal learner to obtain the temporal and spatial correlation features between the audio signal and lip movements.

[0090] In one embodiment, the dynamic contribution evaluation module includes: three prediction networks and three dynamic contribution learning networks; the dynamic contribution learning networks are used to estimate the reliability of each modality sample in real time using the prediction loss as a weak supervision signal; the speech recognition module based on multimodal distortion compensation is further used to input temporal and spatial correlation features, semantic details and scene features of audio modality and video modality into the corresponding prediction networks respectively to obtain cross-modal prediction results, audio modality prediction results and pre-video modality prediction results; the cross-modal prediction results, audio modality prediction results and pre-video modality prediction results are input into the corresponding dynamic contribution learning networks respectively to obtain the final contribution weight of cross-modality, the final contribution weight of audio modality and the final contribution weight of video modality.

[0091] In one embodiment, the dynamic contribution learning network in the speech recognition module based on multimodal distortion compensation includes three sequentially connected fully connected modules and a Sigmoid activation function; the fully connected module includes a fully connected layer, a ReLU activation function, a batch normalization layer, and a Dropout layer.

[0092] In one embodiment, the speech recognition module based on multimodal distortion compensation is further used to transform the fused features through a linear layer, and then process them through a Softmax layer to obtain the recognition probability of Chinese character units; the recognition probability is input into the CTC classifier to obtain the classification result of the Chinese character sequence.

[0093] In one embodiment, the multimodal signal preprocessing module uses a custom-designed headset that simultaneously supports audio and video dual-modal synchronous acquisition to obtain audio signals and corresponding facial video signals. The custom-designed headset is obtained by integrating a high-sensitivity active noise reduction module at the audio acquisition end of the headset and configuring a miniature anti-interference camera at the video acquisition end to focus on the core area of ​​the lips.

[0094] In one embodiment, the process of constructing the total loss function during the training of a speech recognition model based on multimodal distortion compensation includes: firstly, calculating the CTC loss of each modality learner. Transform CTC loss into pseudo-labels To reflect modal confidence;

[0095]

[0096] Where y represents the actual transcribed text. This represents the predicted text for each modality. These are pseudo-tags; For calibration parameters, For the CTC loss of each modality learner, Utilizing weak supervision loss As an optimization objective, the parameters of the dynamically contributing learning network are updated through backpropagation; the weakly supervised loss is:

[0097] in, The weights predicted by DCLN.

[0098] After the feature extraction module parameters stabilize, the feature extraction module is frozen; based on the CTC loss of each modality learner... Each modality learner is assigned sample weights, which are positively correlated with the prediction errors of other learners for that sample; UML-A sample weights. Determined by the minimum loss principle of UML-V and CML, this approach focuses on samples that are difficult for video modality and cross-modality learners to recognize. UML-V sample weights Determined by the minimum loss of UML-A and CML, this approach focuses on samples that are difficult for audio modal and cross-modal learners to recognize. CML sample weights Determined by the minimum loss of UML-V and UML-A, the focus is on samples that are difficult for video and audio modal learners to recognize. The sample weights are obtained by performing Softmax normalization. And build a collaborative compensation update loss This is used to optimize the learners for each modality, making them pay more attention to difficult cross-modality samples during training; specifically:

[0099]

[0100] Using CTC loss function This enables the model to automatically learn the correspondence between the fused sequence and the Chinese character sequence; where the CTC loss function... for:

[0101] in, Given an input sequence x, the probability of outputting a Chinese character sequence label y.

[0102] Based on the weak supervision loss, collaborative compensation update loss, and CTC loss function, the total loss of the speech recognition method and system based on multimodal distortion compensation is determined as follows:

[0103] in For the total loss, For the weight size, and .

[0104] It is understood that for a detailed explanation of the speech recognition system based on multimodal distortion compensation, please refer to the corresponding explanations of the various embodiments of the speech recognition method based on multimodal distortion compensation above, which will not be repeated here. Each module in the above-described speech recognition system based on multimodal distortion compensation can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in hardware or independently of a device with data processing capabilities, or stored in software in the memory of the aforementioned device, so that the processor can call and execute the operations corresponding to each module. The aforementioned device can be, but is not limited to, various types of data processing computer devices already existing in the art.

[0105] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0106] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and all such modifications and improvements fall within the scope of protection of this application.

Claims

1. A speech recognition method based on multimodal distortion compensation, characterized in that, Including the following steps: The acquired audio signal and the corresponding facial video signal are preprocessed to obtain an audio sequence and a corresponding facial image sequence; The trained speech recognition model is used to process the audio sequence and the corresponding facial image sequence to obtain the speech recognition result; The speech recognition model includes a feature extraction module, a multimodal learner, a dynamic contribution evaluation module, a collaborative fusion module, and a decoder. The feature extraction module extracts audio modal features and video modal features from the audio sequence and the corresponding facial image sequence. The multimodal learner uses a single-modal learner to model the audio and video modal features separately, preserving the semantic details and scene features of each modality. A cross-modal learner captures the temporal and spatial correlation between the speech signal and lip movements from the output features of the single-modal learner. The dynamic contribution evaluation module obtains corresponding prediction results based on the single-modal and cross-modal learning results, and uses the prediction loss of each modality as a weak supervision signal to convert the prediction loss into modal confidence. The collaborative fusion module fuses the prediction results of each modality learner based on the modal confidence to obtain fused features. The decoder decodes the fused features to obtain the speech recognition result.

2. The speech recognition method based on multimodal distortion compensation according to claim 1, characterized in that, The audio sequence and the corresponding facial image sequence are processed using a trained speech recognition model based on multimodal distortion compensation to obtain speech recognition results, including: The audio sequence and the corresponding facial image sequence are input into the feature extraction module to obtain audio modal features and video modal features; The audio modal features and the video modal features are input into the multimodal learner to obtain the semantic details and scene features of the corresponding modality and the temporal and spatial correlation features between the audio signal and lip movements; The temporal and spatial correlation features, semantic details of the audio modality and video module, and scene features are input into the dynamic contribution evaluation module to obtain the prediction results across modalities and each single modality and the corresponding final contribution weights. The prediction results of audio modality, video modality, and cross-modality, along with their corresponding final contribution weights, are input into the collaborative fusion module for weighted summation to obtain the fusion features. The fused features are input into the decoder to obtain the speech recognition result.

3. The speech recognition method based on multimodal distortion compensation according to claim 2, characterized in that, The feature extraction module includes: an audio feature extraction module and a video feature extraction module; The audio sequence and the corresponding facial image sequence are input into the feature extraction module to obtain audio modal features and video modal features, including: The audio sequence is input into the audio feature extraction module, the audio sequence is converted into a Mel spectrogram, and the Mel spectrogram is processed by a two-dimensional convolutional layer and a linear transformation layer to obtain the audio module features; The facial image sequence is input into the video feature extraction module, and a three-dimensional convolutional neural network is used to extract the temporal and spatial information in the facial image sequence. The extracted temporal and spatial information is processed by a ResNet network, and the processing result is processed by global average pooling to obtain video modal features.

4. The speech recognition method based on multimodal distortion compensation according to claim 2, characterized in that, The multimodal learner includes: an audio unimodal learner, a video unimodal learner, and a cross-modal learner; the audio unimodal learner consists of three Conformer encoders, one Inter-CTC residual module, two Conformer encoders, and two Inter-CTC residual modules connected in sequence; the audio unimodal learner and the video unimodal learner have the same structure; The audio modal features and the video modal features are input into the multimodal learner to obtain the semantic details and scene features of the corresponding modality, as well as the temporal and spatial correlation features between the audio signal and lip movements, including: The audio modal features are input into the audio unimodal learner to obtain the semantic details and scene features of the audio modality; The video modal features are input into the video unimodal learner to obtain the semantic details and scene features of the video modality; The semantic details and scene features of the audio modality and the video modality are input into the cross-modal learner to obtain the temporal and spatial correlation features between the audio signal and lip movements.

5. The speech recognition method based on multimodal distortion compensation according to claim 2, characterized in that, The dynamic contribution evaluation module includes three prediction networks and three dynamic contribution learning networks; the dynamic contribution learning network is used to estimate the reliability of each modality sample level network module in real time with the prediction loss as a weak supervision signal. The temporal and spatial correlation features, semantic details of the audio modality and video module, and scene features are input into the dynamic contribution evaluation module to obtain the prediction results across modalities and for each single modality, along with the corresponding final contribution weights, including: The temporal and spatial correlation features, semantic details of audio modality and video modality, and scene features are respectively input into the corresponding prediction networks to obtain cross-modal prediction results, audio modality prediction results, and video modality prediction results; The cross-modal prediction results, audio modal prediction results, and video modal prediction results are input into the corresponding dynamic contribution learning networks to obtain the final contribution weights for the cross-modal, audio modal, and video modal predictions.

6. The speech recognition method based on multimodal distortion compensation according to claim 2, characterized in that, The dynamic contribution learning network comprises three sequentially connected fully connected modules and a Sigmoid activation function; each fully connected module comprises a fully connected layer, a ReLU activation function, a batch normalization layer, and a Dropout layer.

7. The speech recognition method based on multimodal distortion compensation according to claim 2, characterized in that, The fused features are input into the decoder to obtain the speech recognition result, including: The fused features are transformed through a linear layer and then processed through a Softmax layer to obtain the recognition probability of Chinese character units. The recognition probability is input into the CTC classifier to obtain the classification result of the Chinese character sequence.

8. The speech recognition method based on multimodal distortion compensation according to claim 1, characterized in that, A custom-designed headset that simultaneously supports audio and video dual-modal synchronous acquisition is used to obtain audio signals and corresponding facial video signals; The customized earphones are obtained by integrating a high-sensitivity active noise cancellation module into the earphone's audio acquisition end and configuring a miniature anti-interference camera in the video acquisition end to focus on the core area of ​​the lips.

9. The speech recognition method based on multimodal distortion compensation according to claim 1, characterized in that, The training process of a speech recognition model based on multimodal distortion compensation involves constructing the total loss function, which includes: First, calculate the CTC loss of each modality learner. Transform CTC loss into pseudo-labels To reflect modal confidence; Where y represents the actual transcribed text. This represents the predicted text for each modality. These are pseudo-tags; For calibration parameters, For the CTC loss of each modality learner, Utilizing weak supervision loss As an optimization objective, the parameters of the dynamically contributing learning network are updated through backpropagation; the weakly supervised loss is: in, The weights predicted by DCLN; After the parameters of the feature extraction module stabilize, the feature extraction module is frozen; Based on the CTC loss of each modality learner Each modality learner is assigned sample weights, which are positively correlated with the prediction errors of other learners for that sample. UML-A Sample Weights Determined by the minimum loss principle of UML-V and CML, this approach focuses on samples that are difficult for video modality and cross-modality learners to recognize. ; UML-V sample weights Determined by the minimum loss of UML-A and CML, this approach focuses on samples that are difficult for audio modal and cross-modal learners to recognize. ; CML Sample Weights Determined by the minimum loss of UML-V and UML-A, the focus is on samples that are difficult for video and audio modal learners to recognize. ; The sample weights are obtained by performing Softmax normalization. And build a collaborative compensation update loss This is used to optimize the learners for each modality, making them pay more attention to difficult cross-modality samples during training; specifically: Using CTC loss function This enables the model to automatically learn the correspondence between the fused sequence and the Chinese character sequence; where the CTC loss function... for: in, Given an input sequence x, the probability of outputting a Chinese character sequence label y. Based on the weak supervision loss, the cooperative compensation update loss, and the CTC loss function, the total loss of the speech recognition method and system based on multimodal distortion compensation is determined as follows: in For the total loss, For the weight size, and .

10. A speech recognition system based on multimodal distortion compensation, characterized in that, include: The multimodal signal preprocessing module is used to preprocess the acquired audio signal and the corresponding facial video signal to obtain the audio sequence and the corresponding facial image sequence. The speech recognition module based on multimodal distortion compensation is used to process the audio sequence and the corresponding facial image sequence using a trained speech recognition model to obtain the speech recognition result; The speech recognition model includes a feature extraction module, a multimodal learner, a dynamic contribution evaluation module, a collaborative fusion module, and a decoder. The feature extraction module extracts audio modal features and video modal features from the audio sequence and the corresponding facial image sequence. The multimodal learner uses a single-modal learner to model the audio and video modal features separately, preserving the semantic details and scene features of each modality. A cross-modal learner captures the temporal and spatial correlation between the speech signal and lip movements from the output features of the single-modal learner. The dynamic contribution evaluation module obtains corresponding prediction results based on the single-modal and cross-modal learning results, and uses the prediction loss of each modality as a weak supervision signal to convert the prediction loss into modal confidence. The collaborative fusion module fuses the prediction results of each modality learner based on the modal confidence to obtain fused features. The decoder decodes the fused features to obtain the speech recognition result.

Images