A real-time speech transcription method and system based on multi-modal deep learning

Abstract

The present application relates to the field of multi-modal model data processing, in particular to a real-time speech transcription method and system based on multi-modal deep learning, first, the timing alignment of audio and lip video frames is realized through a hardware clock; secondly, the audio spectral entropy is extracted as an acoustic aliasing index, and the lip movement displacement rate is extracted as a lip movement saliency index by using a deep neural network; then, based on the cross-modal consistency deviation ratio, dynamic gating weights are generated and the audio and video features are weighted and fused; finally, the text is generated by inputting into an end-to-end model based on a self-attention mechanism. The present application effectively overcomes the acoustic interference in the multi-speaker overlapping scene, automatically increases the weight of the visual lip movement feature which is not affected by the environmental noise when the audio aliasing is serious, greatly improves the robustness and accuracy of the transcription in the complex conference scene; at the same time, through the pipeline architecture, the system delay is extremely low, meeting the real-time interaction requirements.

Description

Technical Field

[0001] This invention relates to the field of multimodal model data processing, and specifically to a real-time speech transcription method and system based on multimodal deep learning. Background Technology

[0002] Speech-to-text technology aims to convert continuous human speech streams into text sequences in real time, automatically, and accurately. As a core underlying technology for human-computer interaction, intelligent dictation, and automated meeting recording, it has made significant breakthroughs in recent years driven by deep learning algorithms. Its recognition accuracy in single-person speech and quiet environments has basically met daily needs.

[0003] With the popularization of smart offices and digital transformation, the application of speech-to-text technology has gradually expanded from simple single-person dictation to more realistic and complex collaborative scenarios such as multi-person meetings, business negotiations, and academic seminars. In these practical application scenarios, transcription systems are usually deployed in physical meeting rooms, using hardware collaboration platforms to simultaneously collect the audio streams and facial visual streams of on-site participants in order to generate complete and accurate meeting transcripts.

[0004] However, in real-world multi-person conferences, heated discussions, interruptions, premature responses, and simultaneous speaking by multiple participants are inevitable. When using traditional pure audio transcription solutions or conventional fixed-weight multimodal solutions to address conference transcription issues, severe physical sound wave superposition and aliasing occur when multiple participants speak simultaneously. This phenomenon, which is highly prevalent in conference scenarios, severely contaminates the target audio features acquired by the system. If the system continues to blindly rely on the distorted audio for recognition, the backend transcription engine will become chaotic, resulting in widespread errors, garbled text, or semantic gaps in the transcription of the target speaker. Consequently, the transcription system's accuracy drops drastically in real-world scenarios with overlapping multi-person speaking, severely impacting the usability of the meeting minutes. Summary of the Invention

[0005] To address the widespread issues of misspellings, garbled text, or semantic gaps in multi-person conferencing scenarios, this invention proposes a real-time speech-to-text method based on multimodal deep learning in its first aspect. The method includes: injecting timestamps into the acquired audio and video streams; segmenting the audio stream based on video frame time intervals; extracting lip region images from the video frames to obtain frame-aligned audio frame sequences and lip image frame sequences; performing frequency domain transformation on each frame of the audio frame sequence and extracting Mel spectrum vectors; calculating spectral entropy as an acoustic aliasing index based on the normalized energy proportion of each Mel sub-band, where the acoustic aliasing index characterizes the dispersion of spectral energy in the current frame among the sub-bands; and processing the lip image frame sequence through a convolutional neural network. Spatial feature codes are extracted, and temporal feature vectors are extracted using a recurrent neural network. A lip movement saliency index is calculated based on the mean Euclidean distance between the spatial feature codes of adjacent frames within a sliding window. This lip movement saliency index characterizes the activity level of the target speaker's lip movements. The ratio of the acoustic aliasing index and the lip movement saliency index is calculated after normalization to obtain the cross-modal consistency deviation ratio. Audio channel confidence weights and visual channel confidence weights are obtained through inverse square mapping. The audio channel confidence weights and visual channel confidence weights are used to weight and fuse the dimension-aligned Mel spectrum vector and the lip movement temporal feature vector. The fused multimodal feature sequence is then fed into an end-to-end sequence transcription model to output the real-time transcribed text of the target speaker.

[0006] Compared to existing technologies that rely solely on a single audio signal and suffer significant drops in recognition rates in noisy environments, or multimodal technologies that simply use fixed weights for splicing, this solution can dynamically and adaptively adjust the fusion ratio of audio and visual features in complex real-world application scenarios by assessing the degree of audio interference and the activity level of the target speaker in real time. This allows it to rely more on visual lip reading when audio is impaired and more on audio when visual information is not readily apparent, significantly improving the accuracy and robustness of real-time speech transcription in complex interference environments.

[0007] Furthermore, the specific method for calculating the acoustic aliasing index is as follows:

[0008] ;

[0009] in For the first Acoustic aliasing index of frames; For the first Frame number The normalized energy percentage of each Mel subband; For the first The energy value of each melon subband; Indicates the first Frame number The energy value of each melon subband; This represents the total number of channels in the Mel filter bank.

[0010] By using the spectral entropy calculation formula based on the normalized energy ratio of Mel subband, a lightweight and efficient method for quantifying environmental noise interference is provided for the system. This allows the system to quickly and accurately perceive the spectral interference status of the current speech in actual recording scenarios without the need to deploy a large and complex noise recognition model, thus ensuring the real-time performance and low latency of the system on terminal devices.

[0011] Furthermore, the specific method for calculating the lip movement significance index is as follows:

[0012] ;

[0013] in For the first Indicators of lip movement significance in frames; This is the length of the time series observation window; For the first in the window The lip spatial feature encoding vector of the frame; For the first in the window The lip spatial feature encoding vector of the frame; This represents the Euclidean norm.

[0014] Compared to existing solutions that rely solely on static face detection, this method calculates lip movement saliency by using the mean Euclidean distance of spatial feature encoding within a sliding window. This allows for the accurate capture of dynamic physical deformation of the user's lips during speech, effectively eliminating interference from non-speakers in video conferencing or security monitoring scenarios where multiple people are in the same frame or where static faces are present. This ensures that the visual channel remains focused on the real speaker.

[0015] Furthermore, the specific method for calculating the cross-modal consistency deviation ratio is as follows:

[0016] ;

[0017] in For the first Frame cross-modal consistency deviation ratio; This represents the theoretical upper limit of the acoustic aliasing index; where... This represents the total number of channels in the Mel filter bank; This serves as a reference benchmark for the significance of lip movements obtained during the calibration phase. To prevent overflow of tiny positive numbers when divided by zero.

[0018] Furthermore, the method for obtaining the lip movement salience reference benchmark is as follows: during the system initialization phase, the target speaker reads a standard text of no less than 30 seconds in an interference-free environment, and the lip movement salience index values ​​of all speech activity frames are collected. The 50th percentile value is taken as the lip movement salience reference benchmark. .

[0019] Furthermore, the confidence weights for the audio channel and the visual channel are calculated as follows:

[0020] ;

[0021] in Confidence weights assigned to audio channels; The confidence weights assigned to the visual channel; The cross-modal consistency deviation ratio is given.

[0022] By using the inverse square mapping formula to calculate the confidence weights of audio and video channels, a smooth nonlinear transition of weights can be achieved during multimodal feature fusion. This avoids extreme switching of system feature fusion caused by brief and drastic fluctuations in environmental noise in real-life scenarios, thus ensuring the coherence and stability of the final output transcribed text.

[0023] Furthermore, when the cross-modal consistency deviation ratio The value continuously exceeds the extreme aliasing threshold for more than a preset number of consecutive frames. At that time, the confidence weight of the audio channel is set to not exceed a preset minimum value; the extreme aliasing threshold The value range is 2.0 to 5.0, and the preset minimum value is 0.1.

[0024] Furthermore, the convolutional neural network adopts a backbone network structure based on depthwise separable convolution, which includes multiple convolutional modules cascaded in sequence. Each convolutional module consists of a depthwise separable convolutional layer, a batch normalization layer, and a linear rectified activation function. The output of the convolutional module is compressed into a fixed-dimensional spatial feature encoding vector by a global average pooling layer.

[0025] Furthermore, the method also includes a visual channel failure degradation step: when the lip region fails to be located within a preset number of consecutive frames, the visual channel is marked as unavailable, and the confidence weight of the audio channel is forcibly set to 1 and the confidence weight of the visual channel is forcibly set to 0; when a preset number of valid lip location results are detected again, the dual-channel collaborative working mode is restored.

[0026] This greatly improves the system's fault tolerance and availability in real and complex scenarios. When the visual channel fails due to sudden situations such as the user turning their head, leaving the camera's field of vision, or their face being obscured, the system can seamlessly degrade back to pure audio transcription mode and automatically restore dual-channel collaboration after the visual signal is restored. This avoids transcription interruption or program crashes caused by the loss of data from a single sensor, ensuring the continuity of the user experience.

[0027] In a second aspect, the present invention provides a real-time speech-to-text system based on multimodal deep learning, comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the real-time speech-to-text method based on multimodal deep learning of the present invention is implemented.

[0028] The technical effects of this invention are as follows:

[0029] This invention innovatively calculates acoustic aliasing indices in real time to assess audio noise interference and combines them with lip movement saliency indices to quantify the effectiveness of the visual channel. By comparing the deviation ratio between the two, the system can adaptively and dynamically adjust the fusion weights of audio and video features. This mechanism overcomes the vulnerability of single audio recognition to strong noise and, together with strategies such as visual failure degradation and individual benchmark calibration, ensures high accuracy and robustness in speech-to-text output in various realistic and complex interference scenarios. Attached Figure Description

[0030] Figure 1 This is a schematic flowchart illustrating a real-time speech-to-text method based on multimodal deep learning according to an embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram illustrating the mapping curve of the inverse square gating weight function as a function of the cross-modal consistency deviation ratio in an embodiment of the present invention;

[0032] Figure 3 This is a bar chart illustrating the gating weight allocation of the audio and visual channels in different typical scenarios in embodiments of the present invention;

[0033] Figure 4 This is a schematic illustration of the feature heat map of the multimodal joint feature sequence after gated weighted fusion as a function of time in an embodiment of the present invention;

[0034] Figure 5 This is an illustrative comparison of the frame-by-frame character error rate over time between the method described in this embodiment of the invention and the method using only audio baseline and fixed weight fusion.

[0035] Figure 6This is a bar chart illustrating the comparison of character error rates between the method described in this embodiment of the invention and the audio baseline-only and fixed weight fusion method in different scenarios.

[0036] Figure 7 This is a schematic diagram illustrating the comparison of the character error rate trends of the method in this embodiment of the invention with the audio baseline-only and fixed weight fusion method under different signal-to-noise ratio conditions;

[0037] Figure 8 This is a schematic diagram illustrating the structure of a real-time speech-to-text system based on multimodal deep learning, according to an embodiment of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] The real-time speech-to-text method based on multimodal deep learning described in this embodiment is deployed on an audio-video collaborative processing hardware platform. The core processing unit of this platform adopts an embedded computing module based on a high-performance graphics processor. The computing module is equipped with a central processing unit with no less than eight cores and an independent tensor acceleration unit, and the onboard memory capacity is no less than 16GB to meet the real-time inference computing power requirements of deep learning models.

[0040] At the audio acquisition end, the system connects to a circular array of eight MEMS omnidirectional electret microphones via a digital audio interface. The preferred spacing between the microphones is 35 mm, but in practice, this spacing can be adjusted from 20 mm to 80 mm depending on the size of the conference table. The sampling rate of the microphone array... The frequency is set to 16kHz, the quantization depth is 16 bits, and the raw audio data is transmitted to the audio direct storage access channel of the computing module through the internal serial audio bus after analog-to-digital conversion.

[0041] At the visual acquisition end, the system is equipped with an infrared-enhanced industrial camera with a resolution of no less than 1080p. The camera is connected to the computing module via a Universal Serial Bus 3.0 interface, with a frame rate... The preferred frame rate is 30 frames per second, but this frame rate can be adjusted from 15 frames per second to 60 frames per second depending on lighting conditions and computational load. The camera lens focal length is pre-calibrated and fixedly aimed at the facial area of ​​the target speaker.

[0042] At the time synchronization level, the audio acquisition module and the video acquisition module share the hardware real-time clock source inside the computing module. The clock synchronization daemon injects high-precision timestamps into the audio frame and video frame respectively in each acquisition cycle to ensure the time base consistency of cross-modal data.

[0043] The operating system of the computing module is a real-time enhanced embedded Linux distribution, the deep learning inference framework adopts a tensor-accelerated runtime environment, and all models are deployed in fixed-point inference mode after quantization and compression to reduce latency.

[0044] It should be noted that although this embodiment uses the above-mentioned embedded computing module as the hardware platform, in other embodiments, it can also be deployed on a cloud inference platform based on a server-level graphics processor cluster, and the audio and video streams can be uploaded to the cloud for processing and then the transcription results can be returned to achieve the same technical effect.

[0045] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0046] An example of a real-time speech-to-text method based on multimodal deep learning:

[0047] like Figure 1 As shown, a real-time speech transcription method based on multimodal deep learning according to the present invention includes:

[0048] S1. Based on a hardware clock source, timestamp injection and frame-level alignment are performed on the audio frame sequence and the lip video frame sequence to construct a synchronous multimodal data pair.

[0049] First, the computing module continuously reads the raw discrete audio sample point sequence from the microphone array's direct memory access channel. Simultaneously, it continuously reads video frame sequences from the camera's Universal Serial Bus interface. Due to audio sampling rate For 16kHz video frame rate Since the original temporal resolutions of the two methods differ significantly (30 frames per second), a frame-level temporal correspondence needs to be established.

[0050] Furthermore, the clock synchronization daemon uses the arrival timestamp of the video frame as the reference anchor point to synchronize the audio sampling point stream according to the video frame interval. Segmentation is performed. In this embodiment, At 30 frames per second, therefore Approximately 33.3 milliseconds, each audio segment contains Each video frame has one sampling point. The above extraction process results in each video frame having one sampling point. Strictly corresponds to a segment of length audio clips ,in This refers to the sampling point number within the segment.

[0051] Then the system processes each video frame. Perform lip region localization and cropping operations. For example, in this embodiment, a facial landmark detector based on a cascaded regression tree is used to locate 68 facial feature points. The minimum bounding rectangle region enclosed by 20 lip contour landmarks numbered 49 to 68 is extracted from these landmarks, and then cropped into a fixed-size grayscale image after extending the margins by 15% on each side. The cropping size is preferably set to 96 pixels by 96 pixels, but depending on the camera resolution and the speaker distance, this size can be adjusted in the range of 64 pixels by 64 pixels to 128 pixels by 128 pixels.

[0052] After the above processing, a time-aligned audio frame sequence is obtained. With the corresponding lip region image frame sequence ,in For the total number of frames, each pair Corresponding to the same physical moment.

[0053] Specifically, if the facial key point detector fails to locate the lip region in 5 consecutive video frames, the system determines that the target speaker's face may be obscured or has left the camera's field of view. At this time, the system automatically marks the visual channel as unavailable, and the modal fusion weights in subsequent steps will be forcibly set to rely solely on the audio channel for transcription until the lip region is successfully relocated and the dual-channel collaborative working mode is restored.

[0054] It should be noted that although this embodiment uses a facial landmark detector based on cascaded regression trees for lip localization, in other embodiments, an end-to-end face detector based on a lightweight convolutional neural network or a landmark regression network based on depthwise separable convolution can also be used to achieve the same lip region localization effect.

[0055] S2. Extract the spectral energy distribution of the audio frame using the Mel filter bank and quantify the degree of multi-source aliasing based on spectral entropy to obtain the acoustic aliasing index.

[0056] S201. Perform frequency domain decomposition on the audio frame based on Hamming windowing and discrete Fourier transform to obtain the energy distribution of each sub-band.

[0057] First, the computation module reads audio segments frame by frame from the output of step S101. A Hamming window function is applied to each audio frame to suppress spectral leakage, and then the following steps are performed: Frequency domain representation obtained by point discrete Fourier transform ,in In this embodiment, the number of points is preferably set to 512, but depending on the frequency resolution requirements, the number of points can be adjusted in the range of 256 to 1024.

[0058] Furthermore, the frequency domain representation The amplitude spectrum is obtained through a set of A triangular Mel filter, the number of channels of the Mel filter bank In this embodiment, the number of channels is preferably set to 40, but depending on the balance between recognition accuracy and computational load, this number can be adjusted within the range of 26 to 80. The center frequencies of each filter are uniformly distributed on a Mel scale from 0Hz to... The frequency range. Calculate the output for each filter channel to obtain the first... Frame number The energy value of each child This forms the Mel spectrum vector. .

[0059] S202. Calculate the spectral entropy of the normalized energy ratio of each subband based on the information entropy theory to quantify the degree of spectral energy diffusion and thus characterize the severity of acoustic aliasing.

[0060] In obtaining the Mel spectrum vector Subsequently, in order to quantify the severity of multi-source aliasing in the current frame's acoustic signal, the system further constructs an acoustic aliasing index.

[0061] Under single-speaker conditions, speech energy is mainly concentrated in the Mel subbands corresponding to a few formants, resulting in a highly concentrated peak-like spectral energy distribution. However, when multiple speakers simultaneously address each other, the superposition of their individual formant structures causes the energy to tend towards a more uniform distribution across a wider range of subbands. The degree of spectral energy dispersion can statistically reflect the extent to which the current frame is interfered with by multi-source aliasing. Based on this physical law, and drawing on the concept of spectral entropy from information theory to measure this dispersion characteristic, the acoustic aliasing index is defined as follows:

[0062] ;

[0063] in For the first The acoustic aliasing index of a frame, namely the spectral entropy value, is expressed in bits. For the first Frame number The normalized energy percentage of each Mel subband is calculated as follows: ; The first one obtained through step S201 The energy value of each melon subband; This represents the total number of channels in the Mel filter bank, which is 40 in this embodiment.

[0064] As shown in the above formula, when only a single speaker is emitting sound, the energy is highly concentrated in the subbands corresponding to a few formants, and the normalized proportion is... The information entropy value exhibits an extremely uneven distribution. Approaching zero; as the number of simultaneous speakers increases, the formants of each speaker overlap in the frequency domain, causing the energy proportion of each subband to gradually become more uniform, and the information entropy value... The corresponding increase, its theoretical upper limit is In this embodiment When the value is 40, the upper limit is approximately 5.32 bits. This variation reflects the gradual progression of multi-source aliasing in a conference room setting, from mild to severe: When the value increases from a near-zero pure speech state to an extreme aliasing state of nearly 5.32, it reflects that the purity of the target speaker's acoustic features in the signal collected by the microphone is continuously decreasing.

[0065] S3. Extract the spatial morphological features and temporal motion trajectory features of the lips in sequence through convolutional neural networks and recurrent neural networks, and quantify the lip movement activity based on the inter-frame displacement rate.

[0066] S301. Spatial feature encoding is performed on the grayscale image of the lips based on a lightweight convolutional neural network to obtain the geometric representation vector of the lip shape.

[0067] First, the computation module reads the grayscale image of the lip region frame by frame from the output of step S101. Each frame of the image is fed into a pre-trained lightweight convolutional neural network to extract spatial feature encodings. In this embodiment, the convolutional neural network adopts a lightweight backbone network structure based on depthwise separable convolution, containing five convolutional modules. Each module consists of one depthwise separable convolutional layer, one batch normalization layer, and one linear rectified activation function cascaded sequentially. The number of output channels for each convolutional module is 32, 64, 128, 256, and 512, respectively. The kernel size is uniformly 3x3, and the stride is 2. Finally, the output of the convolutional modules is compressed by a global average pooling layer. dimensional vector, In this embodiment, the value is preferably set to 256, but depending on the balance between feature representation accuracy and inference speed, this dimension can be adjusted within the range of 128 to 512.

[0068] The pre-training process of the above convolutional neural network uses a public dataset containing lip video clips of no less than 5,000 different speakers. The optimizer is an adaptive moment estimation optimizer. The initial learning rate is set to 0.001, the batch size is set to 64, and the training epochs are set to 50. The learning rate is reduced to one-tenth of its original value in the 30th and 40th epochs, respectively.

[0069] The spatial feature encoding It contains geometric information such as the opening degree, roundness, and distance between the upper and lower lips of the current frame.

[0070] S302. Based on a bidirectional long short-term memory network, temporal modeling of lip spatial features in consecutive frames is performed to capture lip movement trajectory information in context.

[0071] Furthermore, to capture the temporal articulation information carried by the dynamic movements of the lips, the system will continuously... The spatial feature codes of the lips in each frame are concatenated in chronological order and then fed into a bidirectional long short-term memory network. In this embodiment, The preferred setting is 15 frames, corresponding to a time window of approximately 500 milliseconds. However, depending on differences in speech rate and language characteristics, the window length can be adjusted within the range of 10 to 25 frames. The bidirectional long short-term memory network contains two hidden layers, with each layer's hidden state dimension preferably set to 128 (128 dimensions in the forward direction and 128 dimensions in the backward direction). The output dimension after concatenation is... The value is 256. After processing by the network, a temporal feature vector is obtained that integrates lip movement trajectory information from the preceding and following context. .

[0072] It should be noted that although this embodiment uses a bidirectional long short-term memory network for temporal modeling, in other embodiments, a one-dimensional causal convolutional structure based on a temporal convolutional network or a temporal encoder based on a self-attention mechanism can also be used to achieve the same effect of lip movement temporal feature extraction.

[0073] S303. Calculate the inter-frame displacement rate based on the mean Euclidean distance of the lip feature vectors of adjacent frames within the sliding window to obtain the lip movement significance index.

[0074] In obtaining lip spatial feature encoding Subsequently, in order to quantify the activity level of the target speaker's lip movements in the current frame, the system further constructs a lip movement saliency index.

[0075] In aliased speech segments with multiple speakers, if the target speaker's lips are in obvious motion, then the speaker is indeed phonating at that moment, and their lip movement features carry reliable articulation cues. Conversely, if the target speaker's lips are still, then the semantic contribution of that speaker in the aliased signal captured by the microphone is zero, and the information in the audio channel is entirely interference from other sound sources for the transcription task. Based on this, the displacement rate of the lip feature vector in the feature space between adjacent frames is used to measure the degree of lip movement activity. The lip movement significance index is defined as:

[0076] ;

[0077] in For the first The significance index of lip movement in frames; Represents Euclidean norm operations; This is the length of the time series observation window, which is set to 15 frames in this embodiment; For the first in the window The lip spatial feature encoding vector of the frame is obtained by step S301.

[0078] As shown in the above equation, when the target speaker is speaking, the lip opening and closing movements cause significant differences between the spatial feature encoding vectors of adjacent frames, increasing the Euclidean distance. High values ​​are observed; when the target speaker is not speaking, the lips remain still or there are only slight facial muscle twitches, and the feature encoding vectors of adjacent frames are highly similar. Approaching zero. This change physically maps the gradual process of the target speaker's vocal activity from silence to loud articulation: as... An increase from near zero to a higher value reflects an increase in the range of lip movements of the target speaker, indicating that the speaker is engaging in more active vocalization.

[0079] S4. Based on the acoustic aliasing index of the audio channel and the lip movement significance index of the visual channel, a cross-modal consistency deviation ratio is constructed and dynamic gating weights are generated by inverse square mapping.

[0080] S401. After normalizing the dual-channel indicators based on the calibration stage statistics, calculate their ratio to obtain the cross-modal consistency deviation ratio.

[0081] First, the system reads the acoustic aliasing index from step S202. Read the lip movement significance index from step S303 A consistency checker is constructed between the audio and visual channels. By comparing the judgments of the target speaker's vocal state in the current frame reflected independently by the two channels, a gating signal for subsequent modality fusion is obtained.

[0082] The verifier is shown below: In an interference-free single-person speaking scenario, when the speaker speaks, the spectral energy of the audio channel concentrates, causing... Lower, while active lip movements in the visual channel lead to At higher frequencies, both channels provide consistent vocalization judgments; however, in scenarios with overlapping multiple speakers, even if the target speaker is vocalizing, the two channels may still give the same vocalization judgment. Maintaining a high value, the audio channel suffers from the superposition of multiple sound sources. An abnormal increase in noise levels led to conflicting conclusions from the two channels. This cross-modal consistency deviation precisely indicates the presence and extent of multi-source interference.

[0083] Before constructing the consistency deviation ratio, it is necessary to obtain the reference baseline quantity required for normalization. The acquisition method is as follows: During the system initialization phase, the target speaker is asked to read a standard text of no less than 30 seconds in a quiet environment at a normal speaking speed. The system collects the lip movement saliency index values ​​of all frames identified as speech activity during this process, sorts them in ascending order, and takes the 50th percentile value as the threshold. In this embodiment, The typical value range is a dimensionless number between 0.5 and 2.0, with the specific value depending on the speaker's individual pronunciation habits.

[0084] In another embodiment, in a long-running meeting scenario, the target speaker's vocal state may drift slowly due to fatigue or emotional changes, causing the lip movement salience reference value obtained in the initial calibration phase to be affected. Gradually deviating from the actual level. In this embodiment, the system continuously collects the lip movement significance index values ​​of pure speech frames judged as high confidence by the gating weights during operation. After accumulating no less than 100 valid samples, the index is calculated using an exponential moving average. Perform an online update, with the update coefficient preferably set to 0.1, meaning the new baseline is equal to 90% of the original baseline plus 10% of the median of the newly collected samples.

[0085] Based on the above analysis, the cross-modal consistency deviation ratio is defined as:

[0086] ;

[0087] in For the first The cross-modal consistency deviation ratio of frames is a dimensionless number. The first one obtained in step S202 Frame acoustic aliasing index; This is the theoretical upper limit of spectral entropy, as described in this embodiment. Take 40 Approximately 5.32 bits; The first one obtained in step S303 Significance index of frame lip movement; This serves as a reference benchmark for the significance of lip movements obtained during the calibration phase. To prevent overflow of tiny positive numbers when dividing by zero, this value is preferably set to 0.001 in this embodiment. However, depending on the required numerical precision, this value can be adjusted within the range of 0.0001 to 0.01.

[0088] From the above formula, we can see that the molecule The acoustic aliasing index is used as a ratio relative to the theoretical upper limit, with a value ranging from 0 to 1; the denominator The significance of lip movement is used as a ratio relative to the normal level of vocalization. When the value approaches zero, it indicates that the audio channel is clean and the speaker is speaking; both channels agree that the current frame is a clean target speech segment. When the value is much greater than 1, it indicates that the audio channels are severely aliased, but the speaker is indeed making a sound. The conclusions of the two channels are contradictory, and the reliability of the audio features has been seriously compromised.

[0089] For example, suppose the acoustic aliasing index of a certain frame The bit depth increased from approximately 1.0 bits in the pure speech state to approximately 4.0 bits in the multi-person overlapping state, while the target speaker spoke normally. If it remains around 1.0, then the formula can be obtained. From the agreement Rise to approximately This reflects that the cross-modal consistency deviation increases significantly due to the aggravation of multi-source interference.

[0090] S402. Based on the inverse square function form, the deviation ratio is mapped to complementary confidence weights of the audio and visual channels to generate a continuous and smooth modal gating signal.

[0091] Next, based on the cross-modal consistency deviation ratio obtained in step S401 Calculate the modal gating weights used for backend fusion. The gating weights must satisfy the following condition: when When the audio channel is smaller, it should receive higher weight. As the visual channel weights increase, their weights should rise smoothly to take over semantic judgment, and the sum of the weights of the two channels should always be 1 to ensure energy conservation of the fused features. Based on the decay form of the Lorentz function, an inverse square relationship is used to characterize the decay of audio channel confidence as aliasing increases. The modal gating weights are defined as follows:

[0092] ;

[0093] ;

[0094] in For the first The confidence weight assigned to the audio channel for each frame ranges from 0 to 1; For the first The confidence weight assigned to the visual channel for each frame ranges from 0 to 1. The cross-modal consistency deviation ratio is obtained in step S401.

[0095] From the above formula, we can see that The term makes the confidence weight of the audio channel... The response to the deviation ratio exhibits an inverse square-proportional decay characteristic. When When equal to 0, Equals 1, When the value equals 0, the system fully trusts the audio channel; when When equal to 1, and Each is 0.5, and the system gives equal trust to both channels; when After exceeding 1, according to The rate decays rapidly. As the value approaches 1, the system becomes dominated by the visual channel. This reflects the process where the system's trust in audio features rapidly decreases as multi-source interference intensifies.

[0096] like Figure 2 As shown, the confidence weight of the audio channel exhibits an inverse square-proportional decay characteristic as the cross-modal consistency deviation ratio increases. When the deviation ratio is zero, the confidence weight of the audio channel is 1, and the system fully trusts the audio channel. When the deviation ratio equals 1, the confidence weights of the audio and visual channels are each 0.5, and the system gives equal trust to both channels. As the deviation ratio continues to increase, the confidence weight of the audio channel decays rapidly at an inverse square-proportional rate and approaches zero, while the confidence weight of the visual channel rises accordingly and approaches 1, and the system switches to using the visual channel to dominate semantic judgment. Compared with the linear decay method, the inverse square-proportional mapping produces a steeper decay gradient in the critical region, making the decision boundary between audio credibility and audio untrustworthiness clearer for the gated signal, while maintaining the smoothness of the transition region due to the continuous differentiability of the function itself.

[0097] In the denominator Compared to the linear term, the squared term produces a steeper decay gradient in the critical region, which makes the gated signal produce a clear decision boundary between audio credibility and audio untrustworthiness. At the same time, the continuous differentiability of the function itself maintains the smoothness of the transition region, preventing abrupt jumps in weights from causing step disturbances to the input features of the back-end recognition model.

[0098] For example, Substituting the pure speech state of 0.19 derived in step S401 into the above formula, we get... This indicates that the audio channel receives a confidence weight of approximately 96.5%; this applies to the case of multiple people overlapping. Substituting into the above equation, we get The confidence weight of the audio channel decreased to approximately 64.0%, while the confidence weight of the visual channel increased accordingly to approximately 36.0%. This demonstrates that as the level of multi-source interference intensifies, the gating mechanism automatically allocates a larger proportion of confidence to the visual channel, which is unaffected by acoustic aliasing.

[0099] like Figure 3 As shown, in a pure single-speaker speech scenario, the audio channel acquires a high confidence weight of approximately 0.96, while the visual channel acquires only a confidence weight of approximately 0.04, indicating that the system relies almost entirely on the audio channel for transcription. In a slightly overlapping scenario, the audio channel confidence weight decreases to approximately 0.61, while the visual channel increases to approximately 0.39, and the system begins to incorporate lip movement information from the visual channel for auxiliary judgment. In a severely overlapping scenario, the audio channel confidence weight further decreases to approximately 0.20, while the visual channel increases to approximately 0.80, and the system shifts to using the visual channel to dominate semantic extraction. In an extreme aliasing scenario, the audio channel retains only a minimum confidence weight of approximately 0.06, while the visual channel acquires a dominant weight of approximately 0.94. This weight allocation process verifies that the gating mechanism can adaptively adjust the fusion ratio of the two channels according to the degree of interference from multiple sound sources.

[0100] In particular, if If the aliasing threshold is consistently exceeded for more than 10 consecutive frames, it will be greater than the preset extreme aliasing threshold. The system determines that the current situation is a severe state of continuous overlap among multiple speakers. In this embodiment... The preferred setting is 3.0, but depending on the application scenario and the requirements for transcription robustness, this threshold can be adjusted within the range of 2.0 to 5.0. In this case, the system will forcibly set the confidence weight of the audio channel to a minimum value of no more than 0.1 to maximize the use of lip movement information from the visual channel for transcription and avoid severely contaminated audio features from interfering with the recognition results.

[0101] It should be noted that, although this embodiment uses a method based on... Weight mapping is performed using an inverse square function of the form [formula missing], but in other embodiments, a method based on [formula missing] can also be used. The generalized power-law inverse proportional function achieves the same gating effect, where the power... The value can be selected within the range of 1.5 to 4.0, depending on the desired steepness of the decision boundary.

[0102] In another embodiment, when the target speaker's face is obscured, they turn away from the camera's field of view, or the camera itself experiences a hardware malfunction, the system detects that the visual channel is unavailable and automatically switches to a degraded operating mode that relies solely on the audio channel. In this mode, the gating weight is forcibly set to... , The system is configured with a traditional single-modal speech-to-text workflow to ensure basic transcription services are maintained even when visual information is missing. Once the visual channel becomes available again, the system automatically reverts to a dual-channel collaborative working mode after detecting five consecutive valid lip localization results.

[0103] S5. After performing dimensional alignment of audio features and lip movement features through a fully connected mapping layer, frame-by-frame weighted fusion is performed based on gated weights to construct a multimodal joint feature sequence.

[0104] First, the system reads the Mel spectrum vector from step S201. Read the lip movement temporal feature vector from step S302. Read the audio channel confidence weights from step S402. With visual channel confidence weight .

[0105] Due to the dimension of the Mel spectrum vector Dimension of lip movement temporal feature vector Inconsistent in this embodiment For 40 The value is 256, requiring dimension alignment between the two. The system will use the Mel spectrum vector... Projected through a fully connected mapping layer to In 3D space, we obtain the audio embedding vector. The dimension of the weight matrix of the fully connected mapping layer is... The bias vector dimension is This mapping layer is jointly optimized end-to-end with the backend transcription model during the system training phase.

[0106] Then, the feature vectors of the two channels are weighted and fused frame by frame using the gating weights obtained in step S402:

[0107] ;

[0108] in This is the fused multimodal feature vector; This is the dimension-aligned audio embedding vector; This is the lip movement temporal feature vector; and These are the confidence weights for the audio and visual channels, respectively, and their sum is always 1.

[0109] In clean audio frames Approaching 1, fusion feature Primarily driven by acoustic information, it retains the high-frequency resolution recognition advantage of the audio channel under high signal-to-noise ratio conditions; in multi-speaker overlapping speech frames, Approaching 1, fusion feature The system shifts to lip movement information as the primary driver, leveraging the inherent physical immunity of the visual channel to acoustic aliasing to provide uncontaminated pronunciation cues for the backend recognition model.

[0110] The multimodal fusion feature sequence obtained after gated weighting is obtained through the above processing system. .

[0111] like Figure 4As shown, the gated and weighted multimodal fusion feature sequence exhibits a feature distribution pattern corresponding to the target speaker's vocal activities in the time dimension. During the period of continuous vocalization by the target speaker, each dimension of the fusion features shows regular periodic changes, reflecting the dynamic features of pronunciation co-encoded by lip movement information and acoustic information. During periods of pause or silence, the amplitude and variation of each feature dimension decrease significantly. This feature sequence serves as the input to the self-attention sequence transcription model in subsequent step S6, providing an effective representation of the target speaker's vocal content throughout the entire time period.

[0112] S6. The gated weighted fusion feature sequence is fed into an end-to-end sequence transcription model based on a self-attention mechanism to output the real-time transcribed text of the target speaker in a streaming manner.

[0113] First, the system outputs the multimodal fusion feature sequence from step S5. The input is fed into an end-to-end sequence transcription model based on a self-attention mechanism. In this embodiment, the transcription model adopts an encoder-decoder architecture. The encoder contains 6 multi-head self-attention coding layers, with 8 attention heads per layer, a hidden layer dimension of 512, and a feedforward network intermediate dimension of 2048. The decoder also contains 6 layers with the same structure as the encoder, and adds an extra cross-attention sublayer to each layer for querying the encoder's output.

[0114] Furthermore, the encoder performs self-attention encoding on the fused feature sequence to capture long-range temporal dependencies between frames. Since the input fused features have undergone cross-modal consistency gating verification in step S104, the contaminated audio features in overlapping speech segments with multiple speakers have been proportionally replaced and compensated by the lip movement features of the visual channel. Therefore, the feature sequence received by the encoder maintains an effective representation of the target speaker's pronunciation content throughout the entire time period.

[0115] The decoder then generates the character sequence of the target text stepwise using an autoregressive approach. In this embodiment, the decoder's vocabulary size is preferably set to include 4233 commonly used Chinese characters and punctuation marks, but the vocabulary size can be adjusted between 2000 and 10000 depending on the language used. At each time step, the decoder outputs the probability distribution of the current character, selects the character with the highest probability as the output of the current step, and feeds it back to the input of the next time step.

[0116] During the inference phase, steps S1 to S5 are executed frame-by-frame in a pipelined manner, with the video frame rate as the beat. In step S6, the decoder starts beam search decoding after accumulating no less than 20 frames of encoded features. The beam width is preferably set to 5, but it can be adjusted within the range of 3 to 10 depending on the balance between decoding speed and transcription accuracy. The above pipelined processing mechanism realizes end-to-end streaming real-time transcription from audio and video acquisition to text output. The single-frame processing latency measured on the hardware platform of this embodiment does not exceed 30 milliseconds, which meets the latency requirements for real-time interaction.

[0117] The training process of the transcription model uses an audio-video aligned speech-to-text dataset containing no less than 1000 hours of data, of which at least 30% of the training samples are recordings of multi-person conference scenarios containing overlapping speech segments from two or more speakers. The optimizer employs an adaptive moment estimation optimizer, with a warm-up-cosine annealing scheduling strategy for the learning rate. The warm-up phase consists of the first 4000 steps, the peak learning rate is set to 0.0003, and the total number of training steps is set to 200,000. The loss function is a label-smoothed cross-entropy loss with a smoothing coefficient set to 0.1.

[0118] Specifically, if the system does not detect any valid lip movement activity for more than 30 consecutive frames and the acoustic aliasing index of the audio channel remains at a low value, it is determined that the current scene may be in a completely silent state. At this time, the decoder pauses output and enters a low-power waiting mode until any channel detects a new activity signal and resumes operation, so as to reduce the system's computational resource consumption in invalid input segments.

[0119] In another embodiment, in a conference room deployment scheme including multiple cameras, each camera is pointed at the face area of ​​one participant. The system independently executes all steps S3 to S6 for each speaker. The cross-modal consistency deviation ratio and gating weight of each speaker are calculated and updated independently, thereby achieving parallel real-time transcription of the speech content of multiple speakers. In this embodiment, the audio channel signal is data collected by the same microphone array shared by all cameras, while the lip movement information of the visual channel is specific to each speaker. Thus, the semantic contribution of each speaker in the shared audio is extracted separately through the differentiated information of the visual channel.

[0120] like Figure 5As shown, throughout the entire test time series, the frame-by-frame character error rates of the three methods remained at a low level with little difference in the clean speech segments. However, upon entering the multi-speaker overlapping segment, only the audio baseline method showed a sharp increase in character error rate. While the fixed-weight fusion method mitigated this somewhat, it remained significantly higher than the baseline. Our proposed method, through a dynamic gating mechanism, maintained a relatively low character error rate in the overlapping segment. Once out of the overlapping segment, the character error rates of all three methods quickly returned to the level of the clean speech segment. This time-series comparison demonstrates that our cross-modal consistency gating mechanism effectively suppresses the negative impact of contaminated audio features on transcription accuracy when multi-speaker overlap occurs.

[0121] like Figure 6 As shown, in the overall scenario, the character error rate of the audio baseline method alone is 20.2%, the fixed weight fusion method is 12.8%, and our method is 7.2%. In clean speech segments, the character error rates of the three methods are relatively similar, at 3.1%, 3.4%, and 3.0%, respectively, indicating that our method does not compromise the recognition accuracy of the audio channel under interference-free conditions due to the introduction of the visual channel. In overlapping speech segments, the character error rate of the audio baseline method alone is as high as 46.8%, the fixed weight fusion method is 27.4%, and our method is only 13.7%, achieving a relative reduction of approximately 70.7% compared to the audio baseline method alone. The above comparison results verify the transcription accuracy advantage of our method in multi-speaker overlapping scenarios.

[0122] like Figure 7 As shown, the character error rates of all three methods decreased as the signal-to-noise ratio (SNR) gradually increased from -5dB to 25dB. Under low SNR conditions, only the audio baseline method had the highest character error rate, followed by the fixed-weight fusion method. Our proposed method consistently maintained the lowest character error rate, and the differences between the three methods were more significant in the low SNR range. As the SNR increased to above 20dB, the character error rates of the three methods gradually converged. This trend indicates that our proposed method has stronger robustness under high noise interference conditions, and its dynamic gating mechanism can effectively utilize lip movement information from the visual channel for compensation when the audio signal quality deteriorates.

[0123] An example of a real-time speech-to-text system based on multimodal deep learning:

[0124] On the other hand, the present invention also provides a real-time speech-to-text system based on multimodal deep learning. For example... Figure 8 As shown, a real-time speech-to-text system based on multimodal deep learning includes a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement a real-time speech-to-text method based on multimodal deep learning according to the first aspect of the present invention.

[0125] A real-time speech-to-text system based on multimodal deep learning also includes other components well known to those skilled in the art, such as communication interfaces. Their settings and functions are known in the art and will not be described in detail here.

[0126] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this invention can be implemented using computer-readable / executable instructions stored or otherwise maintained by such a computer-readable medium.

Claims

1. A real-time speech-to-text method based on multimodal deep learning, characterized in that, The method includes: performing timestamp injection on the acquired audio stream and video stream, segmenting the audio stream based on the video frame time interval, and extracting lip region images from the video frames to obtain frame-aligned audio frame sequences and lip image frame sequences. The audio frame sequence is subjected to frequency domain transformation frame by frame, and Mel spectrum vectors are extracted. Spectral entropy is calculated based on the normalized energy proportion of each Mel sub-band as an acoustic aliasing index, which characterizes the dispersion of spectral energy among sub-bands in the current frame. For the lip image frame sequence, spatial feature encoding is extracted using a convolutional neural network, and temporal feature vectors are extracted using a recurrent neural network. The lip movement saliency index is calculated based on the mean Euclidean distance between the spatial feature encodings of adjacent frames within a sliding window, which characterizes the activity level of the target speaker's lip movements. Acoustic aliasing index of frames for: ； , , For the first Frame number The normalized energy percentage and energy value of each Mel subband; For the first Frame number The energy value of each melon subband; This represents the total number of channels in the Mel filter bank; No. Frame lip movement significance index for: ； This is the length of the time series observation window; , The first in the window Frame, First The lip spatial feature encoding vector of the frame; Represents the Euclidean norm; After normalizing the acoustic aliasing index and the lip movement significance index respectively, their ratio is calculated to obtain the cross-modal consistency deviation ratio. The confidence weights of the audio channel and the visual channel are obtained through inverse square mapping. The confidence weights of the audio channel and the visual channel are used to perform weighted fusion of the dimension-aligned Mel spectrum vector and the lip movement temporal feature vector. The fused multimodal feature sequence is fed into the end-to-end sequence transcription model to output the real-time transcribed text of the target speaker. No. Frame cross-modal consistency deviation ratio for: ； This represents the theoretical upper limit of the acoustic aliasing index. This serves as a reference benchmark for the significance of lip movements obtained during the calibration phase. To prevent overflow of tiny positive numbers when divided by zero; The confidence weights for the audio channel and the visual channel are as follows: ； , The confidence weights assigned to the audio and visual channels.

2. The real-time speech-to-text method based on multimodal deep learning according to claim 1, characterized in that, The method for obtaining the lip movement significance reference benchmark is as follows: During the system initialization phase, the target speaker reads a standard text of no less than 30 seconds in an interference-free environment. The lip movement salience index values ​​of all speech activity frames are collected, and the 50th percentile value is taken as the reference benchmark for lip movement salience. .

3. The real-time speech-to-text method based on multimodal deep learning according to claim 1, characterized in that, When the cross-modal consistency deviation ratio The value continuously exceeds the extreme aliasing threshold for more than a preset number of consecutive frames. At that time, the confidence weight of the audio channel is set to not exceed a preset minimum value; the extreme aliasing threshold The value range is 2.0 to 5.0, and the preset minimum value is 0.

1.

4. The real-time speech-to-text method based on multimodal deep learning according to claim 1, characterized in that, The convolutional neural network adopts a backbone network structure based on depthwise separable convolution, which contains multiple convolutional modules cascaded in sequence. Each convolutional module consists of a depthwise separable convolutional layer, a batch normalization layer, and a linear rectified activation function. The output of the convolutional module is compressed into a fixed-dimensional spatial feature encoding vector by a global average pooling layer.

5. The real-time speech-to-text method based on multimodal deep learning according to claim 1, characterized in that, The method also includes a visual channel failure degradation step: If the lip region fails to be located within a preset number of consecutive frames, the visual channel is marked as unavailable, and the confidence weight of the audio channel is forcibly set to 1 and the confidence weight of the visual channel is forcibly set to 0. Once a preset number of valid lip positioning results are detected again, the dual-channel collaborative working mode is restored.

6. A real-time speech-to-text system based on multimodal deep learning, characterized in that, It includes a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, it implements the real-time speech transcription method based on multimodal deep learning as described in any one of claims 1 to 5.

Images