Voice recognition system and method based on audio-visual dual-mode perception

Abstract

The invention discloses a voice recognition system and method based on audiovisual bimodal perception, and relates to the technical field of voice recognition, and the method comprises the steps: carrying out the feature recursion of an aligned bimodal sequence based on an audiovisual frame package, obtaining an audio high-level representation and a visual high-level representation, and calculating the modal reliability; and fusing the audio high-level representation and the visual high-level representation based on the modal reliability, and evaluating the prefix stability and confidence state through a streaming decoder to obtain a streaming identification information packet. According to the method, the audio high-level representation and the visual high-level representation are fused based on the modal reliability, and the streaming decoder is used for evaluating the prefix stability and the confidence state to obtain the streaming identification information packet, so that the decoding output has the confirmable stable fragment and the credibility representation, and the continuity and the consistency of the speech identification text are improved.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a speech recognition system and method based on audiovisual bimodal perception. Background Technology

[0002] Speech recognition belongs to the field of speech signal processing and pattern recognition. In recent years, it has often been combined with cameras to obtain mouth movement information to form a dual-modal audio-visual recognition process: audio and video are acquired simultaneously and then time-stamped. Gain and noise reduction processing are performed on the audio side, and the face is located and the mouth area is cropped on the video side. After feature encoding, the text is jointly modeled in a fusion network and output by streaming decoding. It is suitable for real-time scenarios such as conference transcription and in-vehicle interaction.

[0003] In streaming scenarios, conventional solutions often use fixed synchronization or short window alignment, which can easily lead to inconsistencies in audiovisual timing when faced with clock drift and frame rate jitter. The fusion weights are mostly static or only change with the quality score, lacking a feedback loop driven by prefix stability and confidence state, making it difficult to balance output stability and adaptive acquisition. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a speech recognition method based on audiovisual dual-modal perception to solve the problems of audiovisual alignment mismatch due to drift and insufficient stability of fusion output in the process of streaming recognition, which makes it difficult to achieve adaptive acquisition and adjustment.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a speech recognition method based on audiovisual bimodal perception, which includes: collecting bimodal data and performing unified timestamp alignment and cache maintenance to generate audiovisual frame packets; Based on audio-visual frame packets, audio enhancement frame blocks and mouth normalization sequences are obtained and their quality is evaluated to generate quality-annotated audio-visual frame packets. The audio-visual alignment offset is estimated based on the quality-labeled audio-visual frame packets, and the audio enhancement frame blocks and mouth normalization sequences are temporally resampled and aligned sliced ​​to generate an aligned bimodal sequence. Feature recursion is performed on the aligned bimodal sequence to obtain high-level audio and visual representations, and modal reliability is calculated. Based on modal reliability, the audio high-level representation and the visual high-level representation are fused, and the prefix stability and confidence state are evaluated by a streaming decoder to obtain the streaming recognition information packet. Post-processing of the text based on the streaming recognition information packet generates speech recognition text and perception scheduling instructions, and dynamically adjusts the acquisition parameters of the bimodal data.

[0007] As a preferred embodiment of the speech recognition method based on audiovisual dual-modal perception described in this invention, the audiovisual frame packet includes timestamp-aligned audio frames and video frames; The aligned bimodal sequence includes an aligned audio sequence and a mouth sequence; The streaming identification information package includes incremental text, stable text, prefix stability, and confidence state.

[0008] As a preferred embodiment of the speech recognition method based on audiovisual bimodal perception described in this invention, the steps for generating audiovisual frame packets are as follows: Establish a unified timestamp benchmark, initialize the audio and video sliding buffers, and generate a unified timestamp to mark the acquisition time; Audio and video data are collected based on a unified timestamp and written to the audio sliding buffer and video sliding buffer respectively, generating audio frames and video frames with timestamps. Perform timestamp alignment on audio and video frames with timestamps, update the audio and video sliding buffers, and assemble frame packets to generate audiovisual frame packets.

[0009] As a preferred embodiment of the speech recognition method based on audiovisual bimodal perception described in this invention, the steps of acquiring audio enhancement frame blocks and lip normalization sequences and performing quality assessment are as follows: Gain stabilization and noise suppression are performed on the timestamp-aligned audio frames to obtain audio enhancement frame blocks, and audio quality metrics are calculated based on speech activity detection and signal-to-noise ratio estimation. Face localization and mouth region localization are performed on video frames aligned with timestamps, and the mouth regions are aligned and normalized to obtain a standardized mouth sequence. Sharpness and occlusion were assessed based on the standardized oral sequence, and visual quality indicators were calculated.

[0010] As a preferred embodiment of the speech recognition method based on audiovisual bimodal perception described in this invention, the steps of temporal resampling and aligning slices of the audio enhancement frame block and the lip normalization sequence are as follows: Based on audio enhancement frame blocks and mouth normalization sequences, audio energy change sequences and mouth motion intensity sequences are obtained within the same time window; The correlation between audio energy change sequences and mouth movement intensity sequences was obtained by using the cross-correlation peak search method. The correlation results were then filtered by combining audio quality indicators and visual quality indicators to identify audiovisual alignment offset. Based on the audio-visual alignment offset, the audio enhancement frame block is temporally resampled to obtain an audio resampled frame block aligned with the video timeline; Based on the audio-visual alignment offset, the mouth normalization sequence is time-aligned and resampled to obtain a mouth resampled sequence consistent with the audio time axis; Alignment slicing is performed on the same time grid based on the audio resampled frame block and the mouth resampled sequence to generate an aligned bimodal sequence.

[0011] As a preferred embodiment of the speech recognition method based on audiovisual bimodal perception described in this invention, the steps for obtaining the high-level audio representation and the high-level visual representation are as follows: Based on the aligned audio sequence, streaming feature recursion is performed by combining sequence causal convolutional coding and gated recurrent units, and the intermediate state of acoustic recursion is written into the recursion state of acoustic encoder to obtain high-level audio representation. Based on the aligned mouth sequence, spatiotemporal 3D convolutional coding and LSTM are combined to perform spatiotemporal feature recursion, and the intermediate visual recursion state is written into the recursion state of the visual encoder to obtain the high-level visual representation.

[0012] As a preferred embodiment of the speech recognition method based on audiovisual bimodal perception described in this invention, the steps for calculating modal reliability are as follows: The audio modal reliability is calculated by combining audio quality metrics and high-level audio characterization. Visual modal reliability is calculated by combining high-level visual representations and visual quality indicators.

[0013] As a preferred embodiment of the speech recognition method based on audiovisual bimodal perception described in this invention, the steps for evaluating prefix stability and confidence state using a streaming decoder are as follows: Alignment correction of audio modal reliability and visual modal reliability is performed on the same time grid, and interference from low-reliability modalities is suppressed by dynamic weighting and gated fusion to obtain fusion characterization; The prefix beam search method is used to perform streaming decoding on the fusion representation to obtain the candidate path set and the incremental text of the current time period, and the posterior probability distribution of each candidate path is recorded simultaneously to generate decoding posterior statistics. Prefix consistency detection is performed on incremental text within a sliding window over consecutive time periods, and the degree of divergence in the candidate path set is used as an auxiliary criterion to obtain prefix stability, which characterizes whether the current output has converged. Based on prefix stability, the incremental text is confirmed to be a stable segment, and the prefix segments that meet the stability condition are solidified as stable text, while the tail segments that do not meet the stability condition are retained as incremental parts that can continue to be updated. Based on the confidence state of the decoding post-hoc statistical analysis, the confidence state is associated with the incremental text, stable text and prefix stability by timestamp and encapsulated to form a streaming recognition information packet.

[0014] As a preferred embodiment of the speech recognition method based on audiovisual bimodal perception described in this invention, the steps of performing text post-processing based on streaming recognition information packets to generate speech recognition text and perception scheduling instructions, and dynamically adjusting the acquisition parameters of bimodal data, are as follows: The stable text is processed by punctuation restoration, number format standardization, and word form normalization. The processed stable text and incremental text are then combined to generate speech recognition text. The output mode of speech recognition text is controlled based on prefix stability and confidence state to generate scheduling decision information; Based on scheduling judgment information, audio quality indicators, and visual quality indicators, perception scheduling instructions are obtained, and the acquisition parameters of dual-modal data are dynamically adjusted.

[0015] Secondly, the present invention provides a speech recognition system based on audiovisual dual-modal perception, including a data acquisition module, which acquires dual-modal data and performs unified timestamp alignment and cache maintenance to generate audiovisual frame packets; The quality assessment module, based on audio-visual frame packets, acquires audio enhancement frame blocks and mouth normalization sequences and performs quality assessment to generate quality-annotated audio-visual frame packets. The offset estimation module estimates the audio-visual alignment offset based on the quality-annotated audio-visual frame packets, and performs temporal resampling and alignment slicing on the audio enhancement frame blocks and the mouth normalization sequence to generate an aligned bimodal sequence. A reliable evaluation module performs feature recursion on the aligned bimodal sequence to obtain high-level audio and visual representations and calculate modal reliability. The audio-visual fusion module fuses the high-level audio representation and the high-level visual representation based on modal reliability, and evaluates the prefix stability and confidence state through a streaming decoder to obtain streaming recognition information packets. The optimization module performs post-processing on the streaming recognition information packets to generate speech recognition text and perception scheduling instructions, and dynamically adjusts the acquisition parameters of the dual-modal data. The beneficial effects of this invention are as follows: based on the quality-annotated audio-visual frame packets, the audio-visual alignment offset is estimated, and the audio enhancement frame blocks and the mouth normalization sequence are temporally resampled and aligned sliced ​​to generate an aligned bimodal sequence, thereby achieving synchronous input of audio-visual information on a unified time grid; subsequently, based on modal reliability, the audio high-level representation and the visual high-level representation are fused, and the prefix stability and confidence state are evaluated by a streaming decoder to obtain streaming recognition information packets, so that the decoding output has a verifiable stable segment and a reliable representation, thereby improving the continuity and consistency of speech recognition text. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a speech recognition method based on audiovisual bimodal perception.

[0018] Figure 2 This is a schematic diagram of a speech recognition system based on audiovisual dual-modal perception.

[0019] Figure 3 A flowchart for generating the aligned bimodal sequence.

[0020] Figure 4 This is a flowchart for dynamically adjusting the acquisition parameters of dual-modal data. Detailed Implementation

[0021] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0023] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0024] Reference Figures 1-4 This is one embodiment of the present invention, which provides a speech recognition method based on audiovisual bimodal perception, including the following steps: S1. Collect dual-modal data and perform unified timestamp alignment and cache maintenance to generate audiovisual frame packets.

[0025] Establish a unified timestamp benchmark, initialize the audio and video sliding buffers, and generate a unified timestamp to mark the acquisition time.

[0026] Furthermore, establishing a unified timestamp benchmark includes reading a local high-precision timing source and setting the current timing value as the starting reference point for the unified timestamp. Simultaneously, it determines the increment granularity of the unified timestamp and converts the timing values ​​of subsequent acquisition trigger moments into unified timestamps, writing them into the timestamp record table. Initializing the audio sliding buffer includes allocating a circular buffer space and configuring a unified timestamp field in each buffer unit for subsequent writing of timestamped audio frames and supporting retrieval by unified timestamp during timestamp alignment. Initializing the video sliding buffer includes allocating a circular buffer space and configuring a unified timestamp field in each buffer unit for subsequent writing of timestamped video frames and supporting retrieval by unified timestamp during timestamp alignment. Upon completion, a unified timestamp marking the acquisition moment is generated as the time stamp basis for the next step of acquiring audio and video data.

[0027] Audio and video data are collected based on a unified timestamp and written to the audio sliding buffer and video sliding buffer respectively, generating audio frames and video frames with timestamps.

[0028] Furthermore, the process of acquiring audio and video data based on a unified timestamp includes reading the corresponding unified timestamp from the timestamp record table and binding it to the current acquisition window at each acquisition trigger. Then, audio data is continuously acquired within the acquisition window and segmented into audio frames according to a preset frame length. Simultaneously, the unified timestamp is written to the timestamp field of the audio frame, and the audio frames are written to the audio sliding buffer in the order of the write pointer so that audio frames can be extracted according to the unified timestamp index in the subsequent timestamp alignment stage. Similarly, video data is acquired within the same acquisition window, and video frames are extracted according to the frame rate. The unified timestamp is written to the timestamp field of the video frame, and the video frames are written to the video sliding buffer in the order of the write pointer so that video frames can be extracted according to the unified timestamp index in the subsequent timestamp alignment stage. After writing is complete, timestamped audio and video frames are generated as input for the next step of performing timestamp alignment on the timestamped audio and video frames.

[0029] Perform timestamp alignment on audio and video frames with timestamps, update the audio and video sliding buffers, and assemble frame packets to generate audiovisual frame packets.

[0030] Furthermore, performing timestamp alignment on timestamped audio and video frames involves reading timestamped audio frames from the audio sliding buffer and timestamped video frames from the video sliding buffer. The timestamped audio frames and timestamped video frames are matched according to a unified timestamp to determine the set of paired frames corresponding to the same unified timestamp. After matching, audio frames with timestamps earlier than the smallest unmatched unified timestamp are removed from the audio sliding buffer, and video frames with timestamps earlier than the smallest unmatched unified timestamp are removed from the video sliding buffer, thus updating the audio and video sliding buffers. Subsequently, the paired frame sets are assembled into audiovisual frame packets according to the unified timestamp order and output for subsequent quality assessment.

[0031] The audio-visual frame package includes timestamp-aligned audio and video frames.

[0032] S2. Based on the audio-visual frame packets, obtain audio enhancement frame blocks and mouth normalization sequences and perform quality assessment to generate quality-annotated audio-visual frame packets.

[0033] Gain stabilization and noise suppression are performed on the timestamp-aligned audio frames to obtain audio enhancement frame blocks. Based on speech activity detection and signal-to-noise ratio estimation, an audio quality index characterizing the current audio recognizability is calculated, expressed as: ; in, This indicates an audio quality metric. This indicates the voice activity detection result (1 if there is voice, 0 if there is no voice). This represents the signal-to-noise ratio estimate. Smax represents the lower threshold of the signal-to-noise ratio (SNR), while Smax represents the upper threshold of the SNR. This means limiting the result to between 0 and 1; Furthermore, the timestamp-aligned audio frames undergo amplitude normalization and dynamic gain adjustment to suppress steady-state and transient noise components, ensuring stable speech energy and reduced background interference within a uniform amplitude range. After processing, these frames are aggregated by time window to form audio enhancement frame blocks. Speech activity detection determines the speech presence state on the timestamp-aligned audio frames and outputs the speech activity detection results. Signal-to-noise ratio (SNR) estimation estimates the relationship between speech energy and noise energy on the audio enhancement frame blocks and outputs the SNR estimate. Subsequently, based on the speech activity detection results, the SNR estimate is normalized and constrained to a fixed range (e.g., the SNR estimate is linearly normalized using a lower and upper SNR threshold and limited to the range of 0–1 using a pruning function). This calculates an audio quality index characterizing the current audio recognizability. The audio enhancement frame blocks and audio quality index are directly used as components of subsequent quality-annotated audiovisual frame packets.

[0034] It should be noted that the dynamic gain adjustment and suppression of steady-state and transient noise components are specifically as follows: Short-time Fourier transform is used to calculate the short-time energy of the timestamp-aligned audio frame, and the short-time energy is mapped to the target gain coefficient through proportional-integral update based on the difference between the target energy level and the short-time energy to perform amplitude stretching or compression on the timestamp-aligned audio frame, thereby completing the dynamic gain adjustment. At the same time, the noise power spectrum is estimated in the non-speech region, and spectral subtraction is used to perform spectral domain noise suppression on the timestamp-aligned audio frame to reduce steady-state noise components. Then, transient impact detection is performed on the timestamp-aligned audio frame, and amplitude limiting and smoothing processing is performed on the detected impact segment to suppress transient noise components.

[0035] and Determining the offline calibration set based on the target device and target scenario: First, calculate the signal-to-noise ratio estimate for each speech segment on the calibration set. Then, combining the speech activity detection results, the distribution statistics of the "reliably identifiable" sample subset are performed, using low quantiles as... (Exemplary value range: 0–5 dB), using high quantiles as (Example value range: 20–30 dB), making the normalized audio quality metric more sensitive to changes in the main operating range and less prone to saturation.

[0036] Determining the presence of speech specifically refers to: calculating short-time energy on the timestamp-aligned audio frames, estimating the noise floor, and then performing energy dominance discrimination to output the speech activity detection result. For example, when the short-time energy is consistently higher than the noise floor, the speech activity detection result is "speech present," while when the short-time energy is close to the noise floor, the speech activity detection result is "no speech present."

[0037] Face localization and mouth region localization are performed on video frames aligned with timestamps, and the mouth regions are aligned and normalized to obtain a standardized mouth sequence. Furthermore, a convolutional neural network detector is used to detect candidate face regions in the timestamp-aligned video frames and output face localization results. Within the area defined by the face localization results, feature point regression is used to extract mouth structure information such as lip boundaries and mouth corner positions, and mouth region localization results are output. The mouth region is aligned and normalized, including calculating the rotation and scaling correction relationship of the mouth region based on the mouth region localization results and performing geometric correction on the mouth region. At the same time, the mouth region is cropped to a fixed size and brightness and contrast normalization are performed to obtain the mouth standardization result corresponding to each frame. The mouth standardization results of consecutive frames are spliced ​​together in a unified timestamp order to form a mouth standardization sequence.

[0038] It should be noted that a convolutional neural network detector refers to a target detection network with convolutional operations as its core. It determines the presence and location of a face by extracting multi-level spatial features from video frames after time-stamp alignment and performing position regression and classification scoring on the feature map. The face candidate region detection process involves extracting convolutional features from the video frames after time-stamp alignment and outputting candidate boxes and confidence scores at multiple scales. Then, the candidate boxes are overlapped, merged, and filtered to obtain face candidate regions and output the face localization results.

[0039] Based on the standardized oral sequence, sharpness and occlusion are assessed, and a visual quality index representing the current oral visibility is calculated. The expression is as follows: ; in, It is a visual quality indicator. It is the clarity score of the mouth region (e.g., the Laplacian variance of the grayscale image of the mouth region). It is the lower limit threshold for image sharpness. It is the upper limit threshold of resolution. It represents the number of pixels obscured in the mouth area. This refers to the total number of pixels in the mouth area. This is the statistical value of the number of frames where mouth region localization failed (within the time window). It is the total number of frames within the time window; Furthermore, the Laplacian response is calculated on the grayscale image of the mouth region in each frame of the mouth normalization sequence, and variance statistics are performed on the Laplacian response to obtain the mouth region sharpness score. The sharpness evaluation results are then summarized within the time window. Simultaneously, occluded pixels are identified within the mouth region of each frame based on brightness saturation features and texture response attenuation features, and the number of occluded pixels in the mouth region is counted in relation to the total number of pixels in the mouth region to obtain the occlusion degree evaluation result. Within the time window, the number of frames in which mouth region localization fails is counted in relation to the total number of frames within the time window to obtain the localization stability evaluation result. Subsequently, the sharpness evaluation result is subjected to range restriction normalization processing and multiplicatively fused with the occlusion degree evaluation result and the localization stability evaluation result to obtain the visual quality index.

[0040] It should be noted that the lower and upper thresholds for sharpness are defined based on the statistical distribution of sharpness scores corresponding to the standardized oral sequence under the target device and acquisition parameter conditions, using quantile statistics from the offline calibration set. The exemplary value ranges are: [values ​​to be filled in] within the oral region, with grayscale values ​​ranging from zero to one, and using Laplace variance as [value to be filled in]. hour, Take the powers of 2 to 10 -4 up to 8 to 10 -4. Take the power of 3 x 10 to the power of -3 up to 1 x 10 to the power of -2.

[0041] The quality-annotated audiovisual frame package includes audio enhancement frame blocks, audio quality metrics, mouth normalization sequences, and visual quality metrics.

[0042] S3. Estimate the audio-visual alignment offset based on the quality-labeled audio-visual frame packets, and perform temporal resampling and alignment slicing on the audio enhancement frame blocks and the mouth normalization sequence to generate the aligned bimodal sequence.

[0043] Based on audio enhancement frame blocks and mouth normalization sequences, audio energy change sequences and mouth motion intensity sequences are obtained within the same time window.

[0044] Furthermore, obtaining the audio energy change sequence and mouth motion intensity sequence within the same time window based on the audio enhancement frame block and the mouth normalization sequence includes dividing the audio enhancement frame block into continuous analysis frames according to a unified timestamp and calculating the short-time energy on each analysis frame to obtain the audio energy sequence. Then, the short-time energy of adjacent analysis frames is differentially and smoothed to obtain the audio energy change sequence. The mouth normalization sequence is divided into continuous video frames according to a unified timestamp and the pixel difference energy of the mouth region is calculated between adjacent video frames to obtain the mouth motion sequence. Then, the mouth motion sequence is temporally smoothed to obtain the mouth motion intensity sequence.

[0045] The correlation between audio energy change sequences and mouth movement intensity sequences was obtained by using a cross-correlation peak search method. The correlation results were then filtered by combining audio quality indicators and visual quality indicators to identify audiovisual alignment offset.

[0046] Furthermore, the audio energy change sequence and mouth motion intensity sequence are standardized within the same time window. Candidate delay ranges are set based on the frame rate of the standardized mouth sequence and the frame shift relationship of the audio enhancement block. The cross-correlation peak search method is used to calculate the cross-correlation score under different delay conditions within the candidate delay range to form a correlation result. The delay with the largest cross-correlation score is selected as the candidate audiovisual alignment offset. The credibility of the candidate audiovisual alignment offset is weighted and verified based on audio quality index and visual quality index, and the correlation results corresponding to the time period with insufficient quality are removed. Then, peak consistency verification is performed on the retained correlation results to determine the stable audiovisual alignment offset. It should be noted that before the audio energy change sequence and the mouth movement intensity sequence are included in the cross-correlation calculation, they are both normalized to unify their dimensions.

[0047] Based on the audio-visual alignment offset, the audio enhancement frame block is temporally resampled to obtain an audio resampled frame block aligned with the video timeline.

[0048] Furthermore, the audio-visual alignment offset is mapped to the starting alignment position of the audio enhancement frame block on a unified timestamp, and the audio enhancement frame block is time-shifted according to the audio-visual alignment offset to correct the synchronization relationship between the audio enhancement frame block and the mouth normalization sequence. Then, the audio enhancement frame block is resampled according to the sampling rhythm of the video timeline. For sampling points that cannot be divided by the alignment position, an interpolation method is used to generate aligned sampling values ​​while maintaining the continuity of the audio enhancement frame block. After resampling, the audio enhancement frame block is reframed according to the video timeline and assembled to form an audio resampled frame block aligned with the video timeline.

[0049] Based on the audio-visual alignment offset, the normalized mouth sequence is time-aligned and resampled to obtain a mouth resampled sequence consistent with the audio timeline.

[0050] Furthermore, a unified timestamp index conversion is used to map the audiovisual alignment offset to the starting alignment position of the mouth normalization sequence on a unified timestamp, and the mouth normalization sequence is time-shifted according to the audiovisual alignment offset to correct the synchronization relationship between the mouth normalization sequence and the audio enhancement frame block; then, the target sampling time of the mouth normalization sequence is determined according to the sampling rhythm of the audio timeline, and the corresponding adjacent video frame position is selected for each target sampling time in the mouth normalization sequence. For target sampling times that cannot be directly hit (such as target sampling times falling between adjacent video frames), an interpolation method is used to generate aligned sampling frames while maintaining the temporal continuity of the mouth normalization sequence; after resampling, the aligned sampling frames are spliced ​​according to the audio timeline order to obtain a mouth resampling sequence consistent with the audio timeline.

[0051] Alignment slicing is performed on the same time grid based on the audio resampled frame block and the mouth resampled sequence to generate an aligned bimodal sequence.

[0052] Furthermore, a unified time grid is established and a unified timestamp is used as the slice index. At each unified time grid position, the corresponding audio resampled frame block segment and the corresponding mouth resampled sequence segment are read. Then, the audio resampled frame block segment and the mouth resampled sequence segment are synchronously extracted according to the unified time grid to form a set of paired slices under the same slice number. During the slice alignment process, the missing positions are filled by proximity padding or interpolation to ensure that each slice number contains both audio resampled frame block segment and mouth resampled sequence segment. After completion, the paired slice sets are spliced ​​and assembled according to the slice number order to generate the aligned bimodal sequence.

[0053] The aligned bimodal sequence includes the aligned audio sequence and the mouth sequence.

[0054] S4. Perform feature recursion on the aligned bimodal sequence to obtain high-level audio and visual representations, and calculate modal reliability. Based on the aligned audio sequence, streaming feature recursion is performed by combining sequence causal convolutional coding and gated recurrent units, and the intermediate acoustic recursion state is written into the recursion state of the acoustic encoder to obtain the high-level audio representation.

[0055] Furthermore, based on the aligned audio sequence, frame segments at consecutive moments are fed into sequential causal convolutional coding in a uniform time grid order to extract temporal local features that depend only on historical frame segments (such as short-term dynamic patterns formed by energy envelope fluctuations and frequency band amplitude changes of adjacent frame segments) while maintaining causal constraints. The temporal local features output by sequential causal convolutional coding are input into a gated recurrent unit step by step to perform streaming feature recursion and fuse long-range context across frame segments. The gated recurrent unit generates an acoustic recursion intermediate state at each time step and writes the acoustic recursion intermediate state into the acoustic encoder recursion state to maintain the recursion continuity between consecutive windows. The current output of the gated recurrent unit and the acoustic encoder recursion state together form a high-level audio representation that can characterize the current speech content and context dependence, and maintain the recursion connection with the audio sequence aligned to the next time step.

[0056] It should be noted that causal constraint means that sequential causal convolutional coding uses only current and historical frame fragment information at any time step, without using future frame fragment information. The acoustic recursive intermediate state refers to the instantaneous hidden representation calculated by the gated recurrent unit at the current time step from the aligned audio sequence and previous recursive information. The acoustic encoder recursive state refers to the memory representation that continuously stores the acoustic recursive intermediate state and propagates it across time frames to support the streaming feature recursion of subsequent time steps.

[0057] Based on the aligned mouth sequence, spatiotemporal 3D convolutional coding and LSTM are combined to perform spatiotemporal feature recursion, and the intermediate visual recursion state is written into the recursion state of the visual encoder to obtain the high-level visual representation.

[0058] Furthermore, based on the aligned mouth sequence, continuous mouth frames are organized according to a unified time grid and fed into a spatiotemporal 3D convolutional encoder to simultaneously extract mouth texture details and spatiotemporal local features formed by cross-frame motion changes. The temporal features output by the spatiotemporal 3D convolutional encoder are input into an LSTM step by step to perform spatiotemporal feature recursion and aggregate long-range motion context across frames. The LSTM generates a visual recursion intermediate state at each time step and writes the visual recursion intermediate state into the visual encoder recursion state to maintain the recursion continuity between continuous windows. The current output of the LSTM and the visual encoder recursion state together form a high-level visual representation that can characterize the current mouth motion and context dependence and maintain the recursion connection with the mouth sequence aligned to the next time step.

[0059] It should be noted that mouth texture details refer to the spatial fine-grained representation of the mouth shape, such as the brightness gradient distribution and texture intensity distribution in the lip region of the aligned mouth sequence. Spatiotemporal local features refer to the joint representation of local motion patterns and deformation patterns formed by the changes of mouth texture details over time within a short time slice.

[0060] Combining audio quality metrics and high-level audio characterization, the audio modal reliability is calculated using the following expression: ; in, It is audio modal reliability. It is an audio quality indicator. It is audio entropy. It is the number of components in the audio confidence probability distribution, that is, the number of confidence score components output by the confidence mapping parameters; Combining high-level visual representation and visual quality indicators, the reliability of the visual modality is calculated, expressed as: ; in, It is video modal reliability. It is a video quality indicator. It's video entropy. It is the number of components in the video confidence probability distribution, that is, the number of confidence score components output by the confidence mapping parameters; It should be noted that the process of obtaining audio entropy and visual entropy is as follows: Based on the confidence mapping parameters, the high-level audio representation or high-level visual representation is mapped to a set of confidence scoring components. These confidence scoring components are then exponentialized and normalized to form a confidence probability distribution whose sum is one. The probability magnitudes and logarithmic weights of each component in the confidence probability distribution are summarized, and the information entropy of the confidence probability distribution is measured to characterize its dispersion, thus obtaining the audio entropy or visual entropy. A larger entropy value indicates a more dispersed probability distribution derived from the high-level representation, thus representing higher uncertainty. The confidence mapping parameters include audio confidence mapping parameters and video confidence mapping parameters, which are obtained by linearly projecting the high-level audio representation and high-level visual representation using the Softmax regression method and optimizing through backpropagation of the recognition loss of the training samples.

[0061] S5. Based on modal reliability, the audio high-level representation and the visual high-level representation are fused, and the prefix stability and confidence state are evaluated by the streaming decoder to obtain the streaming recognition information packet. The reliability of audio and visual modalities is aligned and corrected at the same time grid. Interference from low-reliability modalities is suppressed by dynamic weighting and gating fusion to obtain a fusion representation that can directly represent the current speech content.

[0062] Furthermore, when aligning and correcting the audio modal reliability and visual modal reliability within the same time frame, the audio modal reliability and visual modal reliability are resampled and time drift is eliminated according to the unified time frame, so that the audio modal reliability and visual modal reliability correspond to the same audio high-level representation and visual high-level representation in the same time frame. Then, the audio modal reliability and visual modal reliability are normalized and constrained, and the relative proportion is converted to obtain dynamic weighting coefficients. Based on the bias amplitude of the dynamic weighting coefficients, the gating coefficients are obtained by Sigmoid compression. The dynamic weighting coefficients and the gating coefficients are used together to gating and fusing the audio high-level representation and the visual high-level representation to suppress low-reliability modal interference, resulting in a fused representation that can directly represent the current speech content. The fused representation is then passed to the prefix beam search method for streaming decoding.

[0063] The prefix beam search method is used to perform streaming decoding of the fusion representation, obtain the candidate path set that grows over time and the incremental text of the current time period, and synchronously record the posterior probability distribution of each candidate path to generate decoding posterior statistics.

[0064] Furthermore, when using the prefix beam search method to perform streaming decoding on the fused representation, the fused representation is read sequentially along a unified time grid, and candidate prefixes are expanded in each time grid to obtain the path log probability score. The candidate prefixes with the highest path log probability scores are retained to form a candidate path set that grows over time. At the same time, the path log probability scores of the candidate path set are normalized to obtain the posterior probability distribution of each candidate path and written into the decoding posterior statistics. In the current time grid, the candidate prefix with the highest path log probability score in the candidate path set is selected to generate the incremental text for the current time period. The candidate path set, the incremental text, and the decoding posterior statistics enter the prefix consistency detection process in a sliding window.

[0065] Prefix consistency detection is performed on incremental text over consecutive time periods within a sliding window, and the degree of divergence in the candidate path set is used as an auxiliary criterion to obtain prefix stability, which characterizes whether the current output has converged.

[0066] Furthermore, when performing prefix consistency detection on incremental text within a sliding window for consecutive time periods, incremental text of consecutive time grids is collected by the sliding window, and prefix alignment comparison is performed on adjacent incremental texts. The longest common prefix that remains consistent within the window is identified, and the duration of the common prefix is ​​recorded. At the same time, the information entropy is calculated by the posterior probability distribution of the candidate path set, and the degree of divergence of the candidate path set in the same time grid is calculated by combining the maximum probability proportion of the posterior probability distribution. This is used to characterize the convergence state of the candidate path set. The duration of the longest common prefix and the degree of divergence of the candidate path set are fused to obtain the prefix stability that characterizes whether the current output has converged, and the prefix stability is passed to the stable segment confirmation process.

[0067] It should be noted that the degree of divergence of the candidate path set refers to the dispersion of whether the posterior probability distribution of the candidate path set is concentrated in a few candidate paths. The more dispersed the posterior probability distribution, the higher the degree of divergence of the candidate path set.

[0068] Based on prefix stability, the incremental text is confirmed to be a stable segment, and the prefix segments that meet the stability conditions are solidified as stable text. At the same time, the tail segments that do not meet the stability conditions are retained as incremental parts that can continue to be updated to reduce real-time display jitter.

[0069] Furthermore, when performing stable segment confirmation on incremental text based on prefix stability, the prefix stability is read and incremental text prefix segments that meet the stability conditions are located. Incremental text prefix segments that meet the stability conditions are solidified into stable text and continuously appended to the stable text in timestamp order. At the same time, the tail segments of incremental text that do not meet the stability conditions are retained to form incremental parts that can be updated. This allows the stable text to bear the confirmed output while the incremental parts bear the iterative update, thereby reducing real-time display jitter and passing the stable text and incremental text, along with the prefix stability, to the confidence state analysis and encapsulation process.

[0070] It should be noted that the stability condition refers to the prefix stability reaching a continuous consistency and the confidence state remaining concentrated, which is determined by jointly judging the duration of the longest common prefix within the sliding window and the degree of divergence of the candidate path set; the tail segment refers to the unconverged part at the end of the incremental text that is not confirmed by the stable segment.

[0071] Based on the confidence state of the decoding post-hoc statistical analysis, the confidence state is associated with the incremental text, stable text and prefix stability by timestamp and encapsulated to form a streaming recognition information packet.

[0072] Furthermore, when analyzing the confidence state based on the decoding posterior statistical analysis, the decoding posterior statistics are read and the concentration of the posterior probability distribution of the candidate path set and the relative advantage of the optimal candidate path are statistically analyzed to obtain the confidence state that represents the credibility of the current incremental text and stable text. Then, the confidence state, incremental text, stable text and prefix stability are associated with the timestamp and encapsulated to form a streaming recognition information packet.

[0073] Confidence level refers to the degree of credibility of the current incremental text and the stable text as real content to be identified within the current time window.

[0074] The streaming recognition information packet includes incremental text, stable text, prefix stability, and confidence state.

[0075] S6. Perform post-processing of the text based on the streaming recognition information packet to generate speech recognition text and perception scheduling instructions, and dynamically adjust the acquisition parameters of the dual-modal data.

[0076] The system performs punctuation restoration, number format standardization, and word form normalization on the stable text, and then combines the processed stable text with the incremental text to generate speech recognition text.

[0077] Furthermore, when performing punctuation restoration, number format standardization, and lexical normalization on stable text, the stable text is read and the punctuation insertion position is inferred based on the pause patterns and syntactic boundaries of the language model to complete punctuation restoration. Then, the numerical segments in the stable text are standardized to unify the expression of Arabic numerals and Chinese numerals and correct the writing rules of time, date, amount, etc. Subsequently, the lexical changes in the stable text are normalized to unify the morphological representation of uppercase and lowercase letters, tense, plural, etc. The processed stable text and incremental text are concatenated and summarized in the order of timestamps to obtain speech recognition text while maintaining semantic continuity.

[0078] The output mode of the speech recognition text is controlled based on prefix stability and confidence state, generating scheduling decision information for the next round of acquisition control.

[0079] Furthermore, when controlling the output mode of speech recognition text based on prefix stability and confidence state, the prefix stability and confidence state are read, and the speech recognition text is divided into confirmation segments corresponding to stable text and updateable segments corresponding to incremental text. Prefix stability is used to determine the advancement speed of confirmation segments and the refresh frequency of updateable segments, and confidence state is used to determine the display credibility mark of speech recognition text and whether delayed confirmation is triggered. The confirmation segments and updateable segments are arranged according to the output rules associated with timestamps to control the output mode and generate scheduling and judgment information for the next round of acquisition control.

[0080] Based on scheduling judgment information, audio quality indicators, and visual quality indicators, the system obtains perception scheduling instructions and dynamically adjusts the acquisition parameters of the dual-modal data to achieve adaptive updates of microphone acquisition parameters and camera acquisition parameters.

[0081] It should be noted that the output rule associated with timestamps refers to the output constraint that concatenates stable text and incremental text in a uniform timestamp order and specifies the update method of the confirmation segment and the refresh method of the updatable segment corresponding to each timestamp.

[0082] Furthermore, by reading scheduling judgment information and combining audio and visual quality indicators, the availability level and modal bias direction of audio enhancement frame blocks and mouth normalization sequences are determined. For example, if the audio quality indicator remains low while the visual quality indicator remains stable, the modal bias direction is determined to point to the mouth normalization sequence. Based on the judgment result, a perceptual scheduling instruction containing the target modality priority and target sampling intensity is generated. The perceptual scheduling instruction is used to dynamically adjust the acquisition parameters of the dual-modal data. For example, if the visual quality indicator decreases and the number of frames that fail to locate the mouth region increases, the camera acquisition frame rate is increased and the microphone acquisition gain and noise reduction intensity are adjusted simultaneously to complete the adaptive update of the microphone acquisition parameters and camera acquisition parameters. The dynamically adjusted acquisition parameters take effect in the next round of dual-modal data acquisition and are consistent with the subsequent timestamp alignment process.

[0083] This embodiment also provides a speech recognition system based on audiovisual bimodal perception, including: a data acquisition module, which acquires bimodal data and performs unified timestamp alignment and cache maintenance to generate audiovisual frame packets; The quality assessment module, based on audio-visual frame packets, acquires audio enhancement frame blocks and mouth normalization sequences and performs quality assessment to generate quality-annotated audio-visual frame packets. The offset estimation module estimates the audio-visual alignment offset based on the quality-annotated audio-visual frame packets, and performs temporal resampling and alignment slicing on the audio enhancement frame blocks and the mouth normalization sequence to generate an aligned bimodal sequence. A reliable evaluation module performs feature recursion on the aligned bimodal sequence to obtain high-level audio and visual representations and calculate modal reliability. The audio-visual fusion module fuses the high-level audio representation and the high-level visual representation based on modal reliability, and evaluates the prefix stability and confidence state through a streaming decoder to obtain streaming recognition information packets. The optimization module performs post-processing on the streaming recognition information packets to generate speech recognition text and perception scheduling instructions, and dynamically adjusts the acquisition parameters of the dual-modal data. In summary, this invention achieves synchronous input of audiovisual information on a unified time grid by estimating the audiovisual alignment offset based on quality-annotated audiovisual frame packets and performing temporal resampling and alignment slicing on audio-enhanced frame blocks and lip-sync sequences to generate aligned bimodal sequences; subsequently, it fuses audio high-level representations and visual high-level representations based on modal reliability and obtains streaming recognition information packets by evaluating prefix stability and confidence state through a streaming decoder, so that the decoding output has verifiable stable segments and reliable representations, thereby improving the continuity and consistency of speech recognition text.

[0084] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A speech recognition method based on audiovisual bimodal perception, characterized in that: include, Collect dual-modal data and perform unified timestamp alignment and cache maintenance to generate audiovisual frame packets; Based on audio-visual frame packets, audio enhancement frame blocks and mouth normalization sequences are obtained and their quality is evaluated to generate quality-annotated audio-visual frame packets. The audio-visual alignment offset is estimated based on the quality-labeled audio-visual frame packets, and the audio enhancement frame blocks and mouth normalization sequences are temporally resampled and aligned sliced ​​to generate an aligned bimodal sequence. Feature recursion is performed on the aligned bimodal sequence to obtain high-level audio and visual representations, and modal reliability is calculated. Based on modal reliability, the audio high-level representation and the visual high-level representation are fused, and the prefix stability and confidence state are evaluated by a streaming decoder to obtain the streaming recognition information packet. Post-processing of the text based on the streaming recognition information packet generates speech recognition text and perception scheduling instructions, and dynamically adjusts the acquisition parameters of the bimodal data.

2. The speech recognition method based on audiovisual bimodal perception as described in claim 1, characterized in that: The audio-visual frame packet includes timestamp-aligned audio frames and video frames; The aligned bimodal sequence includes an aligned audio sequence and a mouth sequence; The streaming identification information package includes incremental text, stable text, prefix stability, and confidence state.

3. The speech recognition method based on audiovisual bimodal perception as described in claim 1, characterized in that: The steps for generating the audiovisual frame packets are as follows: Establish a unified timestamp benchmark, initialize the audio and video sliding buffers, and generate a unified timestamp to mark the acquisition time; Audio and video data are collected based on a unified timestamp and written to the audio sliding buffer and video sliding buffer respectively, generating audio frames and video frames with timestamps. Perform timestamp alignment on audio and video frames with timestamps, update the audio and video sliding buffers, and assemble frame packets to generate audiovisual frame packets.

4. The speech recognition method based on audiovisual bimodal perception as described in claim 1, characterized in that: The steps for acquiring audio enhancement frame blocks and mouth normalization sequences and performing quality assessment are as follows: Gain stabilization and noise suppression are performed on the timestamp-aligned audio frames to obtain audio enhancement frame blocks, and audio quality metrics are calculated based on speech activity detection and signal-to-noise ratio estimation. Face localization and mouth region localization are performed on video frames aligned with timestamps, and the mouth regions are aligned and normalized to obtain a standardized mouth sequence. Sharpness and occlusion were assessed based on the standardized oral sequence, and visual quality indicators were calculated.

5. The speech recognition method based on audiovisual bimodal perception as described in claim 1, characterized in that: The steps for temporal resampling and aligned slicing of the audio enhancement frame blocks and the mouth normalization sequence are as follows: Based on audio enhancement frame blocks and mouth normalization sequences, audio energy change sequences and mouth motion intensity sequences are obtained within the same time window; The correlation between audio energy change sequences and mouth movement intensity sequences was obtained by using the cross-correlation peak search method. The correlation results were then filtered by combining audio quality indicators and visual quality indicators to identify audiovisual alignment offset. Based on the audio-visual alignment offset, the audio enhancement frame block is temporally resampled to obtain an audio resampled frame block aligned with the video timeline; Based on the audio-visual alignment offset, the mouth normalization sequence is time-aligned and resampled to obtain a mouth resampled sequence consistent with the audio time axis; Alignment slicing is performed on the same time grid based on the audio resampled frame block and the mouth resampled sequence to generate an aligned bimodal sequence.

6. The speech recognition method based on audiovisual bimodal perception as described in claim 1, characterized in that: The steps for obtaining the high-level audio representation and the high-level visual representation are as follows: Based on the aligned audio sequence, streaming feature recursion is performed by combining sequence causal convolutional coding and gated recurrent units, and the intermediate state of acoustic recursion is written into the recursion state of acoustic encoder to obtain high-level audio representation. Based on the aligned mouth sequence, spatiotemporal 3D convolutional coding and LSTM are combined to perform spatiotemporal feature recursion, and the intermediate visual recursion state is written into the recursion state of the visual encoder to obtain the high-level visual representation.

7. The speech recognition method based on audiovisual bimodal perception as described in claim 6, characterized in that: The steps for calculating modal reliability are as follows: The audio modal reliability is calculated by combining audio quality metrics and high-level audio characterization. Visual modal reliability is calculated by combining high-level visual representations and visual quality indicators.

8. The speech recognition method based on audiovisual bimodal perception as described in claim 1, characterized in that: The steps for evaluating prefix stability and confidence state using a streaming decoder are as follows: Alignment correction of audio modal reliability and visual modal reliability is performed on the same time grid, and interference from low-reliability modalities is suppressed by dynamic weighting and gated fusion to obtain fusion characterization; The prefix beam search method is used to perform streaming decoding on the fusion representation to obtain the candidate path set and the incremental text of the current time period, and the posterior probability distribution of each candidate path is recorded simultaneously to generate decoding posterior statistics. Prefix consistency detection is performed on incremental text within a sliding window over consecutive time periods, and the degree of divergence in the candidate path set is used as an auxiliary criterion to obtain prefix stability, which characterizes whether the current output has converged. Based on prefix stability, the incremental text is confirmed to be a stable segment, and the prefix segments that meet the stability condition are solidified as stable text, while the tail segments that do not meet the stability condition are retained as incremental parts that can continue to be updated. Based on the confidence state of the decoding post-hoc statistical analysis, the confidence state is associated with the incremental text, stable text and prefix stability by timestamp and encapsulated to form a streaming recognition information packet.

9. The speech recognition method based on audiovisual bimodal perception as described in claim 8, characterized in that: The steps for post-processing text based on streaming recognition information packets to generate speech recognition text and perception scheduling instructions, and for dynamically adjusting the acquisition parameters of the bimodal data, are as follows: The stable text is processed by punctuation restoration, number format standardization, and word form normalization. The processed stable text and incremental text are then combined to generate speech recognition text. The output mode of speech recognition text is controlled based on prefix stability and confidence state to generate scheduling decision information; Based on scheduling judgment information, audio quality indicators, and visual quality indicators, perception scheduling instructions are obtained, and the acquisition parameters of dual-modal data are dynamically adjusted.

10. A speech recognition system based on audiovisual bimodal perception, based on the speech recognition method based on audiovisual bimodal perception as described in any one of claims 1 to 9, characterized in that: include, The data acquisition module collects dual-modal data, performs unified timestamp alignment and cache maintenance, and generates audiovisual frame packets; The quality assessment module, based on audio-visual frame packets, acquires audio enhancement frame blocks and mouth normalization sequences and performs quality assessment to generate quality-annotated audio-visual frame packets. The offset estimation module estimates the audio-visual alignment offset based on the quality-annotated audio-visual frame packets, and performs temporal resampling and alignment slicing on the audio enhancement frame blocks and the mouth normalization sequence to generate an aligned bimodal sequence. A reliable evaluation module performs feature recursion on the aligned bimodal sequence to obtain high-level audio and visual representations and calculate modal reliability. The audio-visual fusion module fuses the high-level audio representation and the high-level visual representation based on modal reliability, and evaluates the prefix stability and confidence state through a streaming decoder to obtain streaming recognition information packets. The optimization module performs text post-processing based on the streaming recognition information packet to generate speech recognition text and perception scheduling instructions, and dynamically adjusts the acquisition parameters of the dual-modal data.

Images