Method and system for audio separation and character conversion in real-time video stream

Abstract

The invention provides a method and system for separating audios from a real-time video stream and carrying out character conversion, and belongs to the technical field of audio and video processing, and the method comprises the steps: obtaining the real-time video stream, carrying out the decapsulation, and separating an audio stream; the audio stream is in an AAC or PCM format; real-time preprocessing is carried out on the audio stream to obtain a standardized PCM audio frame, and the real-time preprocessing comprises noise reduction, echo cancellation, automatic gain control, resampling and mute detection; inputting the standardized PCM audio frames into a pre-trained streaming speech recognition model according to a time sequence, performing frame-by-frame recognition, and outputting an incremental text token sequence; performing text post-processing on the incremental text token sequence to generate a structured text with time information, and outputting the structured text; the text post-processing comprises language model error correction, punctuation recovery and timestamp alignment. The application integrates audio and video processing and streaming speech recognition, is low in delay and high in accuracy, and meets the real-time requirement.

Description

Technical Field

[0001] This application belongs to the field of audio and video processing technology, specifically relating to a method and system for separating audio from a real-time video stream and converting it into text. Background Technology

[0002] With the development of network communication and multimedia technologies, real-time video has been widely used in online education, remote conferencing, live streaming platforms, security monitoring, and accessibility systems. In these applications, users not only need to watch video content, but also often need to obtain text information corresponding to the audio, such as real-time captions, meeting minutes, and voice search.

[0003] Currently, existing technologies for processing real-time video streams and extracting audio for text conversion typically consist of two independent subsystems: an audio separation subsystem and a speech recognition subsystem. The audio separation subsystem usually uses tools like FFmpeg to decapsulate the video stream, extract the audio stream, and then perform audio encoding and decoding conversion. The speech recognition subsystem typically uses offline or batch processing methods to recognize the audio files. This separate processing approach has the following problems: First, there is high processing latency. Audio classification and speech recognition are performed step-by-step, involving multiple data transfers and format conversions, resulting in significant end-to-end latency, which cannot meet the requirements of applications with high real-time demands. Second, the audio processing subsystem and the speech recognition subsystem are deployed independently, resulting in uncoordinated resource scheduling, high overall system complexity, and high maintenance costs. Third, real-time performance is poor. Traditional speech recognition systems are mostly non-streaming, requiring the entire audio segment to be received before recognition can begin, making frame-by-frame real-time transcription impossible. Finally, environmental adaptability is weak. In complex acoustic environments, audio quality is significantly affected by noise, echoes, etc., and existing methods lack integrated audio preprocessing mechanisms, leading to a decrease in recognition accuracy.

[0004] Therefore, there is an urgent need for a low-latency, real-time audio-to-text method that integrates audio and video classification, audio preprocessing, and streaming speech recognition to meet the demand for rapid generation of text information such as subtitles and meeting minutes in various real-time video application scenarios. Summary of the Invention

[0005] In a first aspect, embodiments of this application provide a method for separating audio and converting it to text in a real-time video stream, comprising the following steps: S1. Acquire a real-time video stream and decapsulate the real-time video stream to separate the audio stream; the audio stream is in AAC or PCM format; S2. Perform real-time preprocessing on the audio stream to obtain standardized PCM audio frames. The real-time preprocessing includes noise reduction, echo cancellation, automatic gain control, resampling, and silence detection. S3. Input the standardized PCM audio frames into the pre-trained streaming speech recognition model in chronological order, perform frame-by-frame recognition, and output an incremental text token sequence; S4. Perform text post-processing on the incremental text token sequence to generate structured text with time information and output it; the text post-processing includes language model error correction, punctuation restoration and timestamp alignment.

[0006] Furthermore, the specific steps of step S1 are as follows: S11. Receive audio and video composite streams via RTMP, WebRTC, or HLS protocols; S12. Perform container format parsing on the audio-video composite stream, identify and separate the video elementary stream encoded data packets and the audio elementary stream encoded data packets; the container format includes MP4, FLV or TS; S13. Decode the separated audio elementary stream coded data packets and convert them into a continuous pulse code modulation audio frame sequence; S14. Output the continuous pulse code modulated audio frame sequence in the order of frame timestamps.

[0007] Furthermore, the specific steps of step S13 are as follows: S131. Determine the format of the audio basic stream encoded data packet; If it is in AAC format, proceed to step S132; If it is in PCM format, proceed to step S133; S132. Decode the audio basic stream encoded data packets into PCM frames using an AAC decoder; S133. Directly split the audio basic stream encoded data packet into PCM frames.

[0008] Furthermore, the specific steps of step S2 are as follows: S21. Perform noise reduction processing on the pulse code modulation audio frames using the following frequency domain-based spectral subtraction method to estimate and reduce noise in the spectrum of each frame of audio signal, and output the noise-reduced audio spectrum:

[0009] in, The spectrum of the input audio frame. This is the noise spectrum estimated based on the silent segment. This is the spectrum after noise reduction. For over-subtraction factor, This is the spectral amplitude threshold coefficient. This is the attenuation coefficient for the residual noise. S22. Perform echo cancellation on the noise-reduced audio signal, using a normalized minimum mean square adaptive filter, and update the filter coefficients in real time according to the reference signal to filter out the echo component from the current frame. S23. Perform automatic gain control on the audio signal after echo cancellation, dynamically adjust the gain according to the short-time energy of the audio signal, and control the amplitude of the audio signal within a preset range; S24. Perform resampling on the audio signal after automatic gain control, and use a multiphase interpolation filter to uniformly convert the sampling rate to a preset frequency value; S25. Perform silence detection on the resampled audio signal, calculate the short-time energy and zero-crossing rate of each frame; when the short-time energy of a frame is lower than the preset energy threshold and the zero-crossing rate is lower than the preset frequency threshold, determine that the corresponding frame is a silent frame and remove it. S26. Output the processed, continuous, normalized PCM audio frame sequence.

[0010] Furthermore, the specific steps of step S3 are as follows: S31. The standardized PCM audio frame sequence is grouped according to time order and a fixed duration window. Features are extracted from each group of audio frames to form an audio feature vector sequence. S32. The audio feature vector sequence is input into the encoder of the pre-trained Conformer-Transducer streaming speech recognition model in real time according to the grouping order, so as to output the corresponding acoustic feature vector sequence. S33. The pre-trained Conformer-Transducer streaming speech recognition model decodes the acoustic feature vector sequence frame by frame to generate a text token sequence; S34. Map the text token sequence to the corresponding Chinese characters or word fragments through a pre-set vocabulary to generate an incremental text token sequence.

[0011] Furthermore, the Conformer-Transducer streaming speech recognition model in step S32 is generated as follows: S321. Construct an end-to-end streaming speech recognition model that includes an encoder, a prediction network, and a joint network; The encoder comprises several stacked Conformer modules, each Conformer module comprising, in sequence, a first feedforward module, a multi-head self-attention module employing relative position encoding, a gated depthwise separable convolution module, and a second feedforward module; the encoder is used to convert the input audio feature vector sequence into a high-level acoustic feature vector sequence; The prediction network is a sequence model based on a long short-term memory network or a single-layer Transformer decoder, used to generate the context semantic vector at the current moment based on the historical output text token sequence; The joint network is a linear transform layer used to fuse the acoustic feature vectors and context semantic vectors at the same time and output the probability distribution of all candidate tokens. S322. Select speech samples that meet the set time length, and perform data augmentation processing on the speech data in the speech samples by adding noise, simulating reverberation and speech rate perturbation, and then add text annotations to construct a speech dataset; S323. Construct a connectionist temporal classification extension of the Transducer loss function, and use a speech dataset and the AdamW optimizer to train and optimize the end-to-end streaming speech recognition model in a distributed computing environment with a global batch size no less than the set data block size, until the character error rate converges or the maximum number of iterations is met, thus obtaining the pre-trained Conformer-Transducer streaming speech recognition model.

[0012] Furthermore, the specific steps of step S33 are as follows: S331. Create an initially empty sequence as the historical text token sequence; S332. Input the historical text token sequence into the prediction network of the pre-trained Conformer-Transducer streaming speech recognition model, and output the context vector at the current time. S333. Input the acoustic feature vector output by the encoder of the pre-trained Conformer-Transducer streaming speech recognition model at the current time step and the context vector output at the current time step into the joint network for fusion calculation, and output the probability distribution of all candidate tokens. S334. Select the token with the highest probability from the probability distribution of all candidate tokens as the decoding output at the current moment; at the same time, append the decoded output token to the end of the historical text token sequence to update the historical text token sequence; S335. Determine whether there are still unprocessed vectors in the acoustic feature vector sequence; If not, proceed to step S336; If so, the updated historical text token sequence is used as the input for the next time step, and the process returns to step S332 to continue processing the acoustic feature vector for the next time step. S336. Arrange the tokens decoded at all times in chronological order to generate a text token sequence; The frame-by-frame decoding process in steps S332, S333, and S334 is represented by the following mathematical formula:

[0013] In the formula, Let be the acoustic feature vector output by the encoder at time t. The prediction network is based on the current historical text token sequence. The calculated context vector, For the model vocabulary set, The conditional probabilities are calculated for the joint network. This is the token output by decoding at time t.

[0014] Furthermore, the specific steps of step S4 are as follows: S41. Input the incremental text token sequence into the pre-trained BERT language model, correct homophone and near-homophone errors based on the context semantic probability, and insert period, comma and question mark punctuation marks at the end of the sentence and at the clauses to generate a coherent preliminary text; S42. Obtain the time information corresponding to each token in the incremental text token sequence output by the pre-trained Conformer-Transducer streaming speech recognition model during the decoding process; the time information represents the start and end sampling point positions of each token in its corresponding original audio frame sequence; S43. Based on the time information, calculate the precise start and end times for each character or word in the preliminary text:

[0015]

[0016] in, and The first in the preliminary text The start and end times of a character or word. This is the starting absolute timestamp of the original audio segment corresponding to the initial text. This refers to the audio sampling rate. S44. Encapsulate the initial text with timestamps according to the SRT subtitle format or JSON format containing text, start, and end fields to generate the final structured text and output it.

[0017] Furthermore, the training process of the BERT language model in step S41 is as follows: S411. Select the Chinese BERT-base model containing a Transformer encoder with a set number of layers as the base model; S412. On top of the basic model, add two parallel linear classification layers for text correction and punctuation restoration tasks to obtain the adapted model; the two linear classification layers are used to predict the correct character type and the type of punctuation mark that should be inserted after the token for each input token, respectively. S413. Construct a fine-tuning dataset containing more than a set number of sentence pairs. Each data point in the fine-tuning dataset includes a text without punctuation containing homophones or near-homophones, and the corresponding correct text with punctuation. S414. Using the fine-tuning dataset, supervised fine-tuning training of the adaptation model is performed with the cross-entropy loss function. During the training process, the Adam optimizer is used to train at a learning rate no greater than a set threshold until the overall accuracy of the adaptation model converges, resulting in a pre-trained BERT language model for text post-processing. The cross-entropy loss function is a weighted sum of character error correction loss and punctuation prediction loss.

[0018] Secondly, embodiments of this application also provide a system for separating audio from a real-time video stream and performing text conversion, including a stream receiving and decapsulation module for acquiring a real-time video stream and decapsulating the real-time video stream to separate an audio stream; the audio stream is in AAC or PCM format. The audio preprocessing module is used to perform real-time preprocessing on the audio stream to obtain standardized PCM audio frames. The real-time preprocessing includes noise reduction, echo cancellation, automatic gain control, resampling, and silence detection. The streaming speech recognition module is used to input standardized PCM audio frames into a pre-trained streaming speech recognition model in chronological order, perform frame-by-frame recognition, and output an incremental text token sequence. The text post-processing and output module is used to perform text post-processing on the incremental text token sequence, generate structured text with time information, and output it; the text post-processing includes language model error correction, punctuation restoration, and timestamp alignment.

[0019] As can be seen from the above technical solutions, this application has the following advantages: The method and system for separating audio from a real-time video stream and converting it to text provided in this application integrate modules such as audio / video decapsulation, audio preprocessing, streaming speech recognition, and text post-processing to achieve low-latency, high-accuracy real-time audio-to-text conversion. This application is applicable to scenarios such as online education, remote conferencing, live streaming platforms, and security monitoring, and can effectively improve real-time performance and resource utilization, enhance user experience, and improve system compatibility. Attached Figure Description

[0020] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the method for separating audio and converting text in a real-time video stream according to the present invention. Figure 1 .

[0022] Figure 2 This is a flowchart illustrating the method for separating audio and converting text in a real-time video stream according to the present invention. Figure 2 .

[0023] Figure 3 This is a schematic diagram of the system for separating audio and converting text in a real-time video stream according to the present invention. Detailed Implementation

[0024] Various embodiments of this disclosure will be described more fully in the detailed steps of the method for separating audio and performing text conversion in a real-time video stream described below. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0025] This embodiment provides a method for separating audio from a real-time video stream and converting it into text, achieving integrated audio-video separation, audio preprocessing, and streaming speech recognition, with low latency, high compatibility, and improved real-time transcription efficiency.

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Please see Figure 1 The diagram shows a flowchart of a method for separating audio and converting it to text in a real-time video stream, according to a specific embodiment. The method includes the following steps: S1. Acquire a real-time video stream and decapsulate the real-time video stream to separate the audio stream; the audio stream is in AAC or PCM format; It should be noted that the system acquires and decapsulates real-time video streams and separates the audio stream, providing a basic data source for audio processing and text conversion; it supports AAC and PCM audio and video formats and protocols, ensuring the system's compatibility and versatility, and adapting to different audio and video application scenarios. S2. Perform real-time preprocessing on the audio stream to obtain standardized PCM audio frames. The real-time preprocessing includes noise reduction, echo cancellation, automatic gain control, resampling, and silence detection. It should be noted that real-time preprocessing of the audio stream, through noise reduction, echo cancellation, automatic gain control, resampling, and silence detection, can effectively improve audio quality, reduce noise and echo interference, unify the audio sampling rate, and remove silence frames, thereby providing high-quality audio input for the speech recognition process and improving the accuracy of speech recognition. S3. Input the standardized PCM audio frames into the pre-trained streaming speech recognition model in chronological order, perform frame-by-frame recognition, and output an incremental text token sequence; It should be noted that by inputting the pre-processed audio frames into the pre-trained streaming speech recognition model for frame-by-frame recognition, an incremental text token sequence can be output in real time. This streaming processing method achieves low-latency speech transcription, meeting the needs of application scenarios with high real-time requirements, such as real-time subtitle generation in online education, remote conferencing, and other scenarios. S4. Perform text post-processing on the incremental text token sequence to generate structured text with time information and output it; the text post-processing includes language model error correction, punctuation restoration and timestamp alignment; It should be noted that post-processing of incremental text token sequences, including language model error correction, punctuation restoration, and timestamp alignment, can generate structured text with time information. This can improve the readability and accuracy of the text, making it more suitable for the needs of practical application scenarios, facilitating subsequent operations such as subtitle generation, content retrieval, and data analysis, and enhancing the user experience.

[0028] This embodiment integrates audio and video decapsulation, audio preprocessing, streaming speech recognition, and text post-processing modules to achieve low-latency, high-accuracy real-time audio-to-text conversion, which is widely applicable to various application scenarios.

[0029] Furthermore, as a refinement and extension of the specific implementation methods of the above embodiments, in order to fully illustrate the specific implementation process in this embodiment, as follows: Figure 2 As shown, another method for separating audio from a real-time video stream and converting it to text is provided. Taking online education live streaming as the application background, based on a server-side deployment architecture, the entire process of real-time video stream audio separation and text conversion is fully realized. Hardware environment The deployment server uses a dual-channel 16-core CPU, 64GB of memory, and a 1TB high-speed solid-state drive, and is equipped with an NVIDIA A10 graphics card to accelerate model inference; the edge acquisition terminal is a regular teaching terminal (CPU i5-12400, memory 16GB), which sends real-time video streams through OBS streaming software. Software environment The server operating system is Ubuntu 22.04 LTS, with FFmpeg 6.0 (for audio and video decapsulation), WebRTC Audio Processing 1.0 (audio preprocessing), Python 3.10 and PyTorch 2.0 (model inference) installed. The dependent libraries include libavcodec (audio decoding), librosa (audio feature extraction), and transformers (BERT model loading). The network protocol supports RTMP (pushing) and WebSocket (real-time text push). Model preparation Pre-train the Conformer-Transducer streaming speech recognition model: supports outputting incremental tokens every 200ms; Complete the fine-tuning of the BERT language model: Based on the 12-layer Transformer encoder, the BERT-base model adds error correction and punctuation prediction classification layers, and trains on a dataset of 100,000 Chinese sentence pairs; The method includes the following steps: S1. Acquire the real-time video stream and decapsulate it to separate the audio stream; the audio stream is in AAC or PCM format; the specific steps of step S1 are as follows: S11. Receive audio and video composite streams via RTMP, WebRTC, or HLS protocols; For example, the teacher pushes the live classroom stream to the server's push address (rtmp: / / xxx.xxx.xxx.xxx / live / class01) via OBS software using the RTMP protocol. The server's video receiving module listens to this port and acquires the audio and video composite stream in real time. The composite stream container format is FLV, the video encoding is H.264, and the audio encoding is AAC. S12. Perform container format parsing on the audio-video composite stream, identify and separate the video elementary stream encoded data packets and the audio elementary stream encoded data packets; the container format includes MP4, FLV or TS; For example, the audio and video decapsulation module calls FFmpeg to parse the FLV container, separating the video basic stream encoded data packets (H.264 frames) and the audio basic stream encoded data packets (AAC frames). The video stream is temporarily stored in the buffer (not decoded, only the synchronization timestamp is retained), and the audio stream is processed first. S13. Decode the separated audio elementary stream coded data packets and convert them into a continuous pulse code modulation audio frame sequence; the specific steps of step S13 are as follows: S131. Determine the format of the audio basic stream encoded data packet; If it is in AAC format, proceed to step S132; If it is in PCM format, proceed to step S133; S132. Decode the audio basic stream encoded data packets into PCM frames using an AAC decoder; S133. Directly split the audio basic stream encoded data packet into PCM frames; For example, first determine that the audio basic stream format is AAC, and then start the AAC decoder; The decoder converts the AAC encoded data packets into a 16-bit, mono, 44.1kHz sampling rate PCM audio frame sequence, thus completing the encoding format conversion. S14. Output the continuous pulse code modulated audio frame sequence in frame timestamp order; For example, PCM frames are sorted according to their timestamps (PTS) to generate a continuous and ordered sequence of PCM audio frames, which is then sent to the next preprocessing step. S2. Perform real-time preprocessing on the audio stream to obtain standardized PCM audio frames. The real-time preprocessing includes noise reduction, echo cancellation, automatic gain control, resampling, and silence detection. The specific steps of step S2 are as follows: S21. Perform noise reduction processing on the pulse code modulation audio frames using the following frequency domain-based spectral subtraction method to estimate and reduce noise in the spectrum of each frame of audio signal, and output the noise-reduced audio spectrum:

[0030] in, The spectrum of the input audio frame. This is the noise spectrum estimated based on the silent segment. This is the spectrum after noise reduction. For over-subtraction factor, This is the spectral amplitude threshold coefficient. This is the attenuation coefficient for the residual noise. For example, the WebRTC Audio Processing tool is invoked to perform noise reduction using the frequency domain spectral subtraction method described in this application: first, the silence segment of the audio is identified through the silence detection module, and the noise spectrum is estimated. Set over-subtraction factor =2.0, spectral amplitude threshold coefficient =0.15, noise residual attenuation coefficient γ=0.5, substitute into the formula to calculate the spectrum after noise reduction. Then, the inverse Fourier transform is used to restore it to the time-domain audio signal, eliminating low-frequency noise from the air conditioner and keyboard typing noise in the environment; S22. Perform echo cancellation on the noise-reduced audio signal, using a normalized minimum mean square adaptive filter, and update the filter coefficients in real time according to the reference signal to filter out the echo component from the current frame. For example, a normalized least mean square (NLMS) adaptive filter is enabled, using the signal collected by the teacher's microphone as the main input and the signal played back by the speaker as the reference signal. The filter coefficients are updated in real time to filter out the echo component caused by the speaker feedback and ensure the purity of the voice signal. S23. Perform automatic gain control on the audio signal after echo cancellation, dynamically adjust the gain according to the short-time energy of the audio signal, and control the amplitude of the audio signal within a preset range; For example, the short-time energy of the audio frame is calculated, and the gain is increased when the energy is below -30dBFS and decreased when it is above -10dBFS, so as to stabilize the audio signal amplitude within the preset range of -20dBFS to -15dBFS and solve the problem of volume fluctuation caused by teachers speaking at close / far distances. S24. Perform resampling on the audio signal after automatic gain control, and use a multiphase interpolation filter to uniformly convert the sampling rate to a preset frequency value (e.g., 16kHz). For example, a multiphase interpolation filter is used to resample the PCM audio with a sampling rate of 44.1kHz to 16kHz (the model's preset sampling rate) to ensure that the audio parameters of the input speech recognition model are consistent. S25. Perform silence detection on the resampled audio signal, calculate the short-time energy and zero-crossing rate of each frame; when the short-time energy of a frame is lower than the preset energy threshold and the zero-crossing rate is lower than the preset frequency threshold, determine that the corresponding frame is a silent frame and remove it. For example, the short-time energy and zero-crossing rate of each audio frame are calculated, and the energy threshold is set to -40dBFS and the zero-crossing rate threshold is set to 50 times / frame. Silence frames (such as short pauses in class) that are below the two thresholds are removed to reduce invalid data processing. S26. Output the processed, continuous, normalized PCM audio frame sequence; For example, a continuous normalized PCM audio frame sequence with a sampling rate of 16kHz and a bit depth of 16bit is output to provide high-quality input for speech recognition; S3. Input the standardized PCM audio frames into the pre-trained streaming speech recognition model in chronological order, perform frame-by-frame recognition, and output an incremental text token sequence; the specific steps of step S3 are as follows: S31. The standardized PCM audio frame sequence is grouped according to time order and a fixed duration window. Features are extracted from each group of audio frames to form an audio feature vector sequence. The feature extraction involves extracting Mel frequency cepstral coefficient features, where audio data for every 200ms is extracted as a set of feature vectors. For example, the standardized PCM audio frames are grouped into windows with a duration of 200ms, and the Mel frequency cepstral coefficients (MFCC) features are extracted from each group to generate an audio feature vector sequence with a dimension of 40×10 (40-dimensional MFCC features, extracted once every 10ms). S32. The audio feature vector sequence is input into the encoder of the pre-trained Conformer-Transducer streaming speech recognition model in real time according to the grouping order, so as to output the corresponding acoustic feature vector sequence; the Conformer-Transducer streaming speech recognition model in step S32 is generated in the following way: S321. Construct an end-to-end streaming speech recognition model that includes an encoder, a prediction network, and a joint network; The encoder comprises several stacked Conformer modules, each Conformer module comprising, in sequence, a first feedforward module, a multi-head self-attention module employing relative position encoding, a gated depthwise separable convolution module, and a second feedforward module; the encoder is used to convert the input audio feature vector sequence into a high-level acoustic feature vector sequence; The prediction network is a sequence model based on a long short-term memory network or a single-layer Transformer decoder, used to generate the context semantic vector at the current moment based on the historical output text token sequence; The joint network is a linear transform layer used to fuse the acoustic feature vectors and context semantic vectors at the same time and output the probability distribution of all candidate tokens. S322. Select speech samples that meet the set time length, and perform data augmentation processing on the speech data in the speech samples by adding noise, simulating reverberation and speech rate perturbation, and then add text annotations to construct a speech dataset; S323. Construct a connectionist temporal classification extension of the Transducer loss function, and use a speech dataset and AdamW optimizer to train and optimize the end-to-end streaming speech recognition model in a distributed computing environment with a global batch size of not less than the set data block size (e.g., 128M) until the character error rate converges or the maximum number of iterations is met, thus obtaining the pre-trained Conformer-Transducer streaming speech recognition model. For example, the audio feature vector sequence is input into a pre-trained Conformer-Transducer model encoder in a grouped order. The encoder contains 6 stacked Conformer modules (each module contains a feedforward layer, a relative position encoding multi-head self-attention layer, and a gated depthwise separable convolutional layer), which converts the input features into a high-level acoustic feature vector sequence of dimension 256. S33. The pre-trained Conformer-Transducer streaming speech recognition model decodes the acoustic feature vector sequence frame by frame to generate a text token sequence; the specific steps of step S33 are as follows: S331. Create an initially empty sequence as the historical text token sequence; S332. Input the historical text token sequence into the prediction network of the pre-trained Conformer-Transducer streaming speech recognition model, and output the context vector at the current time. Specifically, at the initial moment, the historical text token sequence is empty, and the prediction network outputs an initial context vector; S333. Input the acoustic feature vector output by the encoder of the pre-trained Conformer-Transducer streaming speech recognition model at the current time step and the context vector output at the current time step into the joint network for fusion calculation, and output the probability distribution of all candidate tokens. S334. Select the token with the highest probability from the probability distribution of all candidate tokens as the decoding output at the current moment; at the same time, append the decoded output token to the end of the historical text token sequence to update the historical text token sequence; S335. Determine whether there are still unprocessed vectors in the acoustic feature vector sequence; If not, proceed to step S336; If so, the updated historical text token sequence is used as the input for the next time step, and the process returns to step S332 to continue processing the acoustic feature vector for the next time step. S336. Arrange the tokens decoded at all times in chronological order to generate a text token sequence; The frame-by-frame decoding process in step S332, step S333, and step S334 is characterized by the following mathematical formula:

[0031] In the formula, is the acoustic feature vector output by the encoder at time t in step S32, is the context vector calculated by the prediction network based on the current historical text token sequence in step S332, is the model vocabulary set, is the conditional probability calculated by the joint network in step S333, is the token decoded and output at time t in step S334; Exemplarily, initialize an empty historical text token sequence; at the initial moment, the historical sequence is empty, and the prediction network (based on a single-layer Transformer decoder) outputs an initial context vector ; input the first-frame acoustic feature vector output by the encoder and into the joint network (linear transformation layer), and output the probability distribution of all-word-table candidate tokens; select the token with the highest probability (such as "同") as the current output, append it to the historical sequence, and the updated sequence is ["同"]; detect that there are still unprocessed acoustic vectors, input the updated historical sequence into the prediction network, and generate the context vector at the next moment, and repeat the above steps; continue decoding until all acoustic vectors are processed, and generate a complete text token sequence (such as ["同","学","们","今","天"]); S34. Map the text token sequence to the corresponding Chinese characters or word segments through a preset vocabulary table to generate an incremental text token sequence; Exemplarily, through a preset Chinese vocabulary table (including 6,000 common Chinese characters and vocabulary), map the token sequence to Chinese character segments, and output the incremental text token sequence "同学们今天"; S4. Perform post-processing on the incremental text token sequence to generate a structured text with time information and output it; the text post-processing includes language model error correction, punctuation restoration, and timestamp alignment; the specific steps of step S4 are as follows: S41. Input the incremental text token sequence into a pre-trained BERT language model, correct homophone and near-homophone errors according to the context semantic probability, and insert punctuation marks such as periods, commas, and question marks at the end of sentences and at the end of clauses to generate a coherent preliminary text; the training process of the BERT language model in step S41 is as follows: S411. Select the Chinese BERT-base model with a set number of layers (e.g., 12 layers) of Transformer encoders as the base model; S412. On top of the base model, add two parallel linear classification layers for text correction tasks and punctuation restoration tasks to obtain an adapted model; the two linear classification layers are respectively used to predict the correct character type of each input token and the punctuation symbol type to be inserted after the token; S413. Construct a fine-tuning dataset containing more than a set number (e.g., 100,000) of sentence pairs. Each piece of data in the fine-tuning dataset includes a punctuation-free text containing homophone or near-homophone errors and the corresponding correct text with punctuation; S414. Use the fine-tuning dataset to perform supervised fine-tuning training on the adapted model with the cross-entropy loss function, and use the Adam optimizer during the training process to train with a learning rate not greater than the set threshold until the comprehensive accuracy of the adapted model converges to obtain a pre-trained BERT language model for text post-processing; the cross-entropy loss function is the weighted sum of the character correction loss and the punctuation prediction loss; Exemplarily, input the incremental token sequence into the pre-trained BERT language model: The model detects a semantic break after "今", and corrects potential homophone errors in combination with the context (no adjustment if there is no error); Insert punctuation at the sentence meaning pause, and supplement "同学们今天" to the preliminary text of "同学们,今天"; S42. Obtain the time information corresponding to each token in the incremental text token sequence output during the decoding process of the pre-trained Conformer-Transducer streaming speech recognition model; the time information characterizes the start and end sampling point positions of each token in its corresponding original audio frame sequence; Exemplarily, retrieve the time stamps during the decoding of the Conformer-Transducer model to obtain the audio sampling point positions corresponding to each token, such as "同" corresponding to sampling points 1600~3200, "学" corresponding to 3200~4800 (at a 16kHz sampling rate, 100ms corresponds to 1600 sampling points; S43. According to the time information, calculate the precise start time and end time for each word or term in the preliminary text:

[0032]

[0033] Where, and are respectively the The start and end times of a character or word, is the starting absolute timestamp of the original audio segment corresponding to the preliminary text, and are the starting and ending sampling point numbers of the token corresponding to the character or word obtained in step S42, is the audio sampling rate; Exemplarily, given that the starting absolute timestamp of this audio segment is 00:00:05.000, substitute it into the formula to calculate the start and end times of each word: The start time of "tong" = 5.000 + 1600 / 16000 = 5.100s, and the end time = 5.000 + 3200 / 16000 = 5.200s; Calculate the times of all words in sequence to generate the text with accurate time markings; S44. Package the preliminary text with timestamps in the SRT subtitle format or the JSON format including the text, start, and end fields to generate the final structured text and output it; Exemplarily, package it in the SRT subtitle format: Generate the subtitle segment of "1\n00:00:05,100-->00:00:05,600\nStudents, today"; At the same time, generate the data {"text": "Students, today", "start": 5.100, "end": 5.600} in the JSON format and push it to the student-side live page in real time through WebSocket to achieve real-time subtitle display; All subtitle segments are synchronously archived to the server for generating a complete class record after class.

[0034] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not imply the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0035] As Figure 3 shown, the following is an embodiment of a system for separating audio from a real-time video stream and performing text conversion provided by an embodiment of the present disclosure. This system belongs to the same inventive concept as the method for separating audio from a real-time video stream and performing text conversion in the above embodiments. For the details not described in detail in the embodiment of the system for separating audio from a real-time video stream and performing text conversion, reference can be made to the embodiment of the method for separating audio from a real-time video stream and performing text conversion.

[0036] The system includes: The stream receiving and decapsulation module is used to acquire real-time video streams and decapsulate them to separate audio streams; the audio streams are in AAC or PCM format. The audio preprocessing module is used to perform real-time preprocessing on the audio stream to obtain standardized PCM audio frames. The real-time preprocessing includes noise reduction, echo cancellation, automatic gain control, resampling, and silence detection. The streaming speech recognition module is used to input standardized PCM audio frames into a pre-trained streaming speech recognition model in chronological order, perform frame-by-frame recognition, and output an incremental text token sequence. The text post-processing and output module is used to perform text post-processing on the incremental text token sequence, generate structured text with time information, and output it; the text post-processing includes language model error correction, punctuation restoration, and timestamp alignment.

[0037] This embodiment achieves audio-video separation, integrated audio preprocessing and streaming speech recognition through the interactive collaboration of the streaming receiving and decapsulation module, audio preprocessing module, streaming speech recognition module, and text post-processing and output module, resulting in low latency, high compatibility, and improved real-time transcription efficiency.

[0038] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for separating audio and converting it to text in a real-time video stream, characterized in that, Includes the following steps: S1. Acquire a real-time video stream and decapsulate the real-time video stream to separate the audio stream; the audio stream is in AAC or PCM format; S2. Perform real-time preprocessing on the audio stream to obtain standardized PCM audio frames. The real-time preprocessing includes noise reduction, echo cancellation, automatic gain control, resampling, and silence detection. S3. Input the standardized PCM audio frames into the pre-trained streaming speech recognition model in chronological order, perform frame-by-frame recognition, and output an incremental text token sequence; S4. Perform text post-processing on the incremental text token sequence to generate structured text with time information and output it; the text post-processing includes language model error correction, punctuation restoration and timestamp alignment.

2. The method for separating audio and converting it to text in a real-time video stream according to claim 1, characterized in that, The specific steps of step S1 are as follows: S11. Receive audio and video composite streams via RTMP, WebRTC, or HLS protocols; S12. Perform container format parsing on the audio-video composite stream, identify and separate the video elementary stream encoded data packets and the audio elementary stream encoded data packets; the container format includes MP4, FLV or TS; S13. Decode the separated audio elementary stream coded data packets and convert them into a continuous pulse code modulation audio frame sequence; S14. Output the continuous pulse code modulated audio frame sequence in the order of frame timestamps.

3. The method for separating audio and converting it to text in a real-time video stream according to claim 2, characterized in that, The specific steps of step S13 are as follows: S131. Determine the format of the audio basic stream encoded data packet; If it is in AAC format, proceed to step S132; If it is in PCM format, proceed to step S133; S132. Decode the audio basic stream encoded data packets into PCM frames using an AAC decoder; S133. Directly split the audio basic stream encoded data packet into PCM frames.

4. The method for separating audio and converting it to text in a real-time video stream according to claim 1, characterized in that, The specific steps of step S2 are as follows: S21. Perform noise reduction processing on the pulse code modulation audio frames using the following frequency domain-based spectral subtraction method to estimate and reduce noise in the spectrum of each frame of audio signal, and output the noise-reduced audio spectrum: in, The spectrum of the input audio frame. This is the noise spectrum estimated based on the silent segment. This is the spectrum after noise reduction. For over-subtraction factor, This is the spectral amplitude threshold coefficient. This is the attenuation coefficient for the residual noise. S22. Perform echo cancellation on the noise-reduced audio signal, using a normalized minimum mean square adaptive filter, and update the filter coefficients in real time according to the reference signal to filter out the echo component from the current frame. S23. Perform automatic gain control on the audio signal after echo cancellation, dynamically adjust the gain according to the short-time energy of the audio signal, and control the amplitude of the audio signal within a preset range; S24. Perform resampling on the audio signal after automatic gain control, and use a multiphase interpolation filter to uniformly convert the sampling rate to a preset frequency value; S25. Perform silence detection on the resampled audio signal, calculate the short-time energy and zero-crossing rate of each frame; when the short-time energy of a frame is lower than the preset energy threshold and the zero-crossing rate is lower than the preset frequency threshold, determine that the corresponding frame is a silent frame and remove it. S26. Output the processed, continuous, normalized PCM audio frame sequence.

5. The method for separating audio and converting it to text in a real-time video stream according to claim 1, characterized in that, The specific steps of step S3 are as follows: S31. The standardized PCM audio frame sequence is grouped according to time order and a fixed duration window. Features are extracted from each group of audio frames to form an audio feature vector sequence. S32. The audio feature vector sequence is input into the encoder of the pre-trained Conformer-Transducer streaming speech recognition model in real time according to the grouping order, so as to output the corresponding acoustic feature vector sequence. S33. The pre-trained Conformer-Transducer streaming speech recognition model decodes the acoustic feature vector sequence frame by frame to generate a text token sequence; S34. Map the text token sequence to the corresponding Chinese characters or word fragments through a pre-set vocabulary to generate an incremental text token sequence.

6. The method for separating audio and converting it to text in a real-time video stream according to claim 5, characterized in that, In step S32, the Conformer-Transducer streaming speech recognition model is generated as follows: S321. Construct an end-to-end streaming speech recognition model that includes an encoder, a prediction network, and a joint network; The encoder comprises several stacked Conformer modules, each Conformer module comprising, in sequence, a first feedforward module, a multi-head self-attention module employing relative position encoding, a gated depthwise separable convolution module, and a second feedforward module; the encoder is used to convert the input audio feature vector sequence into a high-level acoustic feature vector sequence; The prediction network is a sequence model based on a long short-term memory network or a single-layer Transformer decoder, used to generate the context semantic vector of the current moment based on the historical output text token sequence; The joint network is a linear transform layer used to fuse the acoustic feature vectors and context semantic vectors at the same time and output the probability distribution of all candidate tokens. S322. Select speech samples that meet the set time length, and perform data augmentation processing on the speech data in the speech samples by adding noise, simulating reverberation and speech rate perturbation, and then add text annotations to construct a speech dataset; S323. Construct a connectionist temporal classification extension of the Transducer loss function, and use a speech dataset and the AdamW optimizer to train and optimize the end-to-end streaming speech recognition model in a distributed computing environment with a global batch size no less than the set data block size, until the character error rate converges or the maximum number of iterations is met, thus obtaining the pre-trained Conformer-Transducer streaming speech recognition model.

7. The method for separating audio and converting it to text in a real-time video stream according to claim 5, characterized in that, The specific steps of step S33 are as follows: S331. Create an initially empty sequence as the historical text token sequence; S332. Input the historical text token sequence into the prediction network of the pre-trained Conformer-Transducer streaming speech recognition model, and output the context vector at the current time. S333. Input the acoustic feature vector output by the encoder of the pre-trained Conformer-Transducer streaming speech recognition model at the current time step and the context vector output at the current time step into the joint network for fusion calculation, and output the probability distribution of all candidate tokens. S334. Select the token with the highest probability from the probability distribution of all candidate tokens as the decoding output at the current moment; at the same time, append the decoded output token to the end of the historical text token sequence to update the historical text token sequence; S335. Determine whether there are still unprocessed vectors in the acoustic feature vector sequence; If not, proceed to step S336; If so, the updated historical text token sequence is used as the input for the next time step, and the process returns to step S332 to continue processing the acoustic feature vector for the next time step. S336. Arrange the tokens decoded at all times in chronological order to generate a text token sequence; The frame-by-frame decoding process in steps S332, S333, and S334 is represented by the following mathematical formula: In the formula, Let be the acoustic feature vector output by the encoder at time t. The prediction network is based on the current historical text token sequence. The calculated context vector, For the model vocabulary set, The conditional probabilities are calculated for the joint network. This is the token output by decoding at time t.

8. The method for separating audio and converting it to text in a real-time video stream according to claim 1, characterized in that, The specific steps of step S4 are as follows: S41. Input the incremental text token sequence into the pre-trained BERT language model, correct homophone and near-homophone errors based on the context semantic probability, and insert period, comma and question mark punctuation marks at the end of the sentence and at the clauses to generate a coherent preliminary text; S42. Obtain the time information corresponding to each token in the incremental text token sequence output by the pre-trained Conformer-Transducer streaming speech recognition model during the decoding process; the time information represents the start and end sampling point positions of each token in its corresponding original audio frame sequence; S43. Based on the time information, calculate the precise start and end times for each character or word in the preliminary text: in, and The first in the preliminary text The start and end times of a character or word. This is the starting absolute timestamp of the original audio segment corresponding to the initial text. This refers to the audio sampling rate. S44. Encapsulate the initial text with timestamps according to the SRT subtitle format or JSON format containing text, start, and end fields to generate the final structured text and output it.

9. The method for separating audio and converting it to text in a real-time video stream according to claim 8, characterized in that, The training process of the BERT language model in step S41 is as follows: S411. Select the Chinese BERT-base model containing a Transformer encoder with a set number of layers as the base model; S412. On top of the basic model, add two parallel linear classification layers for text correction and punctuation restoration tasks to obtain the adapted model; the two linear classification layers are used to predict the correct character type and the type of punctuation mark that should be inserted after the token for each input token, respectively. S413. Construct a fine-tuning dataset containing more than a set number of sentence pairs. Each data point in the fine-tuning dataset includes a text without punctuation containing homophones or near-homophones, and the corresponding correct text with punctuation. S414. Using the fine-tuning dataset, supervised fine-tuning training of the adaptation model is performed with the cross-entropy loss function. During the training process, the Adam optimizer is used to train at a learning rate no greater than a set threshold until the overall accuracy of the adaptation model converges, resulting in a pre-trained BERT language model for text post-processing. The cross-entropy loss function is a weighted sum of character error correction loss and punctuation prediction loss.

10. A system for separating audio from a real-time video stream and performing text conversion, characterized in that, include: The stream receiving and decapsulation module is used to acquire real-time video streams and decapsulate them to separate the audio stream. The audio stream is in AAC or PCM format; The audio preprocessing module is used to perform real-time preprocessing on the audio stream to obtain standardized PCM audio frames. The real-time preprocessing includes noise reduction, echo cancellation, automatic gain control, resampling, and silence detection. The streaming speech recognition module is used to input standardized PCM audio frames into a pre-trained streaming speech recognition model in chronological order, perform frame-by-frame recognition, and output an incremental text token sequence. The text post-processing and output module is used to perform text post-processing on the incremental text token sequence, generate structured text with time information, and output it; the text post-processing includes language model error correction, punctuation restoration, and timestamp alignment.

Images