A real-time speech recognition method, apparatus, electronic device, and storage medium

Abstract

The present disclosure provides a real-time speech recognition method, device, electronic equipment and storage medium, which comprises the following steps: S1, continuously collecting a real-time audio stream; S2, performing speech activity detection on the real-time audio stream, starting to cache the real-time audio stream when detecting that speech starts, stopping caching when detecting that speech ends, and encapsulating the cached continuous audio data as an independent speech segment data packet; S3, submitting the speech segment data packet to a general offline speech recognition engine, performing overall recognition on the speech segment data packet by using the general offline speech recognition engine, and obtaining a corresponding text recognition result, wherein the general offline speech recognition engine is an offline speech recognition model that does not support stream-based incremental decoding. According to the embodiments of the present disclosure, flexible selection of the general offline speech recognition engine in the real-time speech recognition scene is realized, and the architecture binding limitation of the stream-based model is broken.

Description

Technical Field

[0001] This disclosure relates to the field of speech recognition technology, and in particular to a real-time speech recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] Speech recognition is the technology of converting human speech into a text sequence. In existing technologies, based on different latency requirements of application scenarios, speech recognition is mainly divided into two major technical approaches: offline speech recognition and streaming real-time speech recognition. Offline speech recognition, also known as non-streaming or document transcription, aims to maximize recognition accuracy. It typically performs a one-time overall recognition after receiving a complete audio data segment. The advantage of this method is high recognition accuracy, but the disadvantage is significant processing latency, making it unsuitable for real-time interactive scenarios requiring immediate feedback. Streaming real-time speech recognition aims to continuously output recognition results with extremely low latency while the user speaks. To achieve this goal, this technical approach must employ a specially designed streaming model that supports streaming incremental decoding as its core recognition engine.

[0003] Existing real-time speech recognition methods based on streaming models, while achieving low-latency interactive effects, have the following inherent drawbacks:

[0004] (1) The range of model selection is limited. This method forces the core recognition engine to be a streaming model that supports streaming incremental decoding, excluding the general offline speech recognition model with better recognition performance from the range of options, so that the real-time speech recognition scenario cannot benefit from the technical advantages of the general offline speech recognition model.

[0005] (2) The recognition accuracy is low. The recognition accuracy of the streaming model is naturally lower than that of the general offline speech recognition model, which can recognize the complete audio data as a whole. It is difficult to meet the requirements of real-time speech recognition accuracy in practical applications. Summary of the Invention

[0006] This disclosure provides a real-time speech recognition method, apparatus, electronic device, and storage medium to solve the problems of limited model selection and low recognition accuracy in existing real-time speech recognition methods, which can only rely on streaming models that support streaming incremental decoding.

[0007] In a first aspect, this disclosure provides a real-time speech recognition method, the method comprising:

[0008] S1 continuously captures real-time audio streams;

[0009] S2, perform voice activity detection on the real-time audio stream. When voice is detected to start, start caching the real-time audio stream. When voice is detected to end, stop caching and encapsulate the cached continuous audio data into an independent voice segment data packet.

[0010] S3, the speech segment data packet is submitted to the general offline speech recognition engine, and the general offline speech recognition engine is used to perform overall recognition of the speech segment data packet to obtain the corresponding text recognition result. The general offline speech recognition engine is an offline speech recognition model that does not support streaming incremental decoding.

[0011] S4, receive the text recognition result returned by the general offline speech recognition engine, and output the text recognition result;

[0012] S5, determine whether the identification termination trigger condition has been met. If yes, end this process; otherwise, return to step S2.

[0013] Furthermore, the step of performing voice activity detection on the real-time audio stream, starting to buffer the real-time audio stream when voice activity is detected to begin, and stopping buffering when voice activity is detected to end, specifically includes:

[0014] Detect whether the real-time audio stream contains valid human speech activity;

[0015] In response to the detection of the valid human speech activity, a speech start is determined, and the real-time audio stream is buffered.

[0016] In response to the detection of silence in the real-time audio stream, and the duration of the silence exceeding a preset threshold, the system determines that the speech has ended and stops buffering.

[0017] Furthermore, the step of responding to the detection of the valid human speech activity, determining the start of speech, and initiating the buffering of the real-time audio stream specifically includes:

[0018] In response to the detection of the valid human voice activity, a voice start command is triggered;

[0019] According to the voice start command, the real-time audio stream is stored starting from the current time point.

[0020] Furthermore, the response to detecting silence in the real-time audio stream and determining that the speech has ended, and then stopping buffering, specifically includes:

[0021] In response to the detection of silence in the real-time audio stream, and the duration of the silence exceeding a preset threshold, a voice end command is triggered;

[0022] According to the voice end command, the buffering stops at the current time point.

[0023] Furthermore, the step of encapsulating the cached continuous audio data into an independent speech segment data packet specifically includes:

[0024] The cached continuous audio data can be integrated into a structured data packet containing raw audio data and sampling rate information, or integrated into a WAV format waveform audio file in memory.

[0025] Furthermore, the offline speech recognition model is an offline automatic speech recognition (ASR) model based on the Transformer architecture or the Conformer architecture with convolutional enhancement.

[0026] Furthermore, the output of the text recognition result specifically includes:

[0027] The text recognition results are displayed in the user interface of the real-time interactive application.

[0028] Secondly, this disclosure provides a real-time speech recognition device, the device comprising:

[0029] The real-time audio stream acquisition module is used to continuously acquire real-time audio streams.

[0030] The voice activity detection module is connected to the real-time audio stream acquisition module and is used to detect voice activity in the real-time audio stream. When voice is detected to start, the real-time audio stream is cached. When voice is detected to end, the caching is stopped, and the cached continuous audio data is encapsulated into an independent voice segment data packet.

[0031] An offline speech recognition module, connected to the speech activity detection module, is used to submit the speech segment data packet to a general offline speech recognition engine, and use the general offline speech recognition engine to perform overall recognition of the speech segment data packet to obtain the corresponding text recognition result. The general offline speech recognition engine is an offline speech recognition model that does not support streaming incremental decoding.

[0032] A text recognition output module, connected to the offline speech recognition module, is used to receive the text recognition result returned by the general offline speech recognition engine and output the text recognition result;

[0033] The loop judgment module, connected to the text recognition output module, is used to determine whether the recognition termination trigger condition has been met. If so, the process ends; otherwise, it returns to the voice activity detection module.

[0034] Thirdly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the real-time speech recognition method described in the first aspect above.

[0035] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the real-time speech recognition method described in the first aspect.

[0036] The real-time speech recognition method, apparatus, electronic device, and storage medium disclosed herein detect speech activity in a real-time audio stream and cache and encapsulate it into independent speech segment data packets as needed. These data packets are then submitted to a general-purpose offline speech recognition engine that does not support streaming incremental decoding for overall recognition. A continuous processing flow involving result output and iterative judgment enables flexible selection of a general-purpose offline speech recognition engine in real-time speech recognition scenarios. This breaks the architectural binding limitations of streaming models and significantly broadens the model selection range for real-time speech recognition. Simultaneously, relying on the overall recognition capability of the general-purpose offline speech recognition engine for speech segment data packets, it fully leverages the recognition advantages of the general-purpose offline speech recognition model, effectively improving the accuracy of real-time speech recognition and enabling the practical application of high-precision offline recognition technology in real-time speech recognition scenarios. This solves the problem of limited model selection and low recognition accuracy in existing real-time speech recognition methods that can only rely on streaming models that support streaming incremental decoding. Attached Figure Description

[0037] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings:

[0038] Figure 1 A flowchart of a real-time speech recognition method provided in this embodiment of the disclosure;

[0039] Figure 2 This is a schematic diagram of the structure of a real-time speech recognition system provided in an embodiment of the present disclosure;

[0040] Figure 3 A flowchart of yet another real-time speech recognition method provided in this disclosure embodiment;

[0041] Figure 4 A block diagram of a real-time speech recognition device provided in an embodiment of this disclosure;

[0042] Figure 5 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0044] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0045] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0046] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0047] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0048] Figure 1 A flowchart illustrating a real-time speech recognition method provided in an embodiment of this disclosure. (Refer to...) Figure 1 The method includes:

[0049] S1 continuously captures real-time audio streams.

[0050] Specifically, raw audio data is continuously acquired through an audio input device to form a real-time audio stream. The audio input device can be a microphone, microphone array, or other hardware capable of picking up sound. The acquisition process begins immediately after system startup and continues continuously, providing an uninterrupted data source for subsequent speech activity detection.

[0051] S2, perform voice activity detection on the real-time audio stream. When voice is detected to start, start caching the real-time audio stream. When voice is detected to end, stop caching and encapsulate the cached continuous audio data into an independent voice segment data packet.

[0052] Specifically, the acquired real-time audio stream undergoes continuous speech activity detection to determine whether it contains valid human speech. This detection process analyzes the audio stream in real time. Once the start point of the speech is identified, a caching mechanism is triggered to cache the audio data. When the end point of the speech is identified, caching stops, and all continuous audio data from the start to the end is integrated into a single, complete speech segment data packet for subsequent processing.

[0053] In some embodiments, the step of performing voice activity detection on the real-time audio stream, starting to buffer the real-time audio stream when voice is detected to begin, and stopping buffering when voice is detected to end, specifically includes:

[0054] Detect whether the real-time audio stream contains valid human speech activity;

[0055] In response to the detection of the valid human speech activity, a speech start is determined, and the real-time audio stream is buffered.

[0056] In response to the detection of silence in the real-time audio stream, and the duration of the silence exceeding a preset threshold, the system determines that the speech has ended and stops buffering.

[0057] Specifically, detecting whether the real-time audio stream contains valid human speech activity refers to using speech activity detection technology to distinguish human speech from non-speech (such as environmental noise, silence, etc.). Those skilled in the art can implement this using any existing speech activity detection technology, such as energy threshold-based VAD (Voice Activity Detection), statistical VAD based on Gaussian mixture models, or deep VAD based on neural networks. The term "valid" aims to exclude non-semantic human voices such as coughing and throat clearing, or brief bursts of noise, ensuring that the detected speech corresponds to the semantic content that truly needs to be identified.

[0058] In some embodiments, the step of determining the start of speech and starting to buffer the real-time audio stream in response to detecting the valid human speech activity specifically includes:

[0059] In response to the detection of the valid human voice activity, a voice start command is triggered;

[0060] According to the voice start command, the real-time audio stream is stored starting from the current time point.

[0061] Specifically, when valid human speech activity is detected, the system internally generates and triggers a control signal to begin speech. After this signal is triggered, it instructs that subsequent incoming audio data be written to a temporary buffer immediately. The buffer can take the form of a circular buffer in memory, a temporary file, etc., with the moment when valid human speech is detected serving as the starting point for recording that segment of speech data.

[0062] In some embodiments, the step of stopping buffering in response to detecting silence in the real-time audio stream and the duration of the silence exceeding a preset threshold, indicating the end of speech, specifically includes:

[0063] In response to the detection of silence in the real-time audio stream, and the duration of the silence exceeding a preset threshold, a voice end command is triggered;

[0064] According to the voice end command, the buffering stops at the current time point.

[0065] Specifically, during the audio data caching process, voice activity is continuously detected in the real-time audio stream. When the voice energy drops to a silence level and the silence lasts for more than a preset threshold, the current speech is determined to have ended. The preset threshold can be flexibly adjusted according to the actual application scenario; for example, it can be set to 500 milliseconds in a conference scenario and 300 milliseconds in a fast-paced conversation scenario. At this time, the system internally generates and triggers a control signal to end the speech, immediately stopping the audio data writing operation with the current moment as the endpoint. Thus, the temporary buffer area stores the complete and continuous audio data from the start to the end of the speech.

[0066] In some embodiments, encapsulating the cached continuous audio data into a single speech segment data packet specifically includes:

[0067] The cached continuous audio data can be integrated into a structured data packet containing raw audio data and sampling rate information, or integrated into a WAV (Waveform Audio File Format) file in memory.

[0068] Specifically, the encapsulation operation integrates and standardizes the format of all continuous audio data from the beginning to the end of the speech in the buffer, making it a data unit that can be directly processed by the offline speech recognition engine. The structured data packet refers to a custom or standard data structure that includes at least raw audio data (i.e., the original Pulse Code Modulation (PCM) sample values) and necessary metadata, such as sampling rate and number of channels. The WAV format file in memory refers to a data block directly generated in system memory that conforms to the WAV file format specification, including a file header (RIFF header, fmt sub-block) and audio data sub-blocks. Through encapsulation, the originally continuous audio stream is transformed into discrete, self-contained speech segment data packets that can be processed independently later.

[0069] S3, the speech segment data packet is submitted to a general offline speech recognition engine, and the general offline speech recognition engine is used to perform overall recognition of the speech segment data packet to obtain the corresponding text recognition result. The general offline speech recognition engine is an offline speech recognition model that does not support streaming incremental decoding.

[0070] Specifically, the encapsulated speech segment data packet is sent as an independent, complete recognition request to a general-purpose offline speech recognition engine. Upon receiving the complete speech segment data packet, the engine invokes its internal recognition algorithm to decode and process the entire segment in one go. Because this engine is an offline speech recognition model that does not support streaming incremental decoding, it can utilize all audio information within the entire speech segment during the recognition process, including both forward and backward context, to perform globally optimal decoding and ultimately generate a complete text recognition result corresponding to that speech segment. After processing, the engine returns the text result.

[0071] In some embodiments, the offline speech recognition model is an offline ASR (Automatic Speech Recognition) model based on the Transformer (Transformer Model) architecture or the Conformer (Convolution-augmented Transformer) architecture.

[0072] Specifically, the Transformer architecture is a neural network model based on a self-attention mechanism, which can effectively capture long-range dependencies in sequential data and performs excellently in offline speech recognition tasks. The Conformer architecture is an enhanced architecture that combines a convolutional neural network module with the Transformer, and can simultaneously utilize the global modeling capability of the self-attention mechanism and the local feature extraction capability of the convolutional neural network, making it particularly suitable for processing the temporal and spectral characteristics of speech signals. These models are typical offline speech recognition models that do not support streaming incremental decoding; they require a complete input sequence before computation and output. It should be noted that the general offline speech recognition engine in this application is not limited to the two architectures mentioned above. Any other offline speech recognition model that does not support streaming incremental decoding, such as models based on recurrent neural networks (RNNs) or models based on bidirectional long short-term memory (BiLSTM) networks, can be applied to this application.

[0073] S4, receive the text recognition result returned by the general offline speech recognition engine, and output the text recognition result.

[0074] Specifically, after the general offline speech recognition engine completes the recognition processing of the speech segment data packet, it receives the returned text recognition result. This text recognition result is the complete text sequence corresponding to the speech segment. Subsequently, the text recognition result is output in a manner that can be perceived by the user or used by subsequent applications, and the output method can be flexibly selected according to the actual application scenario.

[0075] In some embodiments, outputting the text recognition result specifically includes:

[0076] The text recognition results are displayed in the user interface of the real-time interactive application.

[0077] Specifically, the real-time interactive applications refer to various application scenarios that require real-time voice interaction with users, such as online meeting caption generation, intelligent voice assistants, real-time translation, classroom dictation, and real-time interview transcription. In these applications, the recognized text results are presented in real-time in designated areas of the graphical user interface (such as caption areas, speech bubbles, and text boxes), enabling users to read and understand instantly. The display method can be sentence-by-sentence refresh, scrolling update, or overlay display, which can be configured according to the needs of the application scenario.

[0078] S5, determine whether the identification termination trigger condition has been met. If yes, end this process; otherwise, return to step S2.

[0079] Specifically, after outputting the text recognition result of the current speech segment, the system performs a termination condition check to determine whether to continue the real-time speech recognition process. The termination trigger condition is a preset criterion used to end the entire recognition process. If the current state meets any termination trigger condition, audio acquisition and recognition stop, and the entire process ends; if no termination trigger condition is met, the system automatically returns to step S2 to continue detecting speech activity in subsequent real-time audio streams to process the next speech segment, thereby achieving continuous, real-time speech recognition.

[0080] The termination trigger conditions include, but are not limited to, any of the following: receiving a stop command actively issued by the user (e.g., the user clicks the stop button on the user interface, enters a specific voice command such as "stop recognition", or closes the application); the audio input device disconnects or malfunctions; the operating system shuts down or the application is forcibly exited; the time during which no valid voice activity is detected exceeds a preset total silence timeout threshold (e.g., no one speaks for a long time after a meeting ends); the preset task termination condition is met (e.g., the preset recognition time has expired, the preset number of sentences has been completed); or automatic termination is triggered by the system detecting insufficient resources (e.g., low memory). The above termination conditions can be set individually or in combination according to actual application needs.

[0081] In one specific embodiment, this real-time speech recognition method is applied to a real-time speech recognition system. This system aims to process a continuous real-time audio stream in segments, then pass these segments to a general-purpose offline speech recognition engine that does not inherently support streaming processing. The system outputs results in sentence units, thereby achieving real-time recognition at the application layer. The structure of this real-time speech recognition system is as follows: Figure 2 As shown, the system includes an audio acquisition module (101), a speech activity detection module (VAD, 102), an audio segment caching module (103), a general offline speech recognition engine (104), and a recognition result output module (105). Detailed descriptions of each part are as follows:

[0082] 1. Audio Acquisition Module (101): Responsible for continuously acquiring raw audio data from audio input devices (such as microphones) to form a real-time audio stream.

[0083] 2. Voice Activity Detection Module (VAD, 102): Connected to the audio acquisition module (101), it is used to analyze the audio stream in real time. The core function of this module is to detect the start and end points of the voice. When the start of a voice is detected, it outputs a "voice start" signal; when the end of a voice is detected (for example, a period of silence lasting longer than a preset threshold), it outputs a "voice end" signal.

[0084] 3. Audio Segment Buffering Module (103): Receives the audio stream from the audio acquisition module (101) and operates according to the control signals output by the speech activity detection module (102). When a "speech start" signal is received, this module begins to buffer the subsequent audio data; when a "speech end" signal is received, this module stops buffering and encapsulates the buffered continuous audio data from start to end into an independent, complete speech segment.

[0085] 4. General Offline Speech Recognition Engine (104): This engine is a standard offline speech recognition model that does not support streaming incremental decoding. It receives complete speech segments encapsulated by the audio segment caching module (103) and processes them as an independent recognition task. The engine can utilize its complete bidirectional context analysis capabilities to perform high-precision decoding of the entire speech segment and output the final text result. The engine can be any public or private offline ASR model, such as models based on Transformer or Conformer architectures.

[0086] 5. Recognition result output module (105): Connected to a general offline speech recognition engine (104), used to receive the text results output by the engine and present them to the user, for example, by displaying them on a graphical user interface.

[0087] Based on the above system, such as Figure 3 As shown, the real-time speech recognition method may include the following steps:

[0088] Step S201: Start and continuously acquire real-time audio streams.

[0089] The system initializes, and the audio acquisition module (101) starts working, acquiring a continuous audio data stream from devices such as microphones.

[0090] Step S202: The voice activity detection module (102) monitors the audio stream in real time.

[0091] The speech activity detection module (102) continuously analyzes the acquired audio stream to determine whether it contains valid human speech activity.

[0092] Step S203: Determine if it is the start of voice recording. If so, trigger the start of buffering.

[0093] Once the speech activity detection module (102) detects the presence of speech energy and confirms it as the start of a valid speech, it sends a "speech start" command to the audio segment buffer module (103).

[0094] Step S204: The audio segment caching module (103) begins recording audio data.

[0095] Upon receiving the “voice start” command, the audio segment caching module (103) begins writing all audio data from that point in time into a temporary buffer.

[0096] Step S205: Continuously determine whether the voice has ended. If so, trigger the end of the buffering.

[0097] During the caching period, the voice activity detection module (102) continuously monitors the audio stream. When a silence is detected and the duration of the silence exceeds a preset threshold (e.g., 500 milliseconds), the module determines that the current sentence has ended and then sends a "voice end" command to the audio segment caching module (103).

[0098] Step S206: Stop recording and encapsulate into independent audio segments.

[0099] Upon receiving the “end of voice” instruction, the audio segment buffer module (103) immediately stops writing. Subsequently, it integrates all the audio data in the buffer from “start of voice” to “end of voice” into a structured, independent voice segment data packet (e.g., an object containing raw audio data and sampling rate information, or a WAV format file in memory).

[0100] Step S207: Submit the packaged complete speech segment to the general offline speech recognition engine.

[0101] The system sends the complete speech segment data packet generated in the previous step as a separate, complete recognition request to the general offline speech recognition engine (104). This process is equivalent to a standard file transcription request.

[0102] Step S208: The offline engine performs overall recognition on the fragment and returns the text result.

[0103] Upon receiving the speech segment, the general-purpose offline speech recognition engine (104) invokes its internal algorithm to analyze and decode the entire segment. Because it can utilize all the contextual information within the segment, the engine is able to produce high-precision recognition results. After processing, the engine returns the final text string.

[0104] Step S209: Output the recognition result and loop to wait for the next speech segment.

[0105] The recognition result output module (105) receives the text result and displays it to the user. At the same time, the system clears the cache and resets the state, returning to step S202 to continuously monitor the audio stream in preparation for processing the user's next sentence.

[0106] The following example, using real-time caption generation for an online meeting, illustrates the implementation process of this disclosure.

[0107] 1) Scenario: User A is speaking in an online meeting, and the meeting software needs to generate real-time subtitles for him / her. The software deploys the system described in this disclosure in the background, in which the general offline speech recognition engine (104) selects a high-precision offline model based on the Conformer architecture.

[0108] 2) Process:

[0109] Time T0: The meeting begins, and User A speaks: "Hello everyone, today's meeting will mainly discuss the financial report for the third quarter."

[0110] Audio Acquisition and VAD Monitoring: The audio acquisition module (101) continuously acquires the voice of user A. When user A utters the first word "da", the voice activity detection module (102) immediately detects the start of the voice and notifies the audio segment buffer module (103) to start recording.

[0111] Audio caching: The audio segment caching module (103) begins caching the audio data of the entire sentence spoken by user A.

[0112] Speech End Judgment: After user A finishes saying "financial report", there is a natural pause. When the silence duration of this pause exceeds the preset 500 milliseconds, the speech activity detection module (102) determines that the sentence has ended and notifies the audio segment caching module (103) to encapsulate and submit: The audio segment caching module (103) encapsulates the recorded audio of the entire sentence into an independent WAV format data block and submits it as a single request to the backend Conformer offline recognition engine via the network.

[0113] Offline Recognition: The Conformer offline recognition engine receives this complete audio data block. It doesn't need any streaming processing; instead, it decodes the entire sentence's audio in one go, just like processing a regular audio file, to obtain a high-precision text result: "Hello everyone, our meeting today will mainly discuss the third-quarter financial report."

[0114] The results show that the recognition result output module (105) displays the text completely in the subtitle area of ​​the conference interface.

[0115] 3) Results: The delay from when the user finishes speaking to when the subtitles appear is only a short silence detection time plus network transmission and offline engine processing time. The user experience is "subtitles appearing sentence by sentence," which, while not "word-by-word" real-time, fully meets the needs of real-time understanding. Most importantly, the accuracy of the subtitles benefits from a powerful offline model, far exceeding traditional streaming recognition solutions. When switching to an offline model that supports translation of less common languages, only the backend engine needs to be replaced; the entire frontend process remains unchanged.

[0116] It should be noted that the real-time speech recognition method provided in this disclosure has the following beneficial effects:

[0117] a) This disclosure, through the collaborative work of front-end speech activity detection and audio caching, divides the continuous real-time audio stream into independent speech segment data packets and submits them to a general offline speech recognition engine that does not support streaming incremental decoding for overall recognition. This removes the mandatory dependence of real-time speech recognition tasks on specific streaming model architectures and extends the range of model selection to any high-precision general offline model, breaking through the problem of limited model selection caused by traditional real-time recognition solutions that can only use streaming models.

[0118] b) This disclosure redefines the function of the speech activity detection module, making it assume the core role of a "task mode converter". It transforms a continuous real-time streaming task with extremely high requirements for low latency into a series of independent discrete batch processing tasks with high requirements for processing integrity. This enables the backend to use the global context information of the complete speech segment for one-time decoding, which significantly improves the recognition accuracy in long sentences, inverted sentences and scenarios requiring subsequent semantic judgment while maintaining the real-time interactive experience.

[0119] c) This disclosure implements a completely decoupled architecture between the front-end audio acquisition and processing and the back-end speech recognition engine. The front-end is responsible for converting the real-time stream into data blocks, while the back-end, as a "pluggable" module, focuses on processing data blocks. This allows the back-end recognition engine to be replaced by any more advanced and powerful offline model at any time without modifying the front-end interaction logic, thereby obtaining an extremely flexible and efficient performance upgrade path and continuously enjoying the latest technological benefits in the field of offline recognition.

[0120] The real-time speech recognition method provided in this disclosure detects speech activity in a real-time audio stream and caches and encapsulates it into independent speech segment data packets as needed. These data packets are then submitted to a general-purpose offline speech recognition engine that does not support streaming incremental decoding for overall recognition. A continuous processing flow involving result output and iterative judgment enables flexible selection of a general-purpose offline speech recognition engine in real-time speech recognition scenarios. This breaks the architectural binding limitations of streaming models and significantly broadens the model selection range for real-time speech recognition. Simultaneously, relying on the overall recognition capability of the general-purpose offline speech recognition engine for speech segment data packets, it fully leverages the recognition advantages of the general-purpose offline speech recognition model, effectively improving the accuracy of real-time speech recognition and enabling the practical application of high-precision offline recognition technology in real-time speech recognition scenarios. This solves the problem of limited model selection and low recognition accuracy in existing real-time speech recognition methods that can only rely on streaming models that support streaming incremental decoding.

[0121] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0122] Figure 4 This is a block diagram of a real-time speech recognition device provided in an embodiment of the present disclosure.

[0123] Reference Figure 4 This disclosure provides a real-time speech recognition device for performing the above-described real-time speech recognition method. The device includes:

[0124] Real-time audio stream acquisition module 11 is used to continuously acquire real-time audio streams;

[0125] The voice activity detection module 12 is connected to the real-time audio stream acquisition module 11 and is used to detect voice activity in the real-time audio stream. When voice is detected to start, the real-time audio stream is cached. When voice is detected to end, the caching is stopped, and the cached continuous audio data is encapsulated into an independent voice segment data packet.

[0126] The offline speech recognition module 13 is connected to the speech activity detection module 12 and is used to submit the speech segment data packet to the general offline speech recognition engine. The general offline speech recognition engine is used to perform overall recognition of the speech segment data packet to obtain the corresponding text recognition result. The general offline speech recognition engine is an offline speech recognition model that does not support streaming incremental decoding.

[0127] The text recognition output module 14 is connected to the offline speech recognition module 13 and is used to receive the text recognition result returned by the general offline speech recognition engine and output the text recognition result.

[0128] The loop judgment module 15 is connected to the text recognition output module 14 and is used to determine whether the recognition termination trigger condition has been met. If so, the process ends; otherwise, it returns to the voice activity detection module 12.

[0129] Optionally, the voice activity detection module 12 includes:

[0130] A human effective speech detection unit is used to detect whether the real-time audio stream contains effective human speech activity;

[0131] A cache start unit is configured to start caching the real-time audio stream in response to detecting the valid human speech activity, determining that speech has started;

[0132] The buffer termination unit is used to stop buffering in response to detecting silence in the real-time audio stream and the duration of the silence exceeding a preset threshold, which determines that the speech has ended.

[0133] Optionally, the cache start unit is specifically used for:

[0134] In response to the detection of the valid human voice activity, a voice start command is triggered;

[0135] According to the voice start command, the real-time audio stream is stored starting from the current time point.

[0136] Optionally, the cache termination unit is specifically used for:

[0137] In response to the detection of silence in the real-time audio stream, and the duration of the silence exceeding a preset threshold, a voice end command is triggered;

[0138] According to the voice end command, the buffering stops at the current time point.

[0139] Optionally, the voice activity detection module 12 further includes:

[0140] The data packet integration unit is used to integrate cached continuous audio data into a structured data packet containing raw audio data and sampling rate information, or into a WAV format waveform audio file in memory.

[0141] Optionally, the offline speech recognition model is an offline automatic speech recognition (ASR) model based on the Transformer architecture or the Conformer architecture with convolutional enhancement.

[0142] Optionally, the text recognition output module 14 includes:

[0143] The recognition result display unit is used to display the text recognition result in the user interface of a real-time interactive application.

[0144] Figure 5 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0145] Reference Figure 5 This disclosure provides an electronic device, which includes: at least one processor 701; at least one memory 702; and one or more I / O interfaces 703 connected between the processor 701 and the memory 702; wherein the memory 702 stores one or more computer programs that can be executed by the at least one processor 701, and the one or more computer programs are executed by the at least one processor 701 to enable the at least one processor 701 to perform the above-described real-time speech recognition method.

[0146] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the aforementioned real-time speech recognition method. The computer-readable storage medium may be volatile or non-volatile.

[0147] In summary, the real-time speech recognition method, apparatus, electronic device, and storage medium provided in this disclosure detect speech activity in real-time audio streams and cache and encapsulate them into independent speech segment data packets as needed. These data packets are then submitted to a general-purpose offline speech recognition engine that does not support streaming incremental decoding for overall recognition. A continuous processing flow involving result output and iterative judgment enables flexible selection of a general-purpose offline speech recognition engine in real-time speech recognition scenarios. This breaks the architectural binding limitations of streaming models and significantly broadens the model selection range for real-time speech recognition. Simultaneously, relying on the overall recognition capability of the general-purpose offline speech recognition engine for speech segment data packets, the recognition advantages of the general-purpose offline speech recognition model are fully utilized, effectively improving the accuracy of real-time speech recognition and enabling the application of high-precision offline recognition technology in real-time speech recognition scenarios. This solves the problem of limited model selection and low recognition accuracy in existing real-time speech recognition methods that can only rely on streaming models that support streaming incremental decoding.

[0148] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0149] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0150] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0151] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0152] This disclosure has disclosed exemplary embodiments, and although specific terminology has been used, it is for general illustrative purposes only and should not be construed as limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A real-time speech recognition method, characterized in that, The method includes: S1 continuously captures real-time audio streams; S2, perform voice activity detection on the real-time audio stream. When voice is detected to start, start caching the real-time audio stream. When voice is detected to end, stop caching and encapsulate the cached continuous audio data into an independent voice segment data packet. S3, the speech segment data packet is submitted to the general offline speech recognition engine, and the general offline speech recognition engine is used to perform overall recognition of the speech segment data packet to obtain the corresponding text recognition result. The general offline speech recognition engine is an offline speech recognition model that does not support streaming incremental decoding. S4, receive the text recognition result returned by the general offline speech recognition engine, and output the text recognition result; S5, determine whether the identification termination trigger condition has been met. If yes, end this process; otherwise, return to step S2.

2. The method according to claim 1, characterized in that, The step of performing voice activity detection on the real-time audio stream, which involves starting to buffer the real-time audio stream when voice activity is detected to begin and stopping buffering when voice activity is detected to end, specifically includes: Detect whether the real-time audio stream contains valid human speech activity; In response to the detection of the valid human speech activity, a speech start is determined, and the real-time audio stream is buffered. In response to the detection of silence in the real-time audio stream, and the duration of the silence exceeding a preset threshold, the system determines that the speech has ended and stops buffering.

3. The method according to claim 2, characterized in that, The step of responding to the detection of valid human speech activity, determining the start of speech, and initiating the buffering of the real-time audio stream specifically includes: In response to the detection of the valid human voice activity, a voice start command is triggered; According to the voice start command, the real-time audio stream is stored starting from the current time point.

4. The method according to claim 2, characterized in that, The response to detecting silence in the real-time audio stream, and the duration of the silence exceeding a preset threshold, determining that the speech has ended, and then stopping buffering, specifically includes: In response to the detection of silence in the real-time audio stream, and the duration of the silence exceeding a preset threshold, a voice end command is triggered; According to the voice end command, the buffering stops at the current time point.

5. The method according to claim 1, characterized in that, The process of encapsulating cached continuous audio data into an independent speech segment data packet specifically includes: The cached continuous audio data can be integrated into a structured data packet containing raw audio data and sampling rate information, or integrated into a WAV format waveform audio file in memory.

6. The method according to claim 1, characterized in that, The offline speech recognition model is an offline automatic speech recognition (ASR) model based on the Transformer architecture or the Conformer architecture with convolutional enhancement.

7. The method according to claim 1, characterized in that, The output of the text recognition result specifically includes: The text recognition results are displayed in the user interface of the real-time interactive application.

8. A real-time speech recognition device, characterized in that, The device includes: The real-time audio stream acquisition module is used to continuously acquire real-time audio streams. The voice activity detection module is connected to the real-time audio stream acquisition module and is used to detect voice activity in the real-time audio stream. When voice is detected to start, the real-time audio stream is cached. When voice is detected to end, the caching is stopped, and the cached continuous audio data is encapsulated into an independent voice segment data packet. An offline speech recognition module, connected to the speech activity detection module, is used to submit the speech segment data packet to a general offline speech recognition engine, and use the general offline speech recognition engine to perform overall recognition of the speech segment data packet to obtain the corresponding text recognition result. The general offline speech recognition engine is an offline speech recognition model that does not support streaming incremental decoding. A text recognition output module, connected to the offline speech recognition module, is used to receive the text recognition result returned by the general offline speech recognition engine and output the text recognition result; The loop judgment module, connected to the text recognition output module, is used to determine whether the recognition termination trigger condition has been met. If so, the process ends; otherwise, it returns to the voice activity detection module.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the real-time speech recognition method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the real-time speech recognition method as described in any one of claims 1-7.

Images