A real-time speech recognition method and system integrating voiceprint gating verification

Abstract

This invention discloses a real-time speech recognition method and system integrating voiceprint gating verification, relating to the fields of deep learning and speech recognition technology. The method includes: performing endpoint detection on a real-time audio stream to obtain valid speech segments; recognizing the valid speech segments and outputting semantic segmentation results with timestamps; extracting audio segments corresponding to each semantic unit based on the timestamps; performing voiceprint verification on the audio segments of each semantic unit; if the voiceprint verification of any semantic unit meets preset conditions, then determining that the entire speech segment originates from the target speaker and outputting the recognition result. This invention significantly improves speech recall in complex acoustic environments through semantic-level fine-grained voiceprint comparison and a "short-circuit decision" mechanism, and ensures low latency and high concurrency performance through an asynchronous decoupling architecture, making it widely applicable to identity-sensitive scenarios such as intelligent customer service and business hall quality inspection.

Description

Technical Field

[0001] This invention relates to the field of deep learning neural network technology, and in particular to a real-time speech recognition method and system that integrates voiceprint gating verification, especially a voiceprint gating verification method based on semantic segmentation, for realizing credible verification of the identity of the target speaker in a real-time voice interaction system. Background Technology

[0002] In real-time voice interaction systems, especially in high-concurrency, high-noise scenarios such as intelligent customer service, meeting transcription, and service hall quality inspection, ensuring that automatic speech recognition results originate solely from the target speaker has become a critical requirement for improving system reliability and security. Currently, the industry mainly employs the following two technical approaches: First, there are commercial closed-source integration solutions. Voice cloud services, represented by iFlytek and Alibaba Cloud, provide integrated "ASR + voiceprint verification" capabilities, but their underlying architecture is not open source, resulting in low deployment flexibility and billing based on the number of concurrent connections, making it difficult to meet enterprises' customized needs for data privacy and cost control.

[0003] Second, there is the open-source toolchain splicing solution. For example, the FunASR toolkit and CAM++ voiceprint model open-sourced by Alibaba DAMO Academy. However, these modules are task-decoupled and do not support dynamic speaker registration and semantic-level voiceprint gating in real-time streaming scenarios. Developers have high engineering complexity in building their own systems, and it is difficult to guarantee low latency and high concurrency stability.

[0004] The existing technology has the following key problems: First, voiceprint verification mostly adopts the "unified verification of the whole sentence" strategy, which can easily lead to the discarding of the whole sentence's valid content due to local noise or interruptions from others, resulting in low recall rate; Second, the ASR and SV processes are strongly coupled, and voiceprint inference is serially accessed to the ASR post-processing link, which significantly increases end-to-end latency; In addition, there is a lack of connection-oriented dynamic identity management and resource isolation mechanisms, which can easily cause different session voiceprint states to interfere with each other in multi-user concurrent scenarios, resulting in insufficient system robustness.

[0005] Therefore, there is an urgent need for a real-time speech recognition method and system that can achieve fine-grained voiceprint verification with semantic granularity while ensuring low latency and high concurrency, and can make "short-circuit release" decisions to retain the target speaker's voice with high recall. Summary of the Invention

[0006] This invention aims to at least solve some of the technical problems existing in the prior art. To this end, embodiments of this invention propose a real-time speech recognition method integrating voiceprint gating verification, comprising: S100: Receive a real-time audio stream from the client and perform voice endpoint detection on the audio stream to segment out valid voice segments; S200, perform real-time speech recognition on the effective speech segment and output a semantic segmentation result with a timestamp, wherein the semantic segmentation result contains at least one semantic unit and its corresponding audio time interval; S300, for the effective speech segment, extract at least one audio segment corresponding to the semantic unit based on the timestamp in the semantic segmentation result; for the extracted audio segments corresponding to the at least one semantic unit, perform voiceprint feature extraction and verification respectively; S400, if the voiceprint verification result of any semantic unit meets the preset conditions, then the effective speech segment is determined to originate from the target speaker, and the recognized text of the effective speech segment is output.

[0007] Furthermore, the step of extracting and verifying voiceprint features for the audio segments corresponding to the at least one semantic unit includes: submitting the voiceprint feature extraction and verification task to an asynchronous thread pool for processing to avoid blocking the main process of real-time speech recognition.

[0008] Furthermore, the method also includes: Upon first receiving an audio stream connection from a specific client, the audio data of the first N valid speech segments of that connection is accumulated, and a voiceprint template of the main speaker corresponding to that connection is generated and stored based on the accumulated audio data; where N is an integer greater than or equal to 1. The process of extracting and verifying voiceprint features involves comparing the extracted voiceprint features with the speaker's voiceprint template.

[0009] Furthermore, the voiceprint template, audio buffer, and verification status are stored in isolation using the client connection identifier as the key to achieve context isolation between different connections.

[0010] Furthermore, the real-time speech recognition is implemented based on an open-source speech recognition toolkit, and the voiceprint feature extraction and verification are implemented based on an open-source voiceprint model.

[0011] This application also provides a real-time speech recognition system that integrates voiceprint gating verification, including: The audio receiving and processing module is used to receive real-time audio streams and perform voice endpoint detection; The real-time speech recognition module is used to identify valid speech segments and output semantic segmentation results with timestamps. The semantic segment extraction module is used to extract audio segments corresponding to at least one semantic unit based on the timestamp in the semantic segmentation result; the asynchronous voiceprint verification module includes a dedicated thread pool for voiceprint feature extraction and verification of the extracted audio segments; The result decision and output module is used to decide whether to output the recognized text of the corresponding valid speech segment based on the voiceprint verification result. If the verification of any semantic unit passes, the entire recognized text segment is output.

[0012] Furthermore, it also includes: a voiceprint template management module, which automatically registers the main speaker's voiceprint template when a client connection is established for the first time, and stores and manages it using the connection identifier as the key.

[0013] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above method.

[0014] This application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the methods described above.

[0015] This invention achieves fine-grained semantic voiceprint verification and a "short-circuit release" decision for complete sentences segmented by VAD, ensuring both low latency and high concurrency. This means that if a portion passes the verification, the entire sentence is recognized, thus preserving the target speaker's voice with high recall in complex acoustic environments while ensuring system deployability. This invention systematically resolves this contradiction by deeply integrating FunASR semantic segmentation timestamps, asynchronous decoupling architecture, and the short-circuit decision strategy. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the architecture of the real-time speech recognition system with fused voiceprint verification provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the "short-circuit decision" voiceprint verification mechanism based on semantic segmentation timestamps in an embodiment of the present invention; Figure 3 This is a schematic diagram of the system operation log in an embodiment of the present invention. Detailed Implementation

[0018] Glossary Automatic speech recognition (ASR) refers to the process of automatically converting human speech signals into corresponding text using computer technology. It is one of the core technologies of human-computer voice interaction and is widely used in scenarios such as intelligent customer service, meeting transcription, voice assistants, and in-vehicle systems.

[0019] Voice Activity Detection (VAD): It is a key preprocessing module for real-time ASR, used to dynamically detect the start and end boundaries of valid speech segments from a continuous long audio stream, trigger the recognition process in a timely manner, and suppress silent or noise interference through parameter settings to form a valid audio interval. An efficient VAD can significantly reduce ineffective calculations and improve the system response speed.

[0020] Acoustic Modeling (AM): It is the core engine of ASR, and its architecture directly affects the recognition performance and latency of the system.

[0021] Language Model (LM): In a real-time system, it is mainly used for shallow fusion or rescoring in the decoding stage to correct homophones or word order errors that may occur during the recognition process of the Acoustic Model AM. To balance efficiency and effectiveness, a language model implemented based on Wesp-FST is integrated in the system of this article to improve text fluency without significantly increasing latency.

[0022] Punctuation Restoration (PUNC): It belongs to the postprocessing module of ASR and can be used for punctuation prediction of the text output by a real-time speech recognition model, which is crucial for the readability of real-time transcription results.

[0023] Inverse Text Normalization (ITN): It converts the canonical forms such as numbers and dates in the ASR output into natural writing formats, such as "2025" → "two thousand and twenty-five", to improve the professionalism and readability of the text. In a real-time system, ITN needs to have the ability to respond quickly.

[0024] Speaker Verification (SV): The role of this module is particularly important in real-time interaction scenarios shared by multiple users and scenarios with extremely noisy surrounding environments. It can be used to filter the speech of non-target speakers and ensure the security and pertinence of the recognition results.

[0025] The embodiments of the present invention will be described in detail below, and the examples of the embodiments are shown in the accompanying drawings. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present invention and should not be construed as a limitation to the present invention.

[0026] The following is based on Figure 1 and Figure 2 describe a real-time speech recognition system and method for fusing voiceprint gating verification provided by the embodiments of the present invention.

[0027] As Figure 1As shown, this invention is applicable to real-time voice interaction scenarios in environments with multiple speakers or high noise levels. Typical applications include: intelligent customer service coaching, service quality inspection for sales staff in business halls, on-site service quality inspection by installation and maintenance technicians, and in-vehicle voice assistants. In these scenarios, the system must ensure that only the voice of the target speaker (such as a customer service agent or a conference speaker) is recognized and output, avoiding interruptions from others or interference from environmental noise.

[0028] The system adopts a client-server architecture and transmits real-time audio streams based on the WebSocket protocol. The server is deployed on a general-purpose server, running the voiceprint-gated speech recognition system proposed in this invention. This invention provides a real-time speech recognition system integrating voiceprint gating verification. This system adopts a client-server architecture and transmits real-time audio streams based on the WebSocket protocol. The server is deployed on a general-purpose server, running the voiceprint-gated speech recognition system A1 proposed in this invention. The system software architecture includes the following core modules: Audio input and buffer module 110: Receives a 16kHz single-channel PCM audio stream sent by the client via WebSocket, and maintains an independent circular buffer according to the connection ID for subsequent voiceprint feature extraction.

[0029] Real-time speech transcription and post-processing module 120: integrates the FunASR open-source engine, including sub-modules such as VAD, streaming ASR, offline ASR, PUNC, and ITN, and outputs recognized text with semantic segmentation and timestamps.

[0030] Semantic segment extraction module: Based on the timestamp information output by the real-time speech transcription and post-processing module 120, extract the specific audio segment corresponding to each semantic unit from the audio buffer module 110.

[0031] Asynchronous voiceprint verification module 140: Includes a dedicated thread pool for voiceprint feature extraction and similarity comparison of audio segments input by the semantic segment extraction module, avoiding blocking the main ASR process.

[0032] Result Decision and Output Module 150: Based on the result returned by the asynchronous voiceprint verification module 140, executes the "short-circuit decision" logic to determine whether to send the recognized text back to the client via WebSocket.

[0033] Voiceprint Template Management Module 160: Used to automatically register the main speaker's voiceprint template upon initial connection, and to store and manage it in isolation using the connection ID as the key.

[0034] The following is combined with Figure 2 , Figure 3 The core "short-circuit decision" voiceprint verification mechanism of this invention is described in detail.

[0035] Once the system receives a continuous audio stream from the client, the processing flow is as follows: The audio input and buffering module 110 first divides the audio stream into a complete valid speech segment (such as a complete question) using the VAD module. This valid speech segment is then sent to the real-time speech transcription and post-processing module 120, where the FunASR engine performs recognition. Figure 2 As shown, while outputting the complete recognized text (such as "Hello, how's the weather today? Shall we go out and play together?"), the engine also outputs the semantic segmentation results with timestamps corresponding to the text. A complete sentence may be divided into multiple semantic units, such as: ["Hello", "How's the weather today?", "Shall we go out and play together?"], with each unit corresponding to a time interval (for example, "Hello" corresponds to the interval [9400, 9840] ms).

[0036] The semantic segment extraction module extracts the audio segment corresponding to each semantic unit from the audio buffer based on these timestamps.

[0037] The asynchronous voiceprint verification module 140 allocates these audio segments to the thread pool, extracts voiceprint features one by one, and compares them with the cosine similarity of the registered main speaker's voiceprint template for the connection.

[0038] The result decision and output module 150 applies the "short-circuit decision" logic: as long as the similarity of any semantic unit reaches the preset threshold (such as the similarity of the first "hello" being 0.64872, which is considered to be passing), the "pass" signal is immediately triggered, the verification task of other unfinished semantic units is terminated, and the recognized text of the entire speech is output.

[0039] Compared to traditional techniques, which require extracting global voiceprint features from the entire VAD segment for comparison, any local noise or interruptions can interfere with the global similarity, causing the entire sentence to be discarded. This invention refines the verification granularity to semantic units and employs a "one-vote pass" short-circuit mechanism, greatly preserving valid content and improving recall.

[0040] In one specific embodiment, to ensure low latency, the voiceprint verification task is designed to be executed asynchronously. After generating the timestamped text, the ASR main process immediately submits the voiceprint verification task to the thread pool, while the ASR itself continues processing the next audio stream, achieving asynchronous decoupling between ASR and SV. This avoids the blocking caused by serially waiting for SV inference results in traditional solutions, ensuring that the end-to-end latency is controlled within 400ms.

[0041] The asynchronous decoupling architecture of ASR and Voiceprint Verification (SV) separates the voiceprint verification task from the main ASR recognition process and executes it asynchronously through a dedicated thread pool. The ASR module immediately continues processing the next audio stream after generating timestamped text, and the voiceprint verification result is subsequently filtered through callbacks or event mechanisms. Traditional integration solutions often serially integrate SV into the ASR post-processing link, directly increasing end-to-end latency for voiceprint inference. This invention, through task decoupling, ensures that the main ASR process is not affected by SV computation, avoiding blocking. End-to-end latency is significantly reduced (measured <400ms), meeting the response requirements of real-time voice interaction scenarios; resource utilization is high: ASR and SV can utilize CPU resources in parallel, improving system concurrency.

[0042] In another specific embodiment, to adapt to high-concurrency scenarios with multiple users, the system supports automatic voiceprint registration upon first connection. When a client establishes a connection via WebSocket for the first time, the voiceprint template management module 160 accumulates the audio data of the first two valid VAD speech segments of that connection, concatenates them, and generates and stores a unique speaker voiceprint template for that connection. All voiceprint states, audio buffers, and mutexes are isolated using the WebSocket connection ID as the key, ensuring that the contexts of different connections are independent and avoiding identity confusion and resource contention.

[0043] When a client establishes a connection for the first time, the system automatically accumulates the audio of the first few (e.g., two) valid speech segments divided by VAD, splices them together, and generates and stores a voiceprint template specific to the main speaker for that connection. All voiceprint states, audio buffers, and mutexes are isolated using the WebSocket connection ID as the key. Existing open-source solutions lack dynamic registration capabilities and require pre-recorded voiceprints or reliance on external identity systems. In multi-user scenarios, without connection isolation, voiceprint templates from different sessions are easily confused, and non-thread-safe components (such as the FBank feature extractor) are prone to crashing under high concurrency.

[0044] In this embodiment, no pre-registration is required; the speaker's voiceprint is dynamically registered immediately after connection, improving user experience and deployment flexibility. No deadlocks, memory leaks, or identity obfuscation were observed in tests on a 16-core, 32-thread CPU server with more than 15 concurrent connections, demonstrating production-grade robustness.

[0045] Furthermore, the entire system is built on the fully open-source FunASR (including Paraformer, VAD, and PUNC) and CAM++ voiceprint models. Engineering optimizations include PCM memory passthrough (eliminating file I / O conversion overhead), connection lifecycle management, and structured logging (recording core parameters such as voiceprint similarity scores and verification segment intervals), ensuring low system cost, private deployment capability, high maintainability, and observability.

[0046] like Figure 1 The diagram illustrates the timing of the asynchronous decoupled architecture of ASR and SV. Immediately after ASR completes processing, a callback is sent to the result decision module, which then waits for the results of the asynchronous voiceprint verification tasks. The result decision module makes the final output decision when all voiceprint verification tasks are completed or a short-circuit condition is triggered.

[0047] The embodiments of this application systematically solve the core challenge in real-time speech recognition: "how to ensure the credibility of the speech source with high recall under the premise of low latency and high concurrency." The short-circuit decision mechanism is at its core, and various technical elements work together to support its efficient and stable overall solution in real-world business scenarios.

[0048] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions, and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A real-time speech recognition method integrating voiceprint gating verification, characterized in that, include: S100: Receive a real-time audio stream from the client and perform voice endpoint detection on the audio stream to segment out valid voice segments; S200, perform real-time speech recognition on the effective speech segment and output a semantic segmentation result with a timestamp, wherein the semantic segmentation result contains at least one semantic unit and its corresponding audio time interval; S300, for the effective speech segment, extract at least one audio segment corresponding to the semantic unit based on the timestamp in the semantic segmentation result; for the extracted audio segments corresponding to the at least one semantic unit, perform voiceprint feature extraction and verification respectively; S400, if the voiceprint verification result of any semantic unit meets the preset conditions, then the effective speech segment is determined to originate from the target speaker, and the recognized text of the effective speech segment is output.

2. The method according to claim 1, characterized in that, The step of extracting and verifying voiceprint features for the audio segments corresponding to the at least one semantic unit includes: submitting the voiceprint feature extraction and verification task to an asynchronous thread pool for processing to avoid blocking the main process of real-time speech recognition.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Upon first receiving an audio stream connection from a specific client, the audio data of the first N valid speech segments of that connection is accumulated, and a voiceprint template of the main speaker corresponding to that connection is generated and stored based on the accumulated audio data; where N is an integer greater than or equal to 1. The process of extracting and verifying voiceprint features involves comparing the extracted voiceprint features with the speaker's voiceprint template.

4. The method according to claim 3, characterized in that, The voiceprint template, audio buffer, and verification status are stored in isolation using the client connection identifier as the key to achieve context isolation between different connections.

5. The method according to claim 1, characterized in that, The real-time speech recognition is implemented based on an open-source speech recognition toolkit, and the voiceprint feature extraction and verification are implemented based on an open-source voiceprint model.

6. A real-time speech recognition system integrating voiceprint gating verification, characterized in that, include: The audio receiving and processing module is used to receive real-time audio streams and perform voice endpoint detection; The real-time speech recognition module is used to identify valid speech segments and output semantic segmentation results with timestamps. The semantic segment extraction module is used to extract at least one audio segment corresponding to a semantic unit based on the timestamp in the semantic segmentation result. The asynchronous voiceprint verification module includes a dedicated thread pool for extracting and verifying voiceprint features from extracted audio segments. The result decision and output module is used to decide whether to output the recognized text of the corresponding valid speech segment based on the voiceprint verification result. If the verification of any semantic unit passes, the entire recognized text segment is output.

7. The system according to claim 6, characterized in that, Also includes: The voiceprint template management module is used to automatically register the main speaker's voiceprint template when a client connection is established for the first time, and to store and manage it using the connection identifier as the key.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images