Real-time voice interaction-oriented multi-person voice separation and recognition system and real-time voice interaction-oriented multi-person voice separation and recognition method

Abstract

The invention discloses a multi-person voice separation and recognition system and method for real-time voice interaction, and belongs to the technical field of voice signal processing. The technical principle comprises the following steps: segmenting streaming voice through an audio acquisition module and storing the streaming voice into a circular queue; the preprocessing module converts the wav file into a standard wav file; the voice separation module adopts a SepFormer model to separate a speaker track; the voice recognition module executes end-to-end transcription on the voice through a Paraformer model; and the result management module binds the speaker number and the text and then performs structured output. The multi-thread asynchronous architecture enables acquisition, separation and identification to be processed in parallel, and low delay is realized by combining a circular queue; the SepFormer and the Paraformer are cooperated to improve the separation precision and the recognition accuracy of the overlapped voices; jSON / database multi-format output is supported, and the interaction requirements of real-time conferences and intelligent devices are met.

Description

Technical Field

[0001] The present invention relates to the technical field of speech signal processing and speech recognition, and in particular to a multi-person speech separation and recognition system and method for real-time speech interaction. Background Art

[0002] With the continuous development of voice interaction technology, voice control has been gradually applied to various fields such as smart homes, robot control, remote conferencing, and medical services. In recent years, deep learning-based speech separation technology has developed rapidly, especially separation models using time-frequency masks (TF-Mask) and Transformer structures, which have been able to achieve high-quality speaker speech separation in offline environments. However, the following challenges still exist in the coordinated processing of real-time speech acquisition, separation, and recognition:

[0003] 1. Poor speaker information relevance: After separation, it is difficult to maintain consistency between the audio tracks and the speaker identity, which affects subsequent semantic processing or identity recognition;

[0004] 2. Difficulty in module decoupling: Differences in data formats and processing rhythms between modules such as voice acquisition, separation, and recognition affect system stability.

[0005] To solve the above problems, it is urgent to build a multi-person speech separation and recognition system for real-time voice interaction, realize integrated closed-loop processing from audio acquisition, speech unmixing, speech recognition to command parsing, and ensure the accuracy of the system in multi-person natural conversation scenarios. Summary of the Invention

[0006] The purpose of the present invention is to provide a multi-person speech separation and recognition system and method for real-time voice interaction, which solves the problems of low accuracy, high latency, and strong module coupling in overlapping speech recognition.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A multi-person speech separation and recognition system for real-time speech interaction, including:

[0009] The audio acquisition module collects the voices of multiple people, divides the audio data into audio frames of a set length and stores them in a circular buffer queue;

[0010] An audio preprocessing module, connected to the audio acquisition module, converting the audio frames in the circular buffer queue into files in a standard format;

[0011] A speech separation module, connected to the audio preprocessing module, loads a predictive separation model based on deep learning, separates the WAV file, and outputs independent speech tracks for each speaker;

[0012] A speech recognition module, connected to the speech separation module, normalizes the independent speech tracks of each speaker, loads the automatic speech recognition model to perform end-to-end transcription on each speech track, and generates text with punctuation and segmentation;

[0013] A result management module, connected to the speech recognition module, binds the transcribed text to the corresponding speaker number to form a structured result, and outputs it in a preset format;

[0014] This solution achieves the following benefits: enabling real-time multi-person voice interaction processing, handling overlapping speech, improving speech recognition accuracy, outputting structured results, and providing a complete, automated end-to-end solution.

[0015] As a further improvement of the above technical solution: the audio acquisition module, speech separation module, and speech recognition module adopt a multi-threaded asynchronous architecture, and independent threads respectively perform acquisition, separation, and recognition tasks.

[0016] The beneficial effects of this solution are: improving the system's real-time performance and response speed, increasing the system's processing capabilities, and reducing the coupling between modules.

[0017] As a further improvement of the above technical solution: the audio frame with a set duration is 2 seconds.

[0018] The beneficial effects of this solution are: ensuring real-time performance and low latency. The 2-second frame length ensures separation / recognition accuracy while controlling the end-to-end delay within an acceptable range, reducing the computational load.

[0019] As a further improvement of the above technical solution: the audio acquisition module adopts an asynchronous multi-threaded mode. Every time the acquisition of an audio frame is completed, a separate thread is immediately started to post-process the audio frame, and the separate thread and the audio acquisition main thread realize data interaction through a circular cache queue.

[0020] The beneficial effects of this solution are: fault isolation, and continuous collection without interruption when sudden computing delays occur.

[0021] As a further improvement of the above technical solution: the predictive separation model in the speech separation module is of Model, the predictive separation model includes an encoder, a mask network and a decoder, the encoder is a one-dimensional convolution layer, the mask network includes a layer normalization layer, a linear layer, and , the decoder is a transposed convolutional layer.

[0022] The beneficial effects of this scheme are: efficient feature extraction by the encoder, accurate speaker separation by the mask network, and speech reconstruction by the decoder.

[0023] As a further improvement of the above technical solution: the automatic speech recognition model provided by FunASR The normalization process unifies the sampling rate of the voice track to 16kHz and sets the channel to mono.

[0024] The beneficial effects of this solution are: unifying the input audio to this specification, ensuring that the input data is consistent with the data distribution of model training / expected input, simplifying the module logic, and focusing on the core speech transcription task.

[0025] As a further improvement of the above technical solution: the preset format of the result management module includes JSON format, TXT format or database record format, and the structured results are integrated and output through independent threads, and are decoupled from the audio collection and separation threads through thread queues and task pools.

[0026] The beneficial effects of this solution are: supporting multiple preset output formats, enhancing system flexibility and downstream compatibility.

[0027] The present invention also provides a method for multi-person speech separation and recognition for real-time speech interaction, comprising the following steps:

[0028] S1 Audio Capture and Caching: Use the audio capture tool to perform streaming voice capture in multi-person scenarios, divide the captured audio data into audio frames of a set length, and store them in a circular cache queue;

[0029] S2 audio preprocessing and storage: converting the audio frames in the circular buffer queue into a wav file in standard PCM format, where the sampling rate of the wav file is consistent with the input requirements of the subsequent speech separation model;

[0030] S3 Multi-speaker Speech Separation: The pre-processed WAV file is fed into a pre-trained speech separation model based on deep learning. The encoder of the speech separation model extracts mixed speech features, the mask network estimates the mask of each speaker, and the decoder reconstructs the independent speech tracks of each speaker.

[0031] S4 automatic speech recognition: Normalizes the separated independent speech tracks of each speaker, loads the automatic speech recognition model, performs end-to-end transcription on each track, generates the corresponding text, and adds sentence segments and punctuation;

[0032] S5 result integration and output: Bind the transcribed text to the corresponding speaker number to form a "speaker-text" structured result, which is output in a preset format to support subsequent semantic understanding or control tasks.

[0033] This solution offers the following benefits: It utilizes a multi-threaded asynchronous architecture for streaming processing, decoupling the audio acquisition, separation, and recognition modules from the task pool via a circular cache queue, ensuring stable system operation with low latency and preventing single modules from blocking the main process. It also offers excellent modular scalability: each module functions independently, supporting the integration of expansion modules such as speaker recognition and semantic understanding. The structured "speaker-to-text" output provides standardized input for subsequent tasks such as command parsing and intent recognition.

[0034] As a further improvement of the above technical solution: the audio acquisition tool is PyAudio, and the acquisition parameters are set to: sampling rate 16kHz, mono, and acquisition buffer size 1024 sampling points.

[0035] The beneficial effects of this solution are: improving the reliability and portability of the entire acquisition module, reducing end-to-end latency, clearly specifying all key acquisition parameters, eliminating configuration ambiguity, and ensuring consistent behavior in different deployment environments.

[0036] As a further improvement of the above technical solution: the deep learning-based predictive separation model for multi-person speech separation is the SepFormer model provided by SpeechBrain, and the encoder is a one-dimensional convolutional layer (Conv1D), which is used to convert the mixed speech signal into a feature representation; the mask network includes a layer normalization layer, a linear layer and a SepFormerBlock, and the SepFormerBlock is composed of an IntraTransformer and an InterTransformer, the IntraTransformer is used to construct a short dependency relationship of feature blocks, and the InterTransformer is used to construct a long dependency relationship between feature blocks; the decoder is a transposed convolutional layer, which is used to reconstruct an independent speech track based on the mask and encoder features.

[0037] This solution has the following benefits: It improves the core performance indicators (clarity and fidelity) of multi-person speech separation, especially in highly overlapping and complex acoustic environments. It also ensures efficiency, robustness, and ease of use, and enhances the accuracy and reliability of the entire real-time speech separation and recognition system.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] First, improved recognition accuracy: Through an end-to-end speech separation and recognition process, the SepFormer model is used to accurately separate overlapping speech from multiple speakers. Combined with the high-accuracy transcription capabilities of the Paraformer model, this significantly improves speech recognition accuracy in multi-speaker scenarios.

[0040] Second, real-time performance and stability are enhanced: a multi-threaded asynchronous architecture is used to implement streaming processing. The audio acquisition, separation, and recognition modules are decoupled from the task pool through a circular cache queue, ensuring stable system operation with low latency (2-3 seconds / segment) and preventing a single module from blocking the main process.

[0041] Third, the system boasts good modular scalability: Each module is functionally independent and supports the integration of expansion modules such as speaker recognition and semantic understanding. The structured "speaker-text" output provides standardized input for subsequent tasks such as command parsing and intent recognition.

[0042] Fourth, it has a wide range of application scenarios: it can be directly applied to various real-time voice interaction scenarios such as intelligent voice assistants, multi-user dialogue systems, remote meeting minutes, robot voice control, etc., and has strong practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a block diagram of a multi-person speech separation and recognition system for real-time speech interaction;

[0044] Figure 2 It is a flowchart of audio acquisition and audio preprocessing;

[0045] Figure 3 It is a flowchart of speech separation. DETAILED DESCRIPTION

[0046] The present invention is further described below with reference to the accompanying drawings and specific embodiments.

[0047] Example 1:

[0048] Reference Figure 1 and Figure 2 The present invention discloses a real-time speech separation and recognition system for multi-person scenarios, which mainly includes an audio acquisition module, an audio preprocessing module, a speech separation module, a speech recognition module, and a result management module. These modules work together to achieve real-time separation and recognition of multi-person speech. This embodiment corresponds to the specific implementation of audio acquisition and audio preprocessing, and the steps are as follows:

[0049] S101 audio acquisition initialization: Initialize the PyAudio acquisition device, set the acquisition parameters to a sampling rate of 16kHz, mono input, and an acquisition buffer size of 1024 samples; use asynchronous multi-threading to start real-time acquisition, with the main thread responsible for audio data capture and the child thread responsible for data caching and subsequent task scheduling to avoid blocking the acquisition process.

[0050] S102 Framing and circular buffering: The collected continuous audio data is divided into audio frames of approximately 2 seconds in length (each frame contains 32,000 sampling points) according to time slices, and stored in a circular buffer queue; the maximum buffer length of the circular buffer queue is set to 5 frames to ensure data continuity while avoiding memory overflow.

[0051] S201 Audio Preprocessing: The audio preprocessing module reads audio frames from the circular buffer queue and converts them into standard PCM format WAV files (16-bit depth, 16kHz sampling rate). It also performs endpoint detection and silence trimming on the audio frames, eliminating invalid segments with signal strength below the threshold, reducing the interference of invalid data on the subsequent separation model.

[0052] Example 2:

[0053] Reference Figure 3 This embodiment corresponds to the specific implementation of speech separation, based on the pre-trained SepFormer model of the SpeechBrain platform, the steps are as follows:

[0054] S301 Speech Feature Coding: The speech separation module loads the SepFormer model and inputs the preprocessed WAV file into the model encoder; the encoder uses a one-dimensional convolutional layer (Conv1D, kernel size 3, stride 1) to encode the input speech signal and extract the time-frequency domain feature representation.

[0055] S302 Feature Processing: The features output by the encoder are divided into feature blocks (chunks) of size 256 after layer normalization and linear layer processing; the feature blocks are input into SepFormerBlock, where IntraTransformer independently processes each feature block to build short dependencies, and InterTransformer transforms the output of IntraTransformer to build long dependencies between blocks, realizing global temporal information modeling.

[0056] S303 Mask Generation: The features processed by SepFormerBlock are used to generate a time-frequency mask (TF-Mask) for each speaker through the PReLU activation function and a linear layer. The decoder uses a transposed convolutional layer (TransposedConv1D, kernel size 3, stride 1) to multiply the mask with the encoder output features and reconstruct them to obtain an independent speech track (waveform file) for each speaker.

[0057] Example 3:

[0058] Reference Figure 1 The specific implementation of the speech recognition module and the result management module in this embodiment is based on the FunASR Paraformer model. The steps are as follows:

[0059] S401: The speech recognition module normalizes the separated speaker audio tracks to ensure that the sampling rate is 16 kHz and the number of channels is mono, and saves them as standard WAV format files. Each audio track is associated with a unique speaker ID (such as Speaker_1 or Speaker_2).

[0060] S402: Load the FunASR Paraformer pre-trained model and perform end-to-end transcription on each normalized speech track in turn. The model automatically generates text with punctuation and segmentation (such as "Hello, please turn on the air conditioner.") through feature extraction (Mel-spectrogram), acoustic modeling (feedforward attention network), language modeling (n-gram language model), and decoding.

[0061] S5: The result management module binds the transcribed text to the corresponding speaker ID to form structured data {"speaker_id":"Speaker_1","text":"Hello, please turn on the air conditioner."}; after all results are sorted by timestamp, they can be output as a JSON format file or stored in a relational database to support subsequent semantic understanding module calls (such as extracting the keyword "turn on the air conditioner" to execute a control command).

[0062] This article uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method of the present invention and its core ideas. The above is only a preferred implementation method of the present invention. It should be pointed out that due to the limitations of textual expression, there are objectively infinite specific structures. For ordinary technicians in this technical field, without departing from the principles of the present invention, they can make several improvements, modifications or changes, and can also combine the above technical features in an appropriate manner; these improvements, modifications, changes or combinations, or the direct application of the inventive concept and technical solution to other occasions without improvement, should be regarded as the scope of protection of the present invention.

Claims

1. A multi-person speech separation and recognition system for real-time speech interaction, characterized in that: include: The audio acquisition module collects the voices of multiple people, divides the audio data into audio frames of a set length and stores them in a circular buffer queue; An audio preprocessing module, connected to the audio acquisition module, converts the audio frames in the circular buffer queue into a wav file in a standard PCM format; A speech separation module, connected to the audio preprocessing module, loads a predictive separation model based on deep learning, separates the WAV file, and outputs independent speech tracks for each speaker; A speech recognition module, connected to the speech separation module, processes the independent speech tracks of each speaker, loads the automatic speech recognition model to perform end-to-end transcription on each speech track, and generates text with punctuation and segmentation; The result management module is connected to the speech recognition module, binds the transcribed text with the corresponding speaker number, forms a result, and outputs it in a preset format.

2. A multi-person speech separation and recognition system for real-time speech interaction according to claim 1, characterized in that: The audio acquisition module, speech separation module and speech recognition module adopt a multi-threaded asynchronous architecture to perform acquisition, separation and recognition tasks respectively.

3. The multi-person speech separation and recognition system for real-time speech interaction according to claim 1, characterized in that: The audio frame of the audio acquisition module has a set duration of 2 seconds.

4. The multi-person speech separation and recognition system for real-time speech interaction according to claim 1, characterized in that: The audio acquisition module adopts an asynchronous multi-threaded mode. Every time the acquisition of an audio frame is completed, a separate thread is immediately started to post-process the audio frame, and the separate thread and the audio acquisition main thread realize data interaction through a circular cache queue.

5. The multi-person speech separation and recognition system for real-time speech interaction according to claim 1, characterized in that: The predictive separation model in the speech separation module is the SepFormer model of SpeechBrain, which includes an encoder, a mask network and a decoder. The encoder is a one-dimensional convolutional layer, the mask network includes a layer normalization layer, a linear layer, an IntraTransformer and an InterTransformer, and the decoder is a transposed convolutional layer.

6. The multi-person speech separation and recognition system for real-time speech interaction according to claim 1, characterized in that: The automatic speech recognition model in the speech recognition module is the Paraformer model provided by FunASR. The speech recognition module processes the independent speech track of each speaker so that the sampling rate of the speech track is unified to 16kHz and the channel is set to mono.

7. The multi-person speech separation and recognition system for real-time speech interaction according to claim 1, characterized in that: The preset formats of the result management module include JSON format, TXT format or database record format. The structured results are integrated and output through independent threads, and are decoupled from the audio collection and separation threads through thread queues and task pools.

8. A multi-person speech separation and recognition method for real-time speech interaction, characterized in that: The following steps are involved: S1 Audio Capture and Caching: Use the audio capture tool to perform streaming voice capture in multi-person scenarios, divide the captured audio data into audio frames of a set length, and store them in a circular cache queue; S2 audio preprocessing and storage: converting the audio frames in the circular buffer queue into a wav file in standard PCM format, where the sampling rate of the wav file is consistent with the input requirements of the subsequent speech separation model; S3 Multi-speaker Speech Separation: The pre-processed WAV file is fed into a pre-trained speech separation model based on deep learning. The encoder of the speech separation model extracts mixed speech features, the mask network estimates the mask of each speaker, and the decoder reconstructs the independent speech tracks of each speaker. S4 automatic speech recognition: Normalizes the separated independent speech tracks of each speaker, loads the automatic speech recognition model, performs end-to-end transcription on each track, generates the corresponding text, and adds sentence segments and punctuation; S5 result integration and output: Bind the transcribed text to the corresponding speaker number to form a "speaker-text" structured result, which is output in a preset format to support subsequent semantic understanding or control tasks.

9. A method for multi-person speech separation and recognition for real-time speech interaction according to claim 8, characterized in that The audio acquisition tool is PyAudio, and the acquisition parameters are set as follows: sampling rate 16kHz, mono, and acquisition buffer size 1024 samples.

10. The method for multi-person speech separation and recognition for real-time speech interaction according to claim 8, characterized in that: The deep learning-based predictive separation model for multi-person speech separation is the SepFormer model provided by SpeechBrain. The encoder is a one-dimensional convolutional layer, which is used to convert the mixed speech signal into a feature representation; the mask network includes a layer normalization layer, a linear layer and a SepFormerBlock, and the SepFormerBlock is composed of an IntraTransformer and an InterTransformer. The IntraTransformer is used to construct a short dependency relationship of the feature block, and the InterTransformer is used to construct a long dependency relationship between the feature blocks; the decoder is a transposed convolutional layer, which is used to reconstruct an independent speech track based on the mask and encoder features.

Images