Low flux real-time voice transmission system and method

Abstract

The application discloses a low-throughput real-time voice transmission system and method. The system comprises multiple sending ends and a receiving end. Each sending end is used for collecting voice data of a current speaker and converting the voice data into incremental recognition text, and then assigning a corresponding unique identifier to the current speaker, and sending the incremental recognition text and the corresponding unique identifier of the current speaker to the receiving end. The receiving end is used for storing the received incremental recognition text according to the unique identifier corresponding to the speaker, and then synthesizing the incremental recognition text into audio. The application effectively reduces the data transmission amount, saves the communication bandwidth, removes the voiceprint features in the original voice, protects the privacy of the speaker, reduces the end-to-end transmission delay, and meets the real-time communication demand.

Description

Technical Field

[0001] This invention belongs to the field of speech signal processing, specifically relating to a low-throughput real-time speech transmission system and method. Background Technology

[0002] In special communication scenarios such as confidential meetings, the reliability, real-time performance, and privacy of voice communication are crucial to ensuring the effective transmission of information. Traditional voice transmission schemes mainly employ direct transmission of compressed audio, aiming to accurately reproduce the original speech waveform. However, this approach faces two major problems in practical applications: first, the large volume of audio data, even after compression encoding, easily leads to transmission congestion and latency accumulation in low-bandwidth channels; second, the speaker's voiceprint features contained in the original speech signal are easily intercepted and analyzed during wireless transmission, posing a risk of identity exposure. With the rapid development of deep learning technology, speech recognition and speech synthesis technologies have made breakthrough progress. End-to-end speech recognition models have significantly improved recognition accuracy, enabling real-time conversion of speech into text. Furthermore, the widespread application of neural network-based vocoders and acoustic models has made synthesized speech appear closer to natural human speech. These technological advancements have provided more research directions for the field of voice communication.

[0003] However, existing speech recognition and synthesis-based communication solutions still have the following limitations in practical applications: First, most speech recognition systems adopt a whole-sentence recognition mode, which requires waiting for the speaker to complete a complete sentence before outputting the recognition result. This processing method introduces a large system delay in end-to-end transmission, making it difficult to meet the requirements of real-time voice communication. Second, existing solutions are mainly designed for single-speaker scenarios and lack an effective mechanism to distinguish the identities of different speakers when multiple voices are transmitted concurrently. The receiving end cannot accurately identify the source of information, which can easily lead to information confusion. In addition, high-performance speech recognition and synthesis models have high requirements for computing resources. When deployed on resource-constrained edge devices, a trade-off must often be made between processing speed and recognition and synthesis quality, making it difficult to simultaneously guarantee real-time performance and accuracy. Summary of the Invention

[0004] To address the problems of high bandwidth consumption and easy identification of speaker voice features leading to privacy leaks in traditional voice transmission, the present invention provides a low-throughput real-time voice transmission system and method.

[0005] The technical solution adopted in this invention is: A low-throughput real-time voice transmission system is provided for secure communication of multiple voice data streams. The system includes multiple transmitters and receivers. Each transmitter is used to collect the voice data of the current speaker and convert it into incremental recognition text. Then, a unique identifier is assigned to the current speaker. The incremental recognition text and the corresponding unique identifier of the current speaker are sent to the receiver. The receiver is used to classify and store the received incremental recognition text according to the unique identifier of the speaker, and then synthesize the incremental recognition text into audio.

[0006] Each of the transmitting ends includes a voice acquisition module and a voice recognition module. The voice acquisition module is used to acquire the voice data of the current speaker and assign a corresponding unique identifier to the current speaker. Then, the voice data and identifier of the current speaker are sent to the voice recognition module. The speech recognition module is used to extract incremental recognition text from the current speaker's speech data using a streaming speech recognition method, and then encapsulate the incremental recognition text and the corresponding identifier before sending it to the receiving end.

[0007] The receiving end includes a speech synthesis module and a graphical user interface module. The speech synthesis module is used to create an independent text queue for each speaker, classify the received incremental recognition text according to the corresponding identifier and store it into the corresponding text queue, and then select the corresponding text queue from all text queues according to the target instruction, and take out the incremental recognition text for speech synthesis to generate audio data. The graphical user module is used to acquire target commands and play the audio data output by the speech synthesis module.

[0008] A low-throughput real-time voice transmission method includes the following steps: S1. Each transmitting end acquires the voice data of the current speaker and assigns a unique identifier to the current speaker. S2. Preprocess the speech data of the current speaker, and then use streaming speech recognition to convert the preprocessed speech data into incremental recognition text, and then send the incremental recognition text of the current speaker and the corresponding identifier to the receiving end. S3. The receiving end receives the incremental recognition text and corresponding identifier sent by each sending end, and then creates an independent text queue for each speaker, and stores the received incremental recognition text into the corresponding text queue according to the corresponding identifier. S4. Obtain the target instruction, select the corresponding text queue from all text queues according to the target instruction, and extract the incremental recognition text for speech synthesis to generate audio data.

[0009] Step S1 specifically involves: collecting ambient sound, then using a speech activity detection algorithm to detect the ambient sound, extracting the audio data corresponding to the valid speech as the speech data of the current speaker, and assigning a unique identifier to the current speaker.

[0010] In step S2, the preprocessing includes waveform conversion, amplitude normalization, framing, and windowing.

[0011] In step S2, the preprocessed speech data is converted into incremental recognition text using streaming speech recognition, specifically as follows: 1) Perform a fast Fourier transform on the preprocessed speech data to obtain the spectrum, and then filter the spectrum through a Mel filter bank to obtain the energy value of each frequency channel. 2) Take the logarithm of the energy value of each frequency channel to obtain the FBank feature of the current frame, and then construct a time-frequency two-dimensional feature matrix based on the FBank features of all frames; 3) Normalize and length-align the time-frequency two-dimensional feature matrix sequentially to obtain aligned feature tensors; 4) Input the aligned feature tensor into the pre-trained streaming speech recognition model to obtain incremental recognized text.

[0012] Step S4 specifically involves: S4.1 Obtain the target instruction, select the corresponding text queue from all text queues according to the target instruction, and extract the corresponding incremental recognition text as the text to be synthesized; S4.2. The text to be synthesized is sequentially cleaned, segmented, and converted into phonemes to obtain a phoneme ID sequence. S4.3 Input the phoneme ID sequence into the acoustic model for processing to obtain Mel spectrum features; S4.4 Input the Mel spectrum features into the vocoder to generate a time-domain waveform sequence, and then truncate the time-domain waveform sequence to obtain the audio data corresponding to the text to be synthesized.

[0013] The beneficial effects of this invention are: This invention effectively reduces data transmission volume and saves communication bandwidth by converting speech into text for transmission and then synthesizing the text back into speech at the receiving end. At the same time, it removes the voiceprint features from the original speech, thus protecting the speaker's privacy.

[0014] Furthermore, by employing streaming speech recognition and incremental text output, end-to-end transmission latency is reduced, meeting the requirements for real-time communication. The receiving end establishes independent text queues for each speaker, enabling the categorized storage and on-demand synthesis of multiple audio streams, supporting accurate information reconstruction in scenarios with multiple speakers simultaneously. Attached Figure Description

[0015] Figure 1 This is a block diagram of the low-throughput real-time voice transmission system in this embodiment.

[0016] Figure 2 This is a schematic diagram of the streaming speech recognition and incremental recognition text output in this embodiment.

[0017] Figure 3 This is a flowchart of the speech recognition module processing in this embodiment.

[0018] Figure 4 This is a flowchart of the speech synthesis module processing in this embodiment. Detailed Implementation

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0021] like Figure 1 As shown, this embodiment relates to a low-throughput real-time voice transmission system for secure communication of multiple voice data streams. It includes multiple transmitters and receivers. Each transmitter is used to collect the voice data of the current speaker and convert it into incremental recognition text. Then, it assigns a corresponding unique identifier to the current speaker and sends the incremental recognition text and the corresponding unique identifier to the receiver. The receiver is used to classify and store the received incremental recognition text according to the unique identifier corresponding to the speaker, and then synthesizes the incremental recognition text into audio.

[0022] In this embodiment, each sender has only one speaker and is bound to a unique identifier, with a value ranging from 1 to 255. Figure 1 The “N×” indicates that the system supports at least 5 speakers to transmit low-throughput speech simultaneously, with each speaker’s speech stream being acquired and recognized independently.

[0023] Optionally, when multiple speakers use each transmitter, voiceprint detection is performed on the voice data of the transmitter to bind a unique identifier to the corresponding speaker. In this case, the transmitter and the identifier are not bound together.

[0024] Each transmitter includes a voice acquisition module and a voice recognition module. The voice acquisition module is used to acquire the voice data of the current speaker and assign a unique identifier to the current speaker. Then, the voice data and identifier of the current speaker are sent to the voice recognition module. The speech recognition module is used to extract incremental recognition text from the current speaker's speech data using streaming speech recognition methods, and then encapsulate the incremental recognition text and the corresponding identifier before sending it to the speech synthesis module at the receiving end.

[0025] The speech recognition module establishes and maintains an independent data structure for each speaker, which includes an audio buffer and a historical recognized text storage area.

[0026] The receiving end includes a speech synthesis module and a graphical user module. The speech synthesis module is used to create an independent text queue for each speaker, classify the received incremental recognition text according to the corresponding identifier and store it into the corresponding text queue, realizing the classification and storage of text from multiple speakers. Then, according to the target instruction, the corresponding text queue is selected from all text queues, and the incremental recognition text is retrieved for speech synthesis to generate audio data. The graphical user module is used to acquire target commands and play the audio data output by the speech synthesis module. Specifically, the graphical user module acquires and displays the real-time operating status of the speech synthesis module and sends synthesis commands from the target speaker to the speech synthesis module. In this embodiment, the voice acquisition module and the graphical user module are deployed on the PC, and the voice recognition module and the voice synthesis module are deployed on the Huawei 310B. Communication connections between the voice acquisition module and the voice recognition module, between the graphics user module and the voice synthesis module, and between the voice recognition module and the voice synthesis module are all established via TCP sockets.

[0027] The system operating parameters include the following settings: the voice acquisition module is connected to a USB microphone array and configured with audio parameters of 16000 Hz sampling rate, 16-bit quantization precision, mono, 30ms frame length, and voice activity detection algorithm mode set to 2; the audio output sampling rate of the speech synthesis module is 22050Hz, and the amplitude factor used in both the speech recognition and synthesis modules is 32768.0. The graphical user module is developed based on Python Tkinter, includes a status monitoring panel and synthesis control buttons, and has a built-in audio player.

[0028] This embodiment uses Huawei 310B as the core AI inference hardware and PC as the lightweight processing and human-computer interaction hardware. The process of acquiring multiple speakers' voices is characterized by identifier binding. The speech recognition process adopts incremental speech recognition and incremental text output. The receiving end realizes the classification and storage of texts from multiple speakers through an independent text queue, and realizes the speech synthesis of a specified speaker by combining the directional scheduling of the graphical user module.

[0029] In this embodiment, the speech synthesis module starts the status query server, and the graphical user module sends a status query request to the status query server at regular intervals to update the status monitoring table in the interface and display the running status of each speaker in real time. The graphical user module listens to user actions. When a speaker's synthesis button is clicked, a synthesis instruction is constructed and sent to the speech synthesis module.

[0030] The speech synthesis module receives the synthesis instruction, parses out the speaker identifier, and puts it into the synthesis request queue for processing by the speech synthesis module.

[0031] The low-throughput real-time voice transmission method includes the following steps: S1. The voice acquisition module in each transmitting end acquires the voice data of the current speaker, assigns a unique identifier to the current speaker, and sends the voice data of the current speaker to the voice recognition module after packet processing and corresponding identifier. S2. The speech recognition module preprocesses the speech data of the current speaker, and then uses streaming speech recognition to convert the preprocessed speech data into incremental recognition text, and then sends the incremental recognition text of the current speaker and the corresponding identifier to the receiving end. S3. The receiving end receives the incremental recognition text and corresponding identifier sent by each sending end in the receiving order, and then creates an independent text queue for each speaker, and stores the received incremental recognition text into the corresponding text queue according to the corresponding identifier. S4. Obtain the target instruction, select the corresponding text queue from all text queues according to the target instruction, extract the incremental recognition text, perform speech synthesis to generate audio data and play it.

[0032] S5, the graphical user module, plays audio data.

[0033] Step S1 specifically involves: collecting ambient sound through a microphone, reading it as a frame of audio data every 30ms, then using the Voice Activity Detection (VAD) algorithm to detect the ambient sound, extracting the audio data corresponding to the valid speech as the speech data of the current speaker, and assigning a unique identifier to the current speaker.

[0034] In this embodiment, speech is determined to begin when three consecutive frames (90ms) of speech are detected; speech is determined to end when 30 consecutive frames (900ms) of non-speech are detected. After speech begins, audio frames are buffered. When a fixed duration of audio data is accumulated, or when the end of speech is detected, an audio data packet is immediately constructed. The audio data packet format is as follows: the first four bytes are the audio data length, the fifth byte is the end marker, the sixth byte is the speaker identifier, and the last byte is the original PCM audio data. After the data packet is encapsulated, it is sent to the speech recognition module.

[0035] Specifically, the voice acquisition module assigns a unique identifier to each speaker, collects ambient sound through a microphone, and uses a voice activity detection algorithm to determine whether it contains valid speech from the speaker. The voice activity detection algorithm is a technology used to detect whether there is human voice in an audio signal. When valid speech is detected, the audio data length, end marker, speaker identifier, and fixed duration of audio data are encapsulated into an audio data packet and sent to the voice recognition module. Fixed-duration audio data is generated by dividing a continuous voice data stream into multiple short data streams through a preset audio data time slice; the end flag is used to identify whether the audio data packet is the last data packet corresponding to a complete sentence in the speaker's current voice stream.

[0036] In step S2, preprocessing includes waveform conversion, amplitude normalization, framing, and windowing.

[0037] The preprocessing specifically includes: 1) Waveform conversion: converting the PCM format integer audio data into floating-point waveform data and performing amplitude normalization to obtain a floating-point waveform sequence; In this embodiment, each sampling point of the 16-bit signed integer PCM data with values ​​ranging from [-32768, 32767] is divided by the amplitude factor 32768.0 to obtain a floating-point waveform sequence with values ​​ranging from [-1, 1].

[0038] 2) Frame segmentation and windowing: The floating-point waveform sequence is segmented into frames according to a preset frame length and frame shift to obtain several speech frames. Each frame is multiplied by a window function to obtain a windowed speech frame sequence.

[0039] In this embodiment, the frame length is 25ms (400 sampling points), and the frame shift is 10ms (160 sampling points) to obtain several speech frames. Each frame is multiplied by a Hamming window function to obtain a windowed speech frame sequence, thereby reducing spectral leakage.

[0040] like Figure 2 and Figure 3 As shown, in step S2, the preprocessed speech data is converted into incremental recognition text using streaming speech recognition, specifically as follows: 1) Perform a fast Fourier transform on the preprocessed speech data to obtain the spectrum, and then filter the spectrum through a Mel filter bank to obtain the energy value of each frequency channel. This embodiment uses 80 Mel filters, covering a frequency range of 0-8000Hz.

[0041] 2) Take the logarithm of the energy value of each frequency channel to obtain the FBank feature of each frequency channel in the current frame, and then construct a time-frequency two-dimensional feature matrix with the shape (T, 80) based on the FBank features of all frames, where T is the total number of frames; 3) Normalize and length-align the time-frequency two-dimensional feature matrix sequentially to obtain aligned feature tensors; The feature matrix is ​​normalized, and then the normalized feature matrix is ​​length aligned. The length alignment process is as follows: if the number of frames in the feature matrix is ​​less than the maximum number of input frames allowed by the speech recognition model, zeros are padded to the end of the time dimension until the maximum number of frames is reached; if the number of frames is greater than the maximum number of frames, the first maximum number of frames is truncated to obtain the aligned feature tensor.

[0042] 4) Input the aligned feature tensor into the pre-trained streaming speech recognition model to obtain incremental recognized text.

[0043] Specifically, the aligned feature tensor is input into the pre-trained speech recognition model, which includes a Conformer encoding module and a CTC decoding module. The probability distribution of the lexical unit at each time step is obtained through forward computation. Then, the probability distribution is decoded using CTC decoding to obtain the lexical unit index sequence. This sequence is then mapped to Chinese text through a pre-set vocabulary and used as the recognition result of the current audio segment.

[0044] The streaming speech recognition method is as follows: when the speech recognition module receives the first audio data packet from the speaker, it immediately performs speech recognition on the audio data in the data packet. After the recognition is completed, the audio data is stored in the speaker's audio buffer. Subsequent new data is appended to the speaker's audio buffer. Then, all the audio data in the buffer is concatenated in order, and speech recognition is performed on the concatenated complete audio data stream. For example: Speaker 1's first data packet contains the audio of today, and the recognition result is today; the second data packet contains the audio of air, and the recognition result after splicing is today's air; the third data packet contains the audio of what, and the recognition result after splicing is today's air quality. As the audio accumulates, the recognition result gradually becomes complete.

[0045] The system compares the latest recognized text with the speaker's historical recognized text, extracts new text content based on the difference in text length, achieves incremental text output, and updates the historical recognized text.

[0046] like Figure 4 As shown, step S4 specifically involves: S4.1 Obtain the target instruction, select the corresponding text queue from all text queues according to the target instruction, and extract the corresponding incremental recognition text as the text to be synthesized; S4.2. The text to be synthesized is sequentially cleaned, segmented, and converted into phonemes to obtain a phoneme ID sequence. Text cleaning includes removing special characters and redundant whitespace from the text. Word segmentation: The cleaned continuous text is divided into word sequences.

[0047] Phoneme conversion: Each word in the word sequence is converted into its corresponding phoneme identifier through a pre-defined phoneme mapping table to obtain a phoneme ID sequence.

[0048] The phoneme mapping table contains 156 Chinese phonemes, including initials, finals, and special phonemes; for example, today is mapped as the phoneme ID sequence [jin1, tian1]. The complete phoneme ID sequence is obtained by concatenating the phoneme IDs of all words. S4.3 Input the phoneme ID sequence into the acoustic model for processing to obtain Mel spectrum features; The acoustic model includes an encoding module, a duration and pitch energy prediction module, a decoding module, and a post-processing module; The acoustic model processing is specifically as follows: 1) Encoding module: The phoneme ID sequence is mapped to a high-dimensional hidden feature sequence through encoding, which is used to extract the semantic features of the text; 2) Duration and Pitch Energy Prediction Module: Then, the number of speech frames corresponding to each phoneme is estimated and the feature sequence is expanded according to the prediction results. At the same time, the pitch value and energy value of each frame are estimated and the feature sequence is adjusted accordingly. 3) Decoding module: The adjusted high-dimensional hidden feature sequence is input into the decoding module to generate the corresponding Mel spectrum features frame by frame; 4) Post-processing module: Refines the generated Mel spectrum features to improve spectrum quality.

[0049] S4.4 Input the Mel spectrum features into the vocoder to generate a time-domain waveform sequence, and then truncate the time-domain waveform sequence to obtain the audio data corresponding to the text to be synthesized.

[0050] The vocoder converts spectral features into a time-domain audio waveform sequence step by step through multiple upsampling layers; the vocoder is a HiFi-GAN vocoder, used to restore Mel spectral features to time-domain speech waveforms with high quality.

[0051] Subsequently, the time-domain waveform sequence is post-processed. Specifically, the floating-point waveform data is multiplied by a preset amplitude factor of 32768.0 and converted to a 16-bit integer format. Then, the actual effective audio length is calculated based on the number of frames in the Mel spectrum and the preset frame shift, which is 256 points. Waveform data of this length is then extracted to obtain the final synthesized speech audio data.

[0052] The target instruction is used to determine the target speaker that the user wishes to listen to. The user selects the target speaker through the graphical user interface (GUI). After selecting the target speaker, the GUI sends the corresponding target instruction to the speech synthesis module. Upon receiving the target instruction, the speech synthesis module determines the target speaker based on the instruction, retrieves the incremental recognized text from the text queue corresponding to that target speaker, performs speech synthesis, and generates audio data for the user to listen to.

[0053] In summary, this invention provides a low-throughput real-time voice transmission system based on the Huawei 310B. First, multiple speaker voices are acquired, each speaker is assigned a unique identifier, and then the data is sent to the speech recognition module. The speech recognition module uses a streaming recognition method to output incremental speech-recognized text in real time, offering advantages such as fast recognition speed and high real-time performance. The receiving end's speech synthesis module categorizes and stores the received text according to the identifiers. The graphical user module displays the running status in real time and sends target speaker synthesis instructions to the speech synthesis module. The speech synthesis module extracts the text according to the instructions, performs speech synthesis, generates audio data, and sends it back to the graphical user module for playback. This invention effectively reduces communication resource overhead by replacing raw audio transmission with text transmission, while simultaneously protecting speaker privacy.

[0054] The above detailed embodiments illustrate the technical solution and beneficial effects of the present invention. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A low-throughput real-time voice transmission system for secure communication of multiple voice data streams, characterized in that: The system includes multiple transmitters and receivers. Each transmitter is used to collect the speech data of the current speaker and convert it into incremental recognition text. Then, it assigns a corresponding unique identifier to the current speaker and sends the incremental recognition text and the corresponding unique identifier to the receiver. The receiver is used to classify and store the received incremental recognition text according to the unique identifier of the speaker, and then synthesize the incremental recognition text into audio.

2. The low-throughput real-time voice transmission system according to claim 1, characterized in that: Each of the transmitting ends includes a voice acquisition module and a voice recognition module. The voice acquisition module is used to acquire the voice data of the current speaker and assign a corresponding unique identifier to the current speaker. Then, the voice data and identifier of the current speaker are sent to the voice recognition module. The speech recognition module is used to extract incremental recognition text from the current speaker's speech data using a streaming speech recognition method, and then encapsulate the incremental recognition text and the corresponding identifier before sending it to the receiving end.

3. The low-throughput real-time voice transmission system according to claim 1, characterized in that: The receiving end includes a speech synthesis module and a graphical user interface module. The speech synthesis module is used to create an independent text queue for each speaker, classify the received incremental recognition text according to the corresponding identifier and store it into the corresponding text queue, and then select the corresponding text queue from all text queues according to the target instruction, and take out the incremental recognition text for speech synthesis to generate audio data. The graphical user module is used to acquire target commands and play the audio data output by the speech synthesis module.

4. A low-throughput real-time voice transmission method using the system described in any one of claims 1-3, characterized in that, The method includes the following steps: S1. Each transmitting end acquires the voice data of the current speaker and assigns a unique identifier to the current speaker. S2. Preprocess the speech data of the current speaker, and then use streaming speech recognition to convert the preprocessed speech data into incremental recognition text, and then send the incremental recognition text of the current speaker and the corresponding identifier to the receiving end. S3. The receiving end receives the incremental recognition text and corresponding identifier sent by each sending end, and then creates an independent text queue for each speaker, and stores the received incremental recognition text into the corresponding text queue according to the corresponding identifier. S4. Obtain the target instruction, select the corresponding text queue from all text queues according to the target instruction, and extract the incremental recognition text for speech synthesis to generate audio data.

5. A low-throughput real-time voice transmission method according to claim 4, characterized in that: Step S1 specifically involves: collecting ambient sound, then using a speech activity detection algorithm to detect the ambient sound, extracting the audio data corresponding to the valid speech as the speech data of the current speaker, and assigning a unique identifier to the current speaker.

6. A low-throughput real-time voice transmission method according to claim 4, characterized in that: In step S2, the preprocessing includes waveform conversion, amplitude normalization, framing, and windowing.

7. A low-throughput real-time voice transmission method according to claim 4, characterized in that: In step S2, the preprocessed speech data is converted into incremental recognition text using streaming speech recognition, specifically as follows: 1) Perform a fast Fourier transform on the preprocessed speech data to obtain the spectrum, and then filter the spectrum through a Mel filter bank to obtain the energy value of each frequency channel. 2) Take the logarithm of the energy value of each frequency channel to obtain the FBank feature of the current frame, and then construct a time-frequency two-dimensional feature matrix based on the FBank features of all frames; 3) Normalize and length-align the time-frequency two-dimensional feature matrix sequentially to obtain aligned feature tensors; 4) Input the aligned feature tensor into the pre-trained streaming speech recognition model to obtain incremental recognized text.

8. A low-throughput real-time voice transmission method according to claim 4, characterized in that: Step S4 specifically involves: S4.1 Obtain the target instruction, select the corresponding text queue from all text queues according to the target instruction, and extract the corresponding incremental recognition text as the text to be synthesized; S4.

2. The text to be synthesized is sequentially cleaned, segmented, and converted into phonemes to obtain a phoneme ID sequence. S4.3 Input the phoneme ID sequence into the acoustic model for processing to obtain Mel spectrum features; S4.4 Input the Mel spectrum features into the vocoder to generate a time-domain waveform sequence, and then truncate the time-domain waveform sequence to obtain the audio data corresponding to the text to be synthesized.

Images