A real-time voice conversion method and device in a virtual live broadcast environment

Abstract

The application discloses a real-time voice conversion method and device in a virtual live environment, which highly retains the identity-independent voice features such as tone and intonation of a role player, and provides a service that can be smoothly run on a personal computer without registration and retraining for a user. In addition, the method provides more than 20 virtual timbres for selection and switching, provides a more diversified and personalized virtual role-playing effect for a VTuber, and better serves a virtual live scene. The method can realize low-delay and high-quality real-time voice generation, can realize real-time conversion and control the real-time delay to be only 70.8 ms, is equivalent to the most advanced multi-target voice conversion method based on Diffusion in voice quality and intelligibility, and leads the current mainstream method in voice similarity.

Description

Technical Field

[0001] This invention belongs to the field of speech information processing technology, and relates to a high-quality speech conversion method, and more particularly to a real-time speech conversion method and apparatus in a virtual live streaming environment. Background Technology

[0002] With the increasing popularity of social media platforms such as Bilibili, Douyin, and YouTube, virtual YouTubers (VTubers) are rapidly developing in the online world, attracting more and more attention and popularity. Generally, VTubers require an actor to portray the character and provide voice-over narration. However, this traditional method has many limitations and risks, especially in terms of business models. First, the virtual character's voice is highly dependent on the actor; therefore, if the actor is replaced due to reassignment or illness, the character's popularity may decline, or even the project may be terminated. Furthermore, the choice of character is also limited by the actor's voice; if the actor's voice does not match the character's characteristics, the performance may be subpar. To solve these problems, a voice conversion technology is needed that can create a fixed virtual timbre for the virtual character and allow the actor to convert their own voice into the target virtual voice in real time while portraying the character, thus decoupling the virtual character's voice from the actor's.

[0003] Speech conversion technology is a technique that converts the voice of an original speaker into the voice of a target speaker. It can alter important elements of a person's voice, such as pitch, timbre, and rhythm. This technology has a long history, with numerous researches and inventions. Based on the powerful nonlinear modeling capabilities of neural networks, mainstream speech conversion technologies can achieve high-quality conversion results. However, while traditional speech conversion technologies can successfully convert user speech into target virtual voices, most struggle to meet real-time requirements. These traditional methods typically require pre-recording of complete sentences, leading to significant and unpredictable delays. In live streaming, the conversion process and the user's speech are synchronized, with only a tiny, fixed delay between the output and input speech. Even when applying these traditional methods to segment the input speech stream for real-time processing, adjacent segments are independent and lack effective long-term contextual association, resulting in a significant deterioration in conversion quality. Therefore, most existing speech conversion technologies still fall short of the requirements for real-time virtual voice generation. Summary of the Invention

[0004] The purpose of this invention is to provide a real-time voice conversion method in a virtual live streaming environment, solving the problem that existing technologies cannot perform real-time voice conversion or produce poor conversion results in virtual live streaming scenarios. This method highly preserves the voice features of the performer, such as tone and intonation, which are independent of their identity, providing users with a service that runs smoothly on their personal computers without registration or retraining. Furthermore, this method offers more than 20 virtual voices to choose from and switch between, providing VTubers with more diverse and personalized virtual role-playing effects, better serving the virtual live streaming scenario.

[0005] In a first aspect, embodiments of the present invention provide a real-time speech conversion method in a virtual live streaming environment, characterized in that the method includes:

[0006] System parameters are configured through the front-end interactive interface, and the input and output voice waveforms and spectrum are displayed in real time, with real-time feedback on the running status.

[0007] The user's voice stream is continuously recorded through a microphone as the raw voice input, and the voice stream is segmented into segments with a fixed length of 50-200ms without overlap to obtain segmented voice.

[0008] Standard log-Mel spectrum is extracted from segmented speech to obtain segmented input speech features;

[0009] Based on the pre-set virtual voice target, the segmented input voice feature timbre feature vectors are simultaneously input into a pre-trained feature transformation model for processing, generating segmented output voice features that conform to the target virtual timbre.

[0010] The segmented output speech features are input into the speech reconstruction model to reconstruct speech, generating segmented output speech of the same length as the input speech segment that conforms to the target virtual timbre.

[0011] The segmented output speech is directly spliced ​​together in chronological order to obtain the real-time generated virtual speech.

[0012] In conjunction with the first possible implementation of the first aspect, in the second possible implementation of the first aspect, the extraction of the standard logarithm Mel spectrum processing employs an inter-segment buffering mechanism. Specifically:

[0013] Remove the padding mechanism when extracting the time-frequency plot during standard log-Mel spectrum analysis;

[0014] When extracting the standard log-Mel spectrum of each speech segment, the last (frame length - step length) length of the speech is buffered. Before extracting the standard log-Mel spectrum of each speech segment, the buffer from the previous processing step is concatenated to the beginning of the input segment.

[0015] In the third implementation of the first aspect, in conjunction with the first possible implementation of the first aspect, the feature transformation model includes two application stages: model inference and model training. The model inference stage includes a timbre extraction network and a generator network. The timbre extraction network extracts the timbre code of the target virtual timbre, and the generator network maps the input features to the target timbre domain represented by the timbre code. In the model training stage, in addition to the timbre extraction network and the generator network, a discriminator network is also included for adversarial training together with the generator network.

[0016] In conjunction with the third implementation of the first aspect, the fourth implementation of the first aspect employs a purely causal system structure and implements streaming processing through an inter-segment caching mechanism, including:

[0017] A mapping network is constructed using causal convolutional layers and unidirectional LSTM layers, so that the network can only use information at the current time point and information before the current time point during the training and inference phases, while shielding unknowable future information.

[0018] Cancel the temporal padding mechanism for all causal convolutional layers;

[0019] A buffer is set before all causal convolutional layers. During each segment inference, the last (kernel size - 1) time units of data of the input tensor are stored in the buffer, and the intermediate results stored in the buffer in the previous segment inference are concatenated to the beginning of the input.

[0020] Set up an LSTM cache to cache the output of the LSTM layer (including the output of the last time step and the state of the LSTM cell in the last time step) during each segment inference, so that it can be passed as a parameter to the LSTM layer during the next segment inference.

[0021] The EWMA algorithm is used to dynamically estimate the statistical parameters of the IN layer to adapt to long-term speaking states.

[0022] In conjunction with the first implementation of the first aspect, the fifth implementation of the first aspect modifies the MB-MelGAN network into a streaming model, adopts a pure causal structure, and implements streaming processing through an inter-segment caching mechanism, including:

[0023] Replace ordinary convolutional layers with causal convolutional layers, and eliminate the temporal padding mechanism for all causal convolutional layers;

[0024] A buffer is set before all causal convolutional layers. During each segment inference, the last (kernel size - 1) time units of data of the input tensor are stored in the buffer, and the intermediate results stored in the buffer in the previous segment inference are concatenated to the beginning of the input.

[0025] In conjunction with the first aspect, in the sixth implementation of the first aspect, the segmented speech is processed by a convolutional neural network based on adversarial training to perform speech denoising; the segmented output speech is processed by digital watermarking technology to perform digital watermarking anti-counterfeiting processing on the output time-domain waveform.

[0026] Secondly, embodiments of the present invention also provide a virtual voice real-time generation device, the device comprising:

[0027] The front-end interaction module allows for system parameter configuration via a front-end interface, real-time display of input and output voice waveforms and spectrum, and real-time feedback of operating status.

[0028] The input processing module continuously records the user's voice stream using a microphone as the raw voice input, and segments the voice stream into non-overlapping segments with fixed short lengths of 50-200ms to obtain the segmented voice.

[0029] The acoustic feature extraction module extracts the standard log-Mel spectrum of the segmented speech to obtain the segmented input speech features;

[0030] The acoustic feature conversion module, based on the pre-set virtual speech target, simultaneously inputs the segmented input speech feature timbre feature vectors into the pre-trained feature conversion model for processing, generating segmented output speech features that conform to the target virtual timbre;

[0031] The speech reconstruction module inputs the segmented output speech features into the speech reconstruction model to reconstruct the speech and generate segmented output speech that is the same length as the input speech segment and conforms to the target virtual timbre.

[0032] The output processing module directly splices the segmented output speech in chronological order to obtain the real-time generated virtual speech.

[0033] In conjunction with the first implementation of the second aspect, in the second implementation of the second aspect, the acoustic feature conversion module uses a pre-trained convolutional recurrent neural network (CRNN) based on causal convolution for speech feature conversion, and a pre-trained timbre extraction network for virtual timbre extraction.

[0034] In conjunction with the second implementation of the second aspect, the third implementation of the second aspect describes the following structure of the convolutional recurrent neural network based on causal convolution: a CNN encoder, a convolutional LSTM, and a CNN decoder. The CNN encoder consists of a 6-layer residual block network using instance-norm layers. The CNN decoder consists of a 6-layer adaptive residual block network using instance-norm layers plus a 1×1 convolutional layer. Each adaptive residual block in the CNN decoder receives a timbre vector output from the timbre extraction network to guide virtual speech synthesis.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0036] This invention provides a real-time speech conversion method for virtual live streaming environments, achieving high-quality multi-objective virtual speech generation. Objective and subjective experimental evaluation results show that this method can achieve low-latency and high-quality real-time speech generation, enabling real-time conversion with a controlled real-time latency of only 70.8ms. In terms of speech quality and intelligibility, it is comparable to state-of-the-art Diffusion-based multi-objective speech conversion methods, and it surpasses current mainstream methods in terms of speech similarity.

[0037] This invention provides high-quality speech conversion results. The speech conversion module of this invention is trained using an advanced Generative Adversarial Network (StarGAN) architecture and consistency loss functions such as the cycle consistency loss function and the fundamental frequency consistency loss function, providing high-quality conversion results, especially outperforming mainstream speech conversion algorithms in terms of timbre similarity.

[0038] This invention provides a many-to-many speech conversion function. Based on the StarGAN architecture and timbre extraction module, this invention can achieve many-to-many speech conversion on a single model, supporting more than 20 preset timbres. Based on large-scale training data, the model can adapt to speakers from any source, allowing users to use it immediately without the need for time-consuming registration or retraining on the user's end.

[0039] This invention effectively resists interference from the environment and equipment. The speech conversion module of this invention employs data augmentation technology based on real physical interference during training. Experimental results show that this invention maintains consistent conversion performance under different recording devices and varying environmental noise levels.

[0040] This invention provides low-latency real-time speech conversion. The acoustic feature extraction module, acoustic feature conversion module, and speech reconstruction module all employ a pure causal structure and caching technology. Based on segmented inference, they provide the model with seamless long-term memory and output, effectively eliminating splicing artifacts and quality degradation caused by segmentation in traditional speech conversion methods. This achieves conversion results consistent with non-real-time speech conversion while maintaining low latency (70.8ms). This invention also provides a real-time speech conversion system in a virtual live streaming environment with a user-friendly interface. Users can configure device parameters, conversion parameters, and query input / output status and system operating status through this interface. This interface connects directly to downstream applications via a virtual sound card, allowing users to achieve real-time speech conversion in virtual live streaming scenarios through simple configuration. Attached Figure Description

[0041] Figure 1 This is a system flowchart of a real-time speech conversion system in a virtual live streaming environment according to the present invention.

[0042] Figure 2 This is a schematic diagram comparing streaming speech conversion and traditional non-streaming speech conversion in a real-time speech conversion method under a virtual live streaming environment according to the present invention.

[0043] Figure 3 This is a schematic diagram comparing the streaming model inference and the traditional non-streaming model inference of a real-time speech conversion method in a virtual live streaming environment according to the present invention.

[0044] Figure 4 This is a flowchart illustrating a real-time speech conversion method in a virtual live streaming environment according to the present invention.

[0045] Figure 5 This is a schematic diagram of a streaming feature extraction method for a real-time speech conversion method in a virtual live streaming environment according to the present invention.

[0046] Figure 6 This is a schematic diagram of the feature mapping network structure of a real-time speech conversion method in a virtual live streaming environment according to the present invention.

[0047] Figure 7 This is a schematic diagram comparing causal convolution and conventional convolution in a real-time speech conversion method for a virtual live streaming environment according to the present invention.

[0048] Figure 8 This invention provides a front-end interface for a real-time voice conversion system in a virtual live streaming environment.

[0049] Figure 9 The results are from a subjective evaluation experiment of the present invention and three mainstream multi-target speech conversion methods.

[0050] Figure 10 This invention addresses the time overhead of processing speech signals of different lengths on different computing platforms. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that these embodiments are only intended to assist those skilled in the art in understanding the principles and knowledge of the present invention, and are not intended to limit the scope of the present invention, nor should they be considered as limiting the application scenarios of the present invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various modifications or alterations to the present invention, but any variations, changes, and transformations made to the embodiments based on the principles and spirit of the present invention also fall within the scope defined by the appended claims. And it is obvious that this specification only uses preferred embodiments as examples, and does not need to exhaustively describe all embodiments.

[0052] The purpose of this invention is to provide a real-time speech conversion method based on causal convolution, which solves the problem that traditional speech conversion methods require receiving complete speech input and cannot process the input speech in real time. This method achieves high-quality real-time conversion between user speech and virtual voice, while also eliminating the need for cumbersome steps such as user registration or retraining, and allowing the selection of multiple virtual voices. The invention will be further described below with reference to the accompanying drawings.

[0053] Figure 2 This paper compares traditional non-streaming speech conversion with streaming speech conversion to better illustrate the advantages of the proposed streaming speech conversion method over traditional non-streaming methods. In non-streaming speech conversion, the entire speech input is processed in a single inference, enabling the model to extract contextual information from the complete input field. However, a drawback of this method is that it requires waiting for the entire speech stream recording to complete before conversion, which leads to significant and varying latency.

[0054] To achieve streaming speech conversion, a low-latency, continuous, and parallel approach is used to convert the input speech stream. This method reduces the conversion latency to the sum of the segment size and the time required to process a single segment, representing a significant improvement over processing the entire speech stream at once.

[0055] However, directly feeding the input stream segment by segment into the non-streaming speech conversion model will lead to performance degradation for the following reasons: First, small segments cannot provide complete contextual input information, especially in real-time conditions where future information is unavailable; second, due to different normalization parameters and contexts, the independent segment processing will lead to inconsistent inference results and cause duplicate calculations; third, directly splicing waveforms generated by independent inference will result in unnatural joints, pops, or sharp noise.

[0056] To address the aforementioned issues, this invention provides a fundamental design approach for implementing streaming model inference, namely, using a causal system and historical caching to establish long-term connections between segment processes. This design approach ensures equivalent conversion performance to whole-segment speech processing in traditional speech conversion, while significantly reducing latency caused by short inputs.

[0057] Figure 3 A schematic diagram of this streaming reasoning approach is shown, and compared with non-streaming reasoning, using a four-layer one-dimensional convolutional network with a segment length of 4 as an example. Figure 3 (b) The computational behavior in the streaming inference is consistent; each yellow dot is calculated from the current time and the previous three data points. This is ensured by the causal relationships in the causal system, that is, forward propagation is performed using only current and historical information. In contrast, Figure 3 In (a), non-streaming inference exhibits inconsistent computational behavior, leading to a significant drop in performance at the intersection of adjacent segments. Furthermore, non-streaming inference is limited to data within the current segment, while streaming inference can reuse past data through historical information caching, allowing for a total of ten data points to be used. Therefore, introducing additional perspectives does not increase computational complexity, as they are derived from the results of previous inference. The entire inference process requires only the computation shown by the red line in the figure.

[0058] Based on this segmented streaming inference design, this invention provides a real-time speech conversion method for virtual live streaming environments. This method converts the input speech stream through segmented streaming inference to achieve high-quality conversion results. Figure 4The overall flow of the method is shown. During the conversion process, the speech of the source speaker is continuously captured from the input stream and then fed into a feature mapping system in short, fixed-length segments (i.e., typically 50-200ms). This system has three streaming stages: acoustic feature extraction, acoustic feature conversion, and speech reconstruction. The acoustic feature extraction stage aims to extract the standard log-Mel spectrum (hereinafter referred to as Mel spectrum) from the raw waveform data. To avoid unnatural head-to-tail connections caused by segmentation, buffers are established between adjacent input segments for connection, achieving the same result as non-real-time preprocessing. In the acoustic feature conversion stage, a pre-trained convolutional recurrent neural network (CRNN) model is used to map the segmented Mel spectrum to the feature space of the target speaker. This model employs a fully causal structure to maintain consistency between the training and inference stages and uses a buffering mechanism to transfer long-term information between segment-by-segment inference. Finally, in the speech reconstruction stage, the original MB-MelGAN is modified into a streaming MB-MelGAN to ensure that there are no splicing artifacts when concatenating the converted speech segments. After the three stages, the output waveform also exists in segments of equal length and is directly output for downstream use.

[0059] The following sections will discuss in detail the three modules: acoustic feature extraction, acoustic feature conversion, and speech reconstruction.

[0060] The acoustic feature extraction module is responsible for extracting acoustic features from continuous speech input for subsequent conversion processing. However, traditional continuous speech segment feature extraction methods have some problems, such as discontinuities at joints between segments and displacement between correct features, which can lead to overlap loss and unnatural splicing, such as... Figure 5 As shown. To address these issues, this invention proposes a novel stream processing method: establishing contextual connections between adjacent input segments through buffering to ensure feature extraction performance is equivalent to that of complete speech input. Specifically, acoustic features are extracted by segmenting the input speech, where each segment consists of segments with a specific frame length L. f and step size L s It consists of n acoustic feature frames. The segment length is calculated as L. c =L f +(n-1)×L s This is to ensure that the reconstructed speech has the same length. In each segment extraction, the last L... f -L s The speech is stored in a buffer and concatenated to the beginning of new segments before subsequent feature extraction. This method ensures that the feature extraction is equivalent to the complete speech input and avoids discontinuities at joints between segments and displacement between correct features. Experimental results show that using the segment-flow method to extract acoustic features can significantly improve the performance and efficiency of speech conversion systems, especially when processing long speech inputs.

[0061] The acoustic feature transformation module converts the source Mel spectrum from the source domain into another Mel spectrum in the target domain. The feature transformation model proposed in this paper follows the StarGAN framework, which has been demonstrated to be efficient in multi-objective speech conversion tasks in recent studies.

[0062] Figure 6 The network model structure of the aforementioned feature transformation model is shown, which consists of a generator, a timbre extraction network, and three discriminators. The specific descriptions of these three parts are as follows:

[0063] The generator G employs a fully causal structure, meaning the model's forward propagation depends only on current and past information. It consists of an encoder, a unidirectional LSTM, and a decoder network, with only the decoder network using auxiliary-style encodings. The encoder uses downsampled residual blocks with causal convolutional layers for content embedding extraction. Instance normalization (IN) layers are more efficient at removing the original style and producing clean content embeddings compared to batch normalization (BN) layers. After passing through the encoder, the content is fed into a single-layer unidirectional convolutional LSTM with a kernel size of 1×1 to obtain long-term dependencies. The decoder has a symmetric structure with the encoder, the same number of layers and configuration, but uses adaptive instance normalization (AdaIN) with style codes for style transfer. Finally, the output is reshaped to the same size as the input Mel spectrum through 1×1 convolutions.

[0064] The timbre extraction network M extracts timbre from the target virtual timbre domain y based on random implicit codes z∈Z. trg Generate timbre code s = M(z,y) in ∈Y trg The hindcode is sampled from a Gaussian distribution to provide distinct style representations across all domains. The style mapping network consists of a multilayer perceptron (MLP) architecture with multiple output branches providing style codes for all available domains. Specifically, the style code representations for all domains are shared up to the last layer, where domain-specific projections are applied to the shared representation.

[0065] Three discriminators, D (i) The classes i = 0, 1, and 2 have the same architecture but kernel sizes of 3, 5, and 7 respectively, to obtain different perspectives and thus achieve better joint discrimination results. Each discriminator includes several shared downsampling ResBlock layers to learn common features between real and pseudo samples across all domains. These common features are then input into a target domain-specific binary classifier.

[0066] To perform adversarial training on the causal feature transformation model, timbre extractor, and discriminator network, we employed a series of loss functions during the model training phase. For ease of formulation, variables are defined here. X represents the Mel spectrum of the input speech, y... src and y trg The speaker domains of the source speaker and the target speaker are respectively, z∈Z is a latent variable, and s=M(z,y) trg ) represents the timbre feature vector of the target speaker. Let be the vocal timbre feature vector of the source speaker. The corresponding loss function is as follows:

[0067] Adversarial loss function:

[0068]

[0069] Cycle consistency loss function:

[0070]

[0071] Where ||·||1 is the L1 norm;

[0072] Content consistency loss function:

[0073]

[0074] ASR stands for Speech Content Feature Extractor based on a speech recognition model;

[0075] Fundamental frequency consistency loss function:

[0076]

[0077] Where F0 is the fundamental frequency feature extractor based on the fundamental frequency extraction model;

[0078] Style diversity loss function:

[0079]

[0080] Where z1 and z2 are two random vectors sampled from a standard normal random distribution.

[0081] After obtaining the pre-trained causal feature transformation model, segmented streaming inference based on two-dimensional causal convolution, historical caching, and real-time statistical parameter prediction can be finally achieved.

[0082] Figure 7 This demonstrates the difference between ordinary convolution and causal convolution. Figure 7(a) A comparison of one-dimensional ordinary convolution and causal convolution. A significant difference lies in the padding method, where the padding in causal convolution is placed on the same side, i.e., the historical side of the time dimension. This invention employs a similar approach, achieving two-dimensional causal convolution by modifying the padding technique only in the time dimension, such as... Figure 7 As shown in (b). Using 2-D causal convolution, a history cache for streaming inference can be created (see...). Figure 3 ).

[0083] During each inference, the input from the last (k-1) time point is cached and concatenated to the beginning of the input of the same convolutional layer during the next inference, replacing the need for padding. This approach allows each short input inference to capture the entire data field as a long input without requiring additional computation.

[0084] In addition to cached causal convolution, this invention also uses the Exponentially Weighted Moving Average (EWMA) algorithm to predict statistical parameters of the IN layer in real time, thereby achieving state adaptation in long-duration streaming speech conversion. The basic idea is to assign a cumulative variable θ to statistical parameters (such as mean and scale). i Where i represents the sequence number of the current segment, i.e.

[0085] θ i =βv i +(1-β)θ i-1 ,j∈{1,…,k}

[0086] Where β is a preset parameter, and k is a preset observation range v i These are the observed values ​​of statistical parameters.

[0087] The speech reconstruction module uses a streaming vocoder based on multi-band Mel-GAN (MB-MelGAN) to reconstruct speech. MB-MelGAN significantly improves performance compared to the basic MelGAN by introducing techniques such as extended convolution, multi-resolution STFT loss, and multi-band processing. It takes the Mel spectrum as input, predicts the target waveform for each sub-band, and then combines all sub-band audio signals into a full-band signal using a pseudo-orthogonal mirror filter bank (Pseudo-QMF) to compute the target full-band audio. However, as an extended convolution application, MB-MelGAN requires a wider receptive field to obtain better contextual relationships, and directly applying it to segment speech synthesis can lead to a lack of long-term dependencies and continuity between segments.

[0088] To address these issues, this invention modifies MB-MelGAN into a causal and streaming architecture using causal convolution and history caching techniques. This invention retains the network structure but replaces all convolutions with causal convolutions to ensure the system does not rely on future information during training and inference. This invention uses history caching to pass long-term information from the vocoder and filters to achieve segment-by-segment streaming inference. Due to the caching mechanism, segment-by-segment inference is equivalent to whole-segment inference. Therefore, this invention can directly concatenate output waveforms to obtain high-quality speech without any post-processing for smooth splicing.

[0089] like Figure 1 As shown, this embodiment of the invention proposes a real-time speech conversion system in a virtual live streaming environment, including...

[0090] The input processing module continuously records the voice stream from the user side through the input device and segments the voice stream into non-overlapping segments with fixed short segment lengths (approximately 50–200 ms).

[0091] The acoustic feature extraction module is used to extract the standard log-Mel spectrum from the segmented speech stream data output by the input processing module as speech features;

[0092] The acoustic feature conversion module maps the aforementioned speech features into the target domain space, generating virtual speech features that conform to the target virtual timbre. Specifically, a pre-trained convolutional recurrent neural network (CRNN) based on causal convolution is used for speech feature conversion, a pre-trained timbre extraction network is used for virtual timbre extraction, and a pre-trained fundamental frequency feature extraction network is used for feature extraction.

[0093] The speech reconstruction module uses an improved real-time MB-MelGAN neural network vocoder to restore the virtual speech features into a time-domain waveform of the same length as the input speech segment.

[0094] The output processing module is used to store the aforementioned time-domain waveform into the output buffer of the virtual sound card for downstream real-time application software to read.

[0095] The front-end interface module provides users with a user-friendly interactive window for configuring system parameters, displaying input and output voice waveforms and spectra in real time, and providing feedback on real-time operating parameters.

[0096] To facilitate parameter configuration and real-time monitoring of system operation, this system provides a front-end interface module, such as... Figure 8 As shown. Users can configure input and output devices (i.e., ...) in the configuration area. Figure 8①) and configure the input of the live streaming software to a virtual source provided by the audio virtual sound card (e.g., VB-CABLE1 on Windows and Soundflower2 on MacOS) to enable audio transmission between the system and the output live application. Users can select the target speech and switch it at any time during the live stream. The system continuously records the user's speech and displays it in the input area (i.e., ...). Figure 8 (②) Displays the real-time audio input. Users can view the output area (i.e.,...) Figure 8 (③) Receives system feedback information and can directly listen to the converted audio by turning on the feedback option for real-time adjustments. Status bar (i.e., Figure 8 Section ④) displays key operational information about the system, such as real-time factors, operation time, latency, and other relevant metrics. By using the front-end interface module provided by this system, users can easily control and monitor the speech conversion process and obtain real-time feedback and information.

[0097] In addition to the standard usage modes mentioned above, this system offers several other modes. The recording mode employs a traditional method of recording first and then converting, allowing users to process pre-recorded speech. The speech enhancement mode utilizes real-time neural network speech enhancement tools, effectively suppressing environmental noise and improving conversion quality, especially in noisy environments such as airports and outdoors. It's important to note that speech enhancement is generally unnecessary in normal environments, as the system is robust to noise in such conditions. The GPU-accelerated mode leverages GPU hardware for rapid inference, further reducing overall latency and significantly decreasing CPU load. The digital watermarking mode embeds inaudible and robust digital markers into the audio to aid in the identification and preservation of speech authenticity. These different modes allow the system to be flexibly applied in various scenarios.

[0098] To verify the technical effectiveness of this invention, the described real-time virtual speech generation system was compared with mainstream multi-objective speech conversion methods, including FragmentVC, AutoVC, and DiffVC, on the VCTK dataset. The table below summarizes the Mel Cepstral Distortion (MCD), Word Error Rate (WER), Classification Accuracy (CLS), and Latency of the four methods, with results shown in the table. Experimental results show that this invention outperforms FragmentVC and AutoVC in conversion quality, reducing MCD by 2.21 dB and 2.73 dB, respectively. Compared with DiffVC, this invention exhibits similar capabilities, with an MCD difference of only 0.28 dB. In terms of intelligibility, this invention achieves the best WER of 17.65%, which is 53.74%, 47.71%, and 6.63% higher than the other three methods, respectively. Regarding similarity to the target timbre, the CLS of the four methods were 44.64%, 89.96%, 95.62%, and 92.72%, respectively. This invention demonstrates identity conversion capabilities comparable to DiffVC, FragmentVC, and AutoVC. Furthermore, this invention exhibits the best latency performance, with an average latency of only 0.28s, compared to 61.0s, 8.01s, and 27.29s for the other three methods, respectively. This is attributed to the segment-by-segment streaming inference technique proposed in this invention, which effectively mitigates the duration of individual input speech and reduces the computational overhead of individual inferences.

[0099] Table 1 compares the performance metrics of this invention and three mainstream methods, including MCD, WER, CLS, and latency.

[0100]

[0101]

[0102] A subjective auditory experiment was further conducted using the Mean Opinion Score (MOS) to investigate users' subjective opinions on the real-time speech-to-text conversion method proposed in this invention for a virtual live streaming environment. The experiment recruited 30 hearing-normal volunteers, including 13 men and 17 women, aged 18 to 28. Volunteers were asked to listen to 20 sets of audio, each set containing one source audio, one target audio, and four converted audios generated by four different methods. Volunteers rated their intuitive perception of audio quality, intelligibility, and similarity on a scale of 1 to 5, with higher scores indicating better performance. Figure 9The average subjective scores are shown. Our method achieves scores of 3.71 for audio quality, 3.66 for intelligibility, and 3.96 for similarity. Compared to AutoVC and FragmentVC, our method performs well in all three metrics, consistent with the objective experimental results shown in Table 2, indicating a superior auditory experience. Compared to Diff-VC, our method performs slightly worse in audio quality and intelligibility, with scores decreasing by 0.42 and 0.02 respectively, but outperforms by 0.19 in similarity. These results confirm that, in terms of auditory performance, diffusion-based DiffVC significantly outperforms existing VC methods. However, in terms of conversion similarity, it remains slightly lower than our proposed method. Overall, subjective experiments demonstrate that our method exhibits good overall auditory performance, especially in identity conversion.

[0103] Figure 10 This study demonstrates the time overhead required by the present invention to process speech signals of varying lengths on different computing platforms. The real-time performance of the invention is investigated using key metrics such as RTF and latency. Experiments were conducted on multiple personal computers, including servers equipped with different types of CPUs and GPUs, such as desktops (Intel i5 10400, NVIDIA RTX 2060s), Lenovo Xiaoxin laptops (AMD R5 4600U 35W), Lenovo Y9000K laptops (Intel i7 12800HX, NVIDIA RTX 3070Ti), and Apple Macbook Air laptops (Apple Silicon M2). Figure 10 The average time cost of block processing using different segment sizes on different platforms is shown. The real-time boundary, where the time cost of processing a segment equals the segment length, indicates that the points to its right satisfy the real-time requirement RTF < 1. It was observed that when the segment length is greater than 100, the invention is faster than real-time on all platforms. As the segment length increases from 50 to 250, the time cost on all platforms gradually increases from 48.6 to 66.6, on average, the growth rate is slower than the segment length. This means that the invention can achieve real-time performance with larger segment lengths. Furthermore, all GPU platforms and the Apple M2 (CPU) platform can achieve real-time performance with a latency of 50. This inspires users to use GPU acceleration mode to reduce the segment length while increasing inference speed for lower latency. The table below further illustrates the real-time performance of this method at different segment lengths and its conversion effect. The table shows that the processing time cost increases slowly with increasing block size, while the receiving latency is precisely the segment length. From the three metrics MCD, WER, and CLS, it can be seen that as the segment length increases, the overall performance gradually improves due to the continuously improving accuracy of dynamic parameter estimation. However, the rate of performance improvement gradually slows down, entering a plateau around 200. Therefore, a segment length of 150 or 200 is used to achieve low latency and good results.

[0104] Table 2 shows the real-time performance of this invention on a Lenovo Y9000K laptop with different segment lengths, including the calculation of total time overhead, real-time rate, latency, MCD, WER, and CLS for a segment. The values ​​before and after the " / " represent the measurements on the CPU and GPU, respectively.

[0105]

[0106]

[0107] The above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the content disclosed in this invention should be considered within the scope of protection of this invention.

Claims

1. A real-time speech conversion method in a virtual live streaming environment, characterized in that, The method includes: System parameters are configured through the front-end interactive interface, and the input and output voice waveforms and spectrum are displayed in real time, with real-time feedback on the running status. The system continuously records the user's voice stream using a microphone as the raw voice input, and then segments the voice stream into non-overlapping segments with fixed lengths of 50-200ms to obtain segmented voice. Standard log-Mel spectrum is extracted from segmented speech to obtain segmented input speech features; Based on the pre-set virtual speech target, the segmented input speech features and timbre feature vectors are simultaneously input into a pre-trained feature conversion model for processing, generating segmented output speech features that conform to the target virtual timbre; The segmented output speech features are input into the speech reconstruction model to reconstruct speech, generating segmented output speech of the same length as the input speech segment that conforms to the target virtual timbre; The segmented output speech is directly spliced ​​together in chronological order to obtain the real-time generated virtual speech; The extraction of the standard log-Mel spectrum processing employs an inter-segment buffering mechanism, specifically: Remove the padding mechanism when extracting the time-frequency plot during standard log-Mel spectrum analysis; When extracting the standard log-Mel spectrum of each speech segment, the last part of the speech is buffered. Before extracting the standard log-Mel spectrum of each speech segment, the buffer from the previous processing is concatenated to the beginning of the input segment. The length of the last part of the speech is the frame length minus the step length.

2. The real-time speech conversion method in a virtual live streaming environment according to claim 1, characterized in that, The feature transformation model comprises two stages: model inference and model training. The model inference stage includes a timbre extraction network and a generator network. The timbre extraction network is used to extract the timbre code of the target virtual timbre, and the generator network is used to map the input features to the target timbre domain represented by the timbre code. In the model training stage, in addition to the timbre extraction network and the generator network, three discriminator networks are also included to perform adversarial training together with the generator network.

3. The real-time speech conversion method in a virtual live streaming environment according to claim 2, characterized in that, The generator network described above employs a purely causal system structure and implements streaming processing through an inter-segment caching mechanism, including: A mapping network is constructed using causal convolutional layers and unidirectional LSTM layers, so that the network can only use information at the current time point and information before the current time point during the training and inference phases, while shielding unknowable future information. Cancel the temporal padding mechanism for all causal convolutional layers; A buffer is set before all causal convolutional layers. During each segment inference process, the data of the last time unit of the input tensor is stored in the buffer, and the intermediate results stored in the buffer in the previous segment inference are concatenated to the beginning of the input. The last time unit is the kernel size - 1. Set up an LSTM cache to cache the output of the LSTM layer during each segment inference, so that it can be passed as a parameter to the LSTM layer during the next segment inference; the cached output of the LSTM layer includes the output of the last time step and the state of the LTMCell of the last time step; The EWMA algorithm is used to dynamically estimate the statistical parameters of the IN layer to adapt to long-term speaking states.

4. The real-time speech conversion method in a virtual live streaming environment according to claim 1, characterized in that, The speech reconstruction model is a streaming modification of the MB-MelGAN network, employing a purely causal structure and implementing streaming processing through an inter-segment caching mechanism, including: Replace ordinary convolutional layers with causal convolutional layers, and eliminate the temporal padding mechanism for all causal convolutional layers; A buffer is set before all causal convolutional layers. During each segment inference process, the data of the last time unit of the input tensor is stored in the buffer, and the intermediate results stored in the buffer in the previous segment inference are concatenated to the beginning of the input. The last time unit is the kernel size - 1.

5. The real-time speech conversion method in a virtual live streaming environment according to claim 1, characterized in that, The segmented speech is processed by a convolutional neural network based on adversarial training to perform speech denoising, and the segmented output speech is processed by digital watermarking technology to perform digital watermarking anti-counterfeiting processing on the output time domain waveform.

6. A real-time speech conversion device in a virtual live streaming environment, characterized in that, The device includes: The front-end interface module allows for system parameter configuration, real-time display of input and output voice waveforms and spectrum, and real-time feedback of operating status. The input processing module continuously records the user's voice stream through the microphone as the raw voice input, and segments the voice stream into non-overlapping segments with a fixed length of 50~200ms to obtain the segmented voice. The acoustic feature extraction module extracts the standard log-Mel spectrum from the segmented speech to obtain the segmented input speech features; The acoustic feature conversion module, based on a pre-set virtual speech target, simultaneously inputs the segmented input speech features and timbre feature vectors into a pre-trained feature conversion model for processing, generating segmented output speech features that conform to the target virtual timbre; The speech reconstruction module inputs the segmented output speech features into the speech reconstruction model to reconstruct the speech and generate segmented output speech that is the same length as the input speech segment and conforms to the target virtual timbre. The output processing module directly splices the segmented output speech in chronological order to obtain the real-time generated virtual speech.

7. The real-time speech conversion device in a virtual live streaming environment according to claim 6, characterized in that, The acoustic feature conversion module uses a pre-trained convolutional recurrent neural network (CRNN) based on causal convolution for speech feature conversion and a pre-trained timbre extraction network for virtual timbre extraction.

8. The real-time speech conversion device in a virtual live streaming environment according to claim 7, characterized in that, The convolutional recurrent neural network has the following structure: a CNN encoder, a convolutional LSTM, and a CNN decoder. The CNN encoder consists of a 6-layer residual block network using instance-norm layers. The CNN decoder is an adaptive residual block network consisting of 6 layers using instance-norm layers, with an additional convolutional kernel. The CNN decoder consists of convolutional layers, where each adaptive residual segment receives the timbre vector output by the timbre extraction network to guide virtual speech synthesis.

Images