A low-latency real-time voice conversion method and system

Abstract

The present application relates to the technical field of speech conversion, and particularly relates to a low-delay real-time speech conversion method and system, the method comprising: when a speech conversion task is received, dividing a speech processing procedure in the speech conversion task into multiple independent stages for parallel processing through an asynchronous pipeline architecture; quantifying sensitivity of time delay of each independent stage to total delay of speech conversion, and evaluating influence of the time delay of each independent stage on output sound quality; according to the sensitivity and the evaluation of influence on the output sound quality, optimizing processing time of each independent stage, balancing the total delay of speech conversion and the final output sound quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of voice conversion, in particular to a low-delay real-time voice conversion method and system. BACKGROUND

[0002] Voice conversion (VC) is a technology that aims to change the speaker characteristics in speech while preserving the linguistic content. Its application scenarios mainly include: personalized voice assistants: customizing sounds that meet user preferences, such as star vocal timbres; entertainment and creation: music cover, film dubbing (such as character voice replacement) or virtual anchor voice design; accessibility technology: generating natural speech for aphasia or helping language-impaired people adjust pronunciation; privacy protection: anonymizing voices in calls to protect identity information; education and language learning: imitating the pronunciation of native speakers to assist in correcting accents; virtual reality (VR / AR): generating dynamic and diverse voices for virtual characters to enhance immersion.

[0003] The delay of a voice conversion system mainly comes from algorithm processing delay (including speech framing, feature extraction, algorithm reasoning, etc.), system hardware delay, transmission delay, etc. Therefore, the total delay time of the voice conversion system can be considered as the sum of the input buffer delay time, the algorithm processing delay time, the output buffer delay time, and the transmission delay time. In some use scenarios with high real-time requirements, it is necessary to reduce the system delay time to ensure the efficiency and real-time performance of the voice conversion system.

[0004] Based on this, we propose a method to reduce the delay time of the system from the perspective of reducing the algorithm processing time. SUMMARY

[0005] The present application divides the voice conversion process into multiple independent stages, optimizes the processing time of each stage, and achieves the purpose of optimizing the algorithm delay of the voice conversion system.

[0006] The technical solution proposed by the present application is: a low-delay real-time voice conversion method and system, the method comprising:

[0007] Upon receiving a voice conversion task, the voice processing process in the voice conversion task is divided into multiple independent stages and processed in parallel through an asynchronous pipeline architecture;

[0008] Quantify the sensitivity of the time delay of each independent stage to the total delay of voice conversion, and evaluate the impact of the time delay of each independent stage on the output audio quality;

[0009] According to the sensitivity and the evaluation of the impact on the output audio quality, optimize the processing time of each independent stage to balance the total delay of voice conversion and the final output audio quality.

[0010] Preferably, the voice data is divided into multiple independent stages for parallel processing by the asynchronous pipeline architecture, including:

[0011] The voice processing flow is divided into a recording framing stage, a feature extraction stage, a voice conversion model inference stage, and a waveform synthesis stage.

[0012] Each independent stage is assigned an independent thread, and the multi-core CPU or GPU is used to perform voice processing tasks in parallel.

[0013] Each independent thread uses a lock-free queue or ring buffer to transfer data, reducing thread data blocking.

[0014] Preferably, the sensitivity of the time delay of each independent stage to the total delay of voice conversion is quantified, including:

[0015] Measure the processing time and total delay of each independent stage.

[0016] Adjust the processing time of each stage separately, and re-measure the total delay.

[0017] Calculate the delay sensitivity coefficient of each independent stage , and arrange the sensitivity coefficients in descending order according to their sizes to form a sensitivity sequence ;

[0018] Where, respectively represent the total delay time change, the total delay time, the change of the processing time of the th independent stage, and the processing time of the th independent stage, denotes the number of independent stages.

[0019] The effect of the time delay of each independent stage on the output audio quality is evaluated, including:

[0020] Quantify the impact of reducing the processing time of each independent stage on the quality of the output audio, including:

[0021] Establish a quality attenuation function: ; Where, denotes the attenuation coefficient; denotes the theoretical highest audio quality of the th independent stage, i.e. the highest MOS score; denotes the audio quality of the th independent stage;

[0022] Test the quality of the output audio under different processing times, and determine by linear regression algorithm.

[0023] arranging the voice quality of each independent stage in ascending order according to the numerical value, to form a sequence of attenuation coefficients .

[0024] Preferably, the optimization of the processing time of each independent stage according to the evaluation of sensitivity and the impact on the output voice quality, balances the total delay of voice conversion and the final output voice quality, including:

[0025] building a delay voice optimization model to jointly optimize the total delay and the output voice quality;

[0026] The delay voice optimization model is ;

[0027] wherein, respectively represent the target total delay, the output voice quality, and the target output voice quality; respectively represent the processing time threshold and the voice quality threshold; respectively represent the delay time weight and the score weight; wherein, ;

[0028] According to the arrangement of the sensitivity coefficients in the sensitivity sequence, the corresponding independent stage is selected in turn as the stage to be optimized;

[0029] The delay voice optimization model is optimized by an iterative optimization algorithm to seek the optimal processing time of the stage to be optimized, specifically:

[0030] The gradient descent algorithm is used to solve the solution of the delay optimization voice optimization model, including:

[0031] obtaining the gradient of the loss function ;

[0032] iterative processing:

[0033] ; wherein, learning rate, respectively the processing time after iteration and the processing time before iteration;

[0034] when the maximum number of iterations is reached or the change of the loss function is less than the preset change threshold , stop iteration;

[0035] obtaining the processing time value at this time as the optimal processing time of the stage to be optimized;

[0036] The optimal processing time of all independent stages is added to obtain the optimized total delay time.

[0037] Preferably, the optimization of the processing time of each independent stage according to the evaluation of sensitivity and the impact on the output voice quality, balances the total delay of voice conversion and the final output voice quality, further including:

[0038] By optimizing the parameters of the model in each independent stage, the processing time is minimized while ensuring sound quality, including:

[0039] Obtain the parameters of the model to form a model parameter set ; wherein, represents the i-th parameter of the model; represents the number of model parameters;

[0040] Construct a joint optimization model ; wherein, respectively represent the delay time weight of the model and the score weight of the model;

[0041] Use Latin hypercube sampling or Bayesian optimization algorithm to explore the model parameter space, collect model parameters, and normalize the collected model parameters to form multiple model parameter combination vectors ; ;

[0042] Construct a model processing time function ; wherein represents the parameter fitting coefficient;

[0043] Construct a model sound quality model ; wherein respectively represent the parameter decay coefficient;

[0044] Update the model parameters, including:

[0045] Calculate the gradient of the target loss function in the joint optimization model , ;

[0046] Iteratively update the model parameters, that is ;

[0047] When the number of iterations is reached, stop updating the model parameters, obtain the optimal model parameters, and optimize the parameters of the model in the independent stage to reduce the processing time of the corresponding independent stage.

[0048] Preferably, the processing time of each independent stage is optimized according to the sensitivity and the evaluation of the impact on the output sound quality, and the total delay of the speech conversion and the final output sound quality are balanced, which further includes:

[0049] Add a speaker recognition function in the speech conversion model inference stage, and trigger the model parameter optimization process only when the detected speech is the target speaker, thereby reducing the time delay, which specifically includes:

[0050] Collect the speaker's voiceprint features and normalize them to form a voiceprint recognition feature vector;

[0051] ​Import a pre-trained voiceprint verification model, calculate the cosine similarity between the voiceprint recognition feature vector and the pre-stored voiceprint feature vector of the target speaker;

[0052] If the cosine similarity is greater than a preset matching threshold, the speaker is determined to be the target speaker, and a joint optimization model is started to optimize the speech conversion model in the inference stage of the speech conversion model;

[0053] Otherwise, the joint optimization model is not started.

[0054] Preferably, the processing time of each independent stage is optimized according to the evaluation of sensitivity and the impact on the output voice quality, and the total delay of speech conversion and the final output voice quality are balanced, and further comprising:

[0055] Obtaining the voiceprint recognition feature vector identified by the speech conversion model inference stage;

[0056] If the biometric feature vector is determined to be the target speaker, the voiceprint recognition feature vector is stored in the cache area, and the voiceprint recognition feature vector is embedded in the output speech, specifically including:

[0057] The voiceprint recognition feature vector is added to the speech feature converted by the speech conversion model inference stage, that is, , wherein, represents the voiceprint recognition feature vector, represents the output speech feature vector of the model in the speech conversion model inference stage, represents the output speech feature vector of the speech conversion model inference stage; the dimension of is the same as the dimension of represents the embedding intensity;

[0058] The voiceprint recognition feature vector is extracted from , and it is determined whether it is a target speaker. If it is, a fast vocoder LPCNet is used to synthesize a sound wave signal and output; otherwise, a high-fidelity vocoder WaveNet is used to synthesize a sound wave signal and output.

[0059] Preferably, the embedding intensity is dynamically adapted according to the model parameter adjustment, to balance the robustness of embedding the voiceprint recognition feature vector and the voice quality, including:

[0060] Establishing a mapping relationship between and the model parameters, including:

[0061] Let the adjustment formula of the embedding intensity be ; wherein, respectively represent the attenuation coefficient and the robustness compensation of the parameter change to the embedding intensity; represents the parameter change; respectively represent the values adjusted according to the model parameters and the maximum values allowed.

[0062] A low-latency real-time speech conversion system comprising a processor and a memory and a communication module connected to the processor, the system being configured to perform the low-latency real-time speech conversion method.

[0063] A computer-readable storage medium storing a computer program, the computer program being executed by a processor to implement the low-latency real-time speech conversion method and system.

[0064] Advantages of the present application:

[0065] 1. Based on the asynchronous pipeline technology, the present application divides the algorithm processing process in the speech conversion process into multiple independent stages, and based on the sensitivity of each independent stage to the total delay time, the corresponding independent stage is selected for processing time optimization to reduce the delay time of the algorithm processing process.

[0066] 2. In the process of optimizing the processing time, the parameters of the model in the independent stage are optimized, that is, the model parameters are optimized to optimize the processing time of the stage; and the voiceprint recognition feature is embedded in the output signal of the speech conversion model inference stage, so as to judge whether it is the target speaker by recognizing the voiceprint feature, and then judge whether the model of the waveform synthesis stage needs to be optimized. In the process of embedding the voiceprint recognition feature, the embedding strength is associated with the model parameter adjustment amount, and the embedding strength is adjusted, the embedding strength is reduced during model compression, and the voiceprint recognition feature is avoided to affect the sound quality of the transmitted audio; during model recovery, the embedding strength is increased, and the anti-interference performance of the voiceprint recognition feature is enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0067] Figure 1 A flowchart of the low-latency real-time speech conversion method of the present application. DETAILED DESCRIPTION

[0068] The following description is provided to disclose the present application so that those skilled in the art can implement the present application. The preferred embodiments in the following description are only examples, and other obvious modifications can be made by those skilled in the art. The basic principles of the present application defined in the following description can be applied to other embodiments, modifications, improvements, equivalents and other technical solutions without departing from the spirit and scope of the present application.

[0069] It can be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of one element can be one, and in another embodiment, the number of the element can be multiple, and the term "one" cannot be understood as a limitation on the number.

[0070] Embodiment one:

[0071] Reference Figure 1 The technical scheme provided by the present application is: a low-delay real-time voice conversion method and system, comprising the following steps:

[0072] Step 1: When a voice conversion task is received, the voice processing flow in the voice conversion task is divided into multiple independent stages for parallel processing through an asynchronous pipeline architecture, specifically including the following steps: the voice processing flow is divided into a recording framing stage, a feature extraction stage, a voice conversion model inference stage, and a waveform synthesis stage;

[0073] An independent thread is assigned to each independent stage, and the voice processing task is executed in parallel using a multi-core CPU or GPU;

[0074] Each independent thread uses a lock-free queue or a ring buffer to pass data, reducing thread data blocking.

[0075] Step 2: Quantify the sensitivity of the time delay of each independent stage to the total delay of voice conversion, and evaluate the impact of the time delay of each independent stage on the output audio quality, including the following steps:

[0076] Measure the processing time and total delay of each independent stage;

[0077] Adjust the processing time of each stage individually, and re-measure the total delay;

[0078] Calculate the delay sensitivity coefficient of each independent stage , and arrange the sensitivity coefficients in descending order according to their sizes to form a sensitivity sequence ;

[0079] Where, respectively represent the total delay time change, the total delay time, the change of the processing time of the th independent stage, and the processing time of the th independent stage, denotes the number of independent stages;

[0080] The evaluation of the impact of the time delay of each independent stage on the output audio quality includes:

[0081] Quantify the impact of the reduction of the processing time of each independent stage on the quality of the output audio, specifically including:

[0082] Establish a quality attenuation function: ; Where, denotes the attenuation coefficient; denotes the theoretical highest audio quality of the th independent stage, i.e. the highest MOS score; representing the quality of sound of the first independent stage;

[0083] Testing the quality of the output audio at different processing times, determining the ;

[0084] Arranging the quality of sound of each independent stage in ascending order according to the numerical value, to form a sequence of attenuation coefficients .

[0085] Step 3, according to the sensitivity and the evaluation of the influence on the output quality, optimizing the processing time of each independent stage, balancing the total delay of speech conversion and the final output quality, including the following steps: constructing a delay speech optimization model, jointly optimizing the total delay and the output quality;

[0086] The delay speech optimization model is ;

[0087] Wherein, respectively represent the target total delay, the output quality, the target output quality; respectively represent the processing time threshold and the quality threshold; respectively represent the delay time weight and the score weight; wherein, ;

[0088] According to the arrangement of the sensitivity coefficient in the sensitivity sequence, the corresponding independent stage is selected in turn as the stage to be optimized;

[0089] Through the iterative optimization algorithm delay speech optimization model, seeking the optimal processing time of the stage to be optimized, specifically:

[0090] Solving the solution of the delay optimization speech optimization model by using gradient descent algorithm, including:

[0091] Obtaining the gradient of the loss function ;

[0092] Iterative processing:

[0093] ; wherein, learning rate, respectively the processing time after iteration and the processing time before iteration;

[0094] When the maximum number of iterations is reached or the change of the loss function is less than the preset change threshold , stop iteration;

[0095] Obtaining the processing time value at this time as the optimal processing time of the stage to be optimized;

[0096] Adding up the optimal processing time of all independent stages, the optimized total delay time is obtained.

[0097] For example, the original recording frame stage processing time is 5ms, the sound quality (MOS value) is 4.8; the feature extraction stage processing time is 10ms, the sound quality (MOS value) is 4.6; the speech conversion model inference stage processing time is 30ms, the sound quality (MOS value) is 4.5; the waveform synthesis stage processing time is 15ms, the sound quality (MOS value) is 4.7; the target total delay time , ;

[0098] The calculation obtains ; the sensitivity sequence ;

[0099] According to the sensitivity sequence, the processing time of the speech conversion model inference stage, the waveform synthesis stage, the feature extraction stage and the recording frame stage is optimized in turn;

[0100] Through the iterative optimization algorithm, the optimal processing time of the speech conversion model inference stage is 20ms, the processing time of the feature extraction stage is 15ms, and the other two independent stages remain in the original state.

[0101] The speech conversion model inference time is reduced from 30ms to 20ms; the total delay is reduced by 1.5x(30-20) / 30=5%, and the sound quality is reduced from 4.5 to 4.2; the feature extraction time is optimized from 10ms to 15ms, the delay is increased by 0.8x(15-10) / 10=4%, and the sound quality is improved from 4.6 to 4.7; the total delay time is reduced by 1% (the total delay time is changed to 198ms); the output sound quality of the final system .

[0102] Embodiment two:

[0103] In order to complete the task of this stage, the independent stage usually needs the corresponding model, such as speech conversion model, feature extraction model and waveform synthesis model, etc.

[0104] Optimizing the processing time of each stage can also be realized by optimizing the working parameters of the model in each stage, and for this purpose, on the basis of embodiment one, we propose a technical solution:

[0105] By optimizing the parameters of the model in each independent stage, the processing time is minimized while ensuring the sound quality, including:

[0106] Obtain the parameters of the model to form a model parameter set ; wherein, represents the i-th parameter of the model; represents the number of model parameters;

[0107] ​Construct a joint optimization model ; wherein, respectively represent the delay time weight of the model and the score weight of the model;

[0108] Use Latin hypercube sampling or Bayesian optimization algorithm to explore the model parameter space, collect model parameters, and normalize the collected model parameters to form a plurality of model parameter combination vectors ; ;

[0109] Construct a model processing time function ; wherein represents the parameter fitting coefficient;

[0110] Construct a model sound quality model ; wherein respectively represent the parameter decay coefficient;

[0111] Update the model parameters, including:

[0112] Calculate the gradient of the target loss function in the joint optimization model , ;

[0113] Iteratively update the model parameters, that is, ;

[0114] When the number of iterations is reached, stop updating the model parameters, obtain the optimal model parameters, and optimize the parameters of the model in the independent stage to reduce the processing time of the corresponding independent stage.

[0115] For example, in the speech conversion model inference stage, the Transformer model is optimized.

[0116] The model parameters of the Transformer model include the number of model layers, the number of attention heads, the hidden layer dimension, the model weight quantization bit width, the dynamic batch size, the input speech frame number, and the pruning rate;

[0117] In this example, the selected model parameters are the number of model layers, the number of attention heads, the model weight quantization bit width, and the dynamic batch size. After normalization processing using the selected model parameters, a model parameter combination vector ;

[0118] Through experimental measurement of the model, obtain and , through Latin hypercube sampling of 100 parameter combinations, collect the measured and , fit the model processing time function and the sound quality model, and obtain the parameter fitting coefficient and the parameter decay coefficient;

[0119] Constructing the initialization parameter vector .

[0120] If the parameters of the reference Transformer model are known, the normalized composition ; and ms, the sound quality is 4.0;

[0121] The optimization process of the reference Transformer model is as follows:

[0122] Reducing the number of model layers (corresponding to ): from 12 to 8, the processing time becomes 30-4x2.5=20ms, and the sound quality becomes 4.3;

[0123] Model weight quantization bit width (corresponding to ): from 32 to 8, the processing time becomes 20x0.6=12ms; the sound quality becomes 4.0;

[0124] Dynamic batch size (corresponding to ): from 1 to 4, the processing time becomes 12 / 3=4ms, and the sound quality is 4.0;

[0125] After the above optimization process, the processing time of the speech conversion model inference stage becomes 3ms<20ms.

[0126] Embodiment Three:

[0127] In actual use, in order to protect the privacy of the user or avoid the system being used by non-target users, we propose a technical solution based on Embodiment Two, that is, in the speech conversion system, a two-stage watermark cascade optimization scheme is constructed: in the third stage (speech conversion model inference stage) and the fourth stage (waveform synthesis stage) of the system, respectively embed voiceprint features with identification function (voiceprint recognition watermark), adjust the model parameters, and map the embedding strength and the model parameters, coordinate the parameter adjustment and the embedding strength. The specific scheme is as follows:

[0128] Add a speaker recognition function in the speech conversion model inference stage, trigger the model parameter optimization process only when the detected speech is the target speaker, reduce the time delay, which specifically includes:

[0129] Collect the voiceprint features of the speaker, and normalize them to form a voiceprint recognition feature vector;

[0130] Import a pre-trained voiceprint verification model, calculate the cosine similarity between the voiceprint recognition feature vector and the pre-stored voiceprint feature vector of the target speaker;

[0131] If the cosine similarity is greater than a preset matching threshold, the speaker is determined to be a target speaker, and a joint optimization model is started to optimize a speech conversion model in an inference stage of the speech conversion model;

[0132] Otherwise, the joint optimization model is not started.

[0133] After starting the joint optimization model, a voiceprint recognition feature vector recognized by the speech conversion model in the inference stage is obtained;

[0134] If the biometric feature vector is determined to be a target speaker, the voiceprint recognition feature vector is stored in a cache area, and the voiceprint recognition feature vector is embedded in the output speech, specifically including:

[0135] The voiceprint recognition feature vector is added to the speech feature converted by the speech conversion model in the inference stage, that is, wherein, represents the voiceprint recognition feature vector, represents the output speech feature vector of the model in the speech conversion model inference stage, represents the output speech feature vector of the speech conversion model inference stage; the dimension of is the same as the dimension of

[0136] The voiceprint recognition feature vector is extracted from and it is determined whether it is a target speaker, if so, a fast vocoder LPCNet is used to synthesize a sound wave signal and output; otherwise, a high-fidelity vocoder WaveNet is used to synthesize a sound wave signal and output.

[0137] In the process of adjusting the model parameters, the embedding strength is dynamically adapted according to the model parameter adjustment, to balance the robustness of embedding the voiceprint recognition feature vector and the sound quality, specifically:

[0138] A mapping relationship between and the model parameters is established, including the following steps:

[0139] Let the adjustment formula of the embedding strength be ; wherein, respectively represent the attenuation coefficient and the robustness compensation of the parameter change to the embedding strength; represents the parameter change; respectively represent the value after the model parameter adjustment and the maximum value allowed.

[0140] For example, taking the speech conversion model optimization in the speech conversion model inference stage as an example. The initial state of the speech conversion model (Transformer) parameters:

[0141] Let the parameter​ The model pruning rate is The parameter The model weight quantization bit width is 32 bits; the original embedding strength The MOS is 4.5, and the detection rate is 95%;

[0142] Through model compression, the processing time is reduced; after model compression, the pruning rate is 0.3, and the model weight quantization bit width is 8 bits;

[0143] Therefore, the embedding strength ;

[0144] If the detection rate requirement is 90%, the robustness compensation is adjusted to 1.5, and at this time, the embedding strength ;

[0145] The measured output sound quality of the adjusted model is changed to 4.2, which meets the requirement.

[0146] The application further provides a low-delay real-time voice conversion system, comprising a processor, a memory connected with the processor and a communication module connected with the processor, and the system is used for executing the low-delay real-time voice conversion method.

[0147] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the low-delay real-time voice conversion method.

[0148] The processes described above with reference to the flowcharts can be implemented as computer software programs in accordance with embodiments of the present disclosure. Embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication section, and / or installed from a detachable medium. When the computer program is executed by a central processing unit (CPU), the above-described functions defined in the methods of the present application are performed. It should be noted that the computer readable medium of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as part of a carrier wave, in which the computer readable program code is carried. Such a propagated data signal can take a variety of forms, including but not limited to, an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium that can send, propagate or transfer the program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to, wireless, wire, optical cable, RF or the like, or any suitable combination of the above.

[0149] The computer program product of the present application can be a computer program product comprising a computer-readable medium bearing computer program code embodied therein for use with a computer. The computer program code can be code defining and / or implementing the present application. The computer program code can be written in any suitable computer readable programming language. The computer program code can be stored in a computer- readable storage medium, such as, but not limited to, any type of disk including an optical disk, a CD-ROM, a CD-R, a CD-RW, a DVD, a flash memory, a ROM, a RAM, a magnetic disk or hard drive, or any other suitable type of medium including a medium that holds the software for a particular or specialized computing purpose, or any suitable combination of media. The computer program product can be a computer program product distributed to end users, whether as a stand-alone program, as part of a physical system, or as a software download. The computer program product can be distributed on a physical medium, such as, but not limited to, a floppy disk, a CD-ROM, a CD-R, a CD-RW, a DVD, a flash memory, a ROM, a RAM, a magnetic disk or hard drive, or any other suitable type of medium, or any suitable combination of media. The computer program product can be distributed from a program distribution center, either as a tangible medium or via electronic delivery, such as from a Web site via the Internet, or from one computer to another via electronic transfer, such as by e-mail. The computer program product can be distributed in an encrypted manner, such as via encryption or via password protection.

[0150] Those skilled in the art will understand that the application described above and illustrated in the accompanying drawings is presented by way of example only and is not intended to limit the application. The present application thus extends to any and all embodiments within the scope of the following claims.

Claims

1. A low-latency real-time voice conversion method, characterized by, The method comprises: When receiving a voice conversion task, the voice processing flow in the voice conversion task is divided into multiple independent stages for parallel processing by an asynchronous pipeline architecture; Quantifying the sensitivity of the time delay of each independent stage to the total delay of voice conversion and evaluating the impact of the time delay of each independent stage on the output audio quality, including: Measuring the processing time and total delay of each independent stage; Adjusting the processing time of each stage separately, re-measuring the total delay; Calculate the delay sensitivity coefficient of each independent stage , and arrange them in descending order according to the size of the sensitivity coefficient to form a sensitivity sequence ; wherein respectively represent the total delay time variation, the total delay time, the variation of the first independent stage processing time and the first independent stage processing time, represents the number of independent stages; A quality decay function is established: ; wherein, represents a decay coefficient; represents a theoretical maximum sound quality, i.e. a maximum MOS score, of the first independent phase; represents a sound quality of the first independent phase; The quality of the output audio is tested under different processing times, and a linear regression algorithm is used to determine ; The tone qualities of each independent stage are arranged in ascending order according to the numerical values to form a sequence of attenuation coefficients ; According to the sensitivity and the evaluation of the impact on the output audio quality, optimizing the processing time of each independent stage, balancing the total delay of voice conversion and the final output audio quality, including: Building a delay voice optimization model to jointly optimize the total delay and the output audio quality; The delay voice optimization model is ; wherein, respectively represent target total delay, output sound quality, target output sound quality; respectively represent processing time threshold and sound quality threshold; respectively represent delay time weight and score weight; wherein, ; According to the arrangement of the sensitivity sequence, the corresponding independent stage is selected in turn as the stage to be optimized; Optimizing the delay voice optimization model by an iterative optimization algorithm to find the optimal processing time of the stage to be optimized, specifically: Solving the delay optimization voice optimization model by a gradient descent algorithm, including: obtaining a gradient of the loss function ; Iterative processing: ; wherein, denotes a learning rate, the processing time after the respective iteration and the processing time before the iteration; reaching a maximum number of iterations or a change in the loss function being less than a pre-set change threshold iteration is stopped. Obtain the processing time value at this time as the optimal processing time of the stage to be optimized; Add the optimal processing time of all independent stages to obtain the optimized total delay time.

2. The low-delay real-time voice conversion method of claim 1, wherein, The voice data is divided into multiple independent stages for parallel processing by an asynchronous pipeline architecture, including: The voice processing flow is divided into a recording framing stage, a feature extraction stage, a voice conversion model inference stage, and a waveform synthesis stage; Each independent stage is assigned an independent thread, and the multi-core CPU or GPU is used to perform the voice processing task in parallel; Each independent thread uses a lock-free queue or a ring buffer to transfer data, reducing thread data blocking.

3. The low-delay real-time voice conversion method of claim 2, wherein, According to the sensitivity and the evaluation of the impact on the output audio quality, optimizing the processing time of each independent stage, balancing the total delay of voice conversion and the final output audio quality, also includes: By optimizing the parameters of the model in each independent stage, the processing time is minimized while ensuring the audio quality, including: parameters of the model, constituting a model parameter set ; wherein denotes the i-th parameter of the model; denotes the number of parameters of the model;​ Constructing a joint optimization model ; wherein, respectively represent a delay time weight of the model and a score weight of the model. The model parameter space is explored using Latin hypercube sampling or a Bayesian optimization algorithm, model parameters are collected, and the collected model parameters are normalized to form a plurality of model parameter combination vectors ; ; Model building processing time function ; wherein denotes a parameter fitting coefficient; Constructing a model perceptual model ; wherein respectively represent the parameter decay coefficients; Updating the model parameters, including: Target loss function for joint optimization model Gradient finding, ; iteratively updating model parameters, i.e. ; When the number of iterations is reached, stop updating the model parameters, obtain the optimal model parameters, and optimize the parameters of the model in the independent stage to reduce the processing time of the corresponding independent stage.

4. The low-delay real-time voice conversion method of claim 3, wherein, According to the sensitivity and the evaluation of the impact on the output audio quality, optimizing the processing time of each independent stage, balancing the total delay of voice conversion and the final output audio quality, also includes: Adding a speaker recognition function in the voice conversion model inference stage, triggering the model parameter optimization process only when the voice is detected as the target speaker, reducing the time delay, specifically including: Collecting the speaker's voiceprint features and normalizing them to form a voiceprint recognition feature vector; Importing a pre-trained voiceprint verification model to calculate the cosine similarity between the voiceprint recognition feature vector and the pre-stored voiceprint feature vector of the target speaker; If the cosine similarity is greater than a preset matching threshold, the speaker is determined to be the target speaker, and the joint optimization model is started to optimize the voice conversion model in the voice conversion model inference stage; Otherwise, the joint optimization model is not started.

5. The low-delay real-time voice conversion method of claim 4, wherein, According to the sensitivity and the evaluation of the impact on the output audio quality, optimizing the processing time of each independent stage, balancing the total delay of voice conversion and the final output audio quality, also includes: Acquiring a voice conversion model inference stage to identify a voiceprint recognition feature vector; If the voiceprint recognition feature vector is judged as a target speaker, the voiceprint recognition feature vector is stored in a cache area, and the voiceprint recognition feature vector is embedded in the output voice, specifically including: After the speech features are converted during the inference stage of the speech conversion model, a voiceprint recognition feature vector is added, i.e. ,in, This represents the feature vector for voiceprint recognition. This represents the output speech feature vector of the speech conversion model during the inference phase. This represents the output speech feature vector during the inference phase of the speech conversion model; Dimensions and The dimensions are the same; Indicates embedding strength; From extract the voiceprint recognition feature vector, and determine whether it is the target speaker. If it is, use the fast vocoder LPCNet to synthesize the sound wave signal and output; otherwise, use the high-fidelity vocoder WaveNet to synthesize the sound wave signal and output.

6. The low-delay real-time voice conversion method of claim 5, wherein, Further comprising that the embedding intensity is dynamically adapted according to model parameters to balance the robustness of embedding the voiceprint recognition feature vector and the sound quality, including: establishing a mapping relationship with model parameters, comprising: Let the adjustment formula of the embedding strength be ; wherein, respectively represent the decay coefficient and the robustness compensation of the parameter variation to the embedding strength; represent the parameter variation; respectively represent the value after the model parameter adjustment and the maximum value allowed.

7. A low-latency real-time voice conversion system comprising a processor and a memory and a communication module connected to the processor, characterized in that, The system is used to execute the low-delay real-time voice conversion method in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the low-delay real-time voice conversion method in any one of claims 1-6.

Images