Low-delay real-time voice conversion method and system

Abstract

The invention relates to the technical field of voice conversion, in particular to a low-delay real-time voice conversion method and system, and the method comprises the steps: dividing a voice processing flow in a voice conversion task into a plurality of independent stages for parallel processing through an asynchronous pipeline architecture when the voice conversion task is received; quantifying the sensitivity of the time delay of each independent stage to the total delay of voice conversion, and evaluating the influence of the time delay of each independent stage on the output tone quality; and optimizing the processing time of each independent stage and balancing the total delay of voice conversion and the final output tone quality according to the sensitivity and evaluation on the influence on the output tone quality.

Description

Technical Field

[0001] The present invention relates to the field of voice conversion technology, and in particular to a low-delay real-time voice conversion method and system. Background Art

[0002] Voice conversion (VC) is a technology designed to change the speaker's characteristics in speech while preserving the linguistic content. Its main application scenarios include: personalized voice assistants: customizing voices to suit user preferences, such as celebrity timbre; entertainment and creative: music covers, film and television dubbing (such as character voice replacement), or voice design for virtual anchors; accessibility technology: generating natural speech for people with aphasia or helping those with speech impairments adjust their pronunciation; privacy protection: anonymizing voices during calls to protect identities; education and language learning: imitating native speaker pronunciation to assist with accent correction; and virtual reality (VR / AR): generating dynamic and diverse voices for virtual characters to enhance immersion.

[0003] The main sources of latency in a speech conversion system include algorithmic processing delay (including speech framing, feature extraction, and algorithmic inference), system hardware delay, and transmission delay. Therefore, the total latency of a speech conversion system can be considered the sum of input buffer delay, algorithmic processing delay, output buffer delay, and transmission delay. In some scenarios with high real-time requirements, it is necessary to reduce system latency to ensure the efficiency and real-time performance of the speech conversion system.

[0004] Based on this, we propose a method to reduce the system delay time from the perspective of reducing algorithm processing time. Summary of the Invention

[0005] The present invention divides the voice conversion process into multiple independent stages and optimizes the processing time of each stage to achieve the purpose of optimizing the algorithm delay of the voice conversion system.

[0006] The technical solution proposed by the present invention is: a low-latency real-time speech conversion method and system, the method comprising:

[0007] When receiving a voice conversion task, the voice processing flow in the voice conversion task is divided into multiple independent stages and processed in parallel through an asynchronous pipeline architecture;

[0008] Quantify the sensitivity of the time delay of each independent stage to the total delay of speech conversion, and evaluate the impact of the time delay of each independent stage on the output sound quality;

[0009] Based on sensitivity and an assessment of the impact on output sound quality, the processing time of each individual stage is optimized to balance the overall voice conversion latency and the final output sound quality.

[0010] Preferably, the method of dividing the voice data into multiple independent stages for parallel processing by an asynchronous pipeline architecture includes:

[0011] The speech processing process is divided into the recording frame stage, feature extraction stage, speech conversion model inference stage and waveform synthesis stage;

[0012] Assign independent threads to each independent stage and use multi-core CPU or GPU to execute speech processing tasks in parallel;

[0013] Each independent thread uses a lock-free queue or ring buffer to transfer data, reducing thread data blocking.

[0014] Preferably, quantifying the sensitivity of the time delay of each independent stage to the total delay of speech conversion includes:

[0015] Measure the processing time and total latency of each individual stage;

[0016] Adjust the processing time for each stage individually and remeasure the total latency;

[0017] Calculate the delay sensitivity coefficient for each individual stage And according to the size of the sensitivity coefficient, they are arranged in descending order to form a sensitivity sequence A=(L i |i=1, 2, ..., n);

[0018] Where, ΔT total 、T total , ΔT pi 、T pi They represent the total delay time variation, total delay time, the variation of the processing time of the ith independent stage, and the processing time of the ith independent stage, respectively. n represents the number of independent stages.

[0019] The evaluation of the impact of time delay of each independent stage on the output sound quality includes:

[0020] Quantify the impact of reduced processing time at each individual stage on the quality of the output audio, specifically:

[0021] Establish the mass decay function: Among them, k i represents the attenuation coefficient; Q max,i Indicates the theoretical maximum sound quality of the ith independent stage, that is, the highest MOS score; Q i represents the sound quality of the i-th independent stage;

[0022] The quality of the output audio is tested at different processing times, and k is determined by the linear regression algorithm. i ;

[0023] Arrange the sound quality of each independent stage in ascending order according to the value to form an attenuation coefficient sequence B = (Q1, Q2, ..., Q n ).

[0024] Preferably, the step of optimizing the processing time of each independent stage based on the sensitivity and the evaluation of the impact on the output sound quality, and balancing the total delay of the speech conversion and the final output sound quality, comprises:

[0025] Build a delayed speech optimization model to jointly optimize total delay and output sound quality;

[0026] The delayed speech optimization model is

[0027] Among them, T target , Q total , Q target Respectively represent the target total delay, output sound quality, and target output sound quality; T min , Q min Represent the processing time threshold and sound quality threshold respectively; α and β represent the delay time weight and score weight respectively;

[0028] According to the arrangement of the sensitivity coefficients in the sensitivity sequence, the corresponding independent stages are selected in turn as the stages to be optimized;

[0029] The speech optimization model is delayed through an iterative optimization algorithm to find the optimal processing time for the stage to be optimized. Specifically:

[0030] The solution of the delay optimization speech optimization model is solved using the gradient descent algorithm, including:

[0031] Get the gradient of the loss function

[0032] Perform iterative processing:

[0033] Where η represents the learning rate, T pi (t+1), T pi (t) the processing time after iteration and the processing time before iteration, respectively;

[0034] When the maximum number of iterations is reached or the change in the loss function is less than the preset change threshold γ, the iteration is stopped;

[0035] Get the processing time value at this time as the optimal processing time for the stage to be optimized;

[0036] The optimized processing time of all independent stages is added together to obtain the optimized total delay time.

[0037] Preferably, the optimizing the processing time of each independent stage based on the sensitivity and the evaluation of the impact on the output sound quality, balancing the total delay of the speech conversion and the final output sound quality, further comprises:

[0038] By optimizing the parameters of the model within each individual stage, processing time is minimized while maintaining sound quality, including:

[0039] Get the parameters of the model and form the model parameter set C = {θ j |j=1, 2, 3..., m}; where θ j represents the jth parameter of the model; m represents the number of model parameters;

[0040] Building a joint optimization model Among them, α′ and β′ represent the delay time weight and the score weight of the model respectively;

[0041] Use Latin hypercube sampling or Bayesian optimization algorithm to explore the model parameter space, collect model parameters, and normalize the collected model parameters to form multiple model parameter combination vectors

[0042] j≥1;

[0043] Build model processing time function Among them, b1, b2, b3, and b4 represent parameter fitting coefficients;

[0044] Constructing model sound quality model Q i (θ)=Q max -c1(θ 1max -θ1)-c2log(θ3); where c1 and c2 represent the parameter attenuation coefficients respectively;

[0045] Update model parameters, including:

[0046] For the objective loss function L′ in the joint optimization model loss Find the gradient,

[0047] Iteratively update the model parameters, i.e.

[0048] When the number of iterations is reached, the model parameters are stopped from being updated, the optimal model parameters are obtained, and the parameters of the model in the independent stage are optimized to reduce the processing time of the corresponding independent stage.

[0049] Preferably, the optimizing the processing time of each independent stage based on the sensitivity and the evaluation of the impact on the output sound quality, balancing the total delay of the speech conversion and the final output sound quality, further comprises:

[0050] Add speaker recognition to the inference phase of the speech conversion model. This feature triggers model parameter optimization only when the target speaker is detected, reducing latency. Specifically, it includes:

[0051] Collect the speaker's voiceprint features and normalize them to form a voiceprint recognition feature vector;

[0052] Import the pre-trained voiceprint verification model and calculate the cosine similarity between the voiceprint recognition feature vector and the pre-stored target speaker's voiceprint feature vector;

[0053] If the cosine similarity is greater than the preset matching threshold, the speaker is determined to be the target speaker, and the joint optimization model is started to optimize the speech conversion model in the speech conversion model inference phase;

[0054] Otherwise, the joint optimization model is not started.

[0055] Preferably, the optimizing the processing time of each independent stage based on the sensitivity and the evaluation of the impact on the output sound quality, balancing the total delay of the speech conversion and the final output sound quality, further comprises:

[0056] Obtain the voiceprint recognition feature vector identified during the speech conversion model inference phase;

[0057] If the biometric feature vector is determined to be the target speaker, the voiceprint recognition feature vector is stored in a buffer area, and the voiceprint recognition feature vector is embedded in the output speech, specifically including:

[0058] After the voice conversion model inference phase, the voiceprint recognition feature vector is added to the voice features converted, that is, X out =X vc +χW, where W represents the voiceprint recognition feature vector, X vc represents the output speech feature vector of the model during the inference phase of the speech conversion model, X out Represents the output speech feature vector of the speech conversion model inference stage; the dimension of W is the same as χ vc The dimensions of are the same; χ represents the embedding strength;

[0059] From X out Extract the voiceprint recognition feature vector and determine whether it is the target speaker. If so, use the fast vocoder LPCNet to synthesize the sound wave signal and output it; otherwise, use the high-fidelity vocoder WaveNet to synthesize the sound wave signal and output it.

[0060] Preferably, the embedding strength is dynamically adapted according to the model parameters to balance the robustness and sound quality of the embedded voiceprint recognition feature vector, including:

[0061] Establish a mapping relationship between χ and model parameters, including:

[0062] Assume that the adjustment formula of embedding strength is Among them, b k , δ represent the attenuation coefficient and robustness compensation of parameter changes on embedding strength respectively; Δθ j represents parameter changes; θ j,new ,θ j,base They represent the adjusted value of the model parameter and the maximum value allowed, respectively.

[0063] A low-delay real-time voice conversion system includes a processor, a memory connected to the processor, and a communication module. The system is used to execute a low-delay real-time voice conversion method.

[0064] A computer-readable storage medium stores a computer program, which is executed by a processor to implement a low-latency real-time speech conversion method and system.

[0065] Beneficial effects of the present invention:

[0066] 1. Based on asynchronous pipeline technology, the present invention divides the algorithm processing process in the speech conversion process into multiple independent stages. Based on the sensitivity of each independent stage to the total delay time, the corresponding independent stage is selected for processing time optimization to reduce the delay time of the algorithm processing process.

[0067] 2. The present invention optimizes the parameters of the model within an independent stage during the processing time optimization process, thereby optimizing the processing time of that stage by optimizing the model parameters. Furthermore, the present invention embeds voiceprint recognition features in the output signal of the speech conversion model inference stage, identifies the voiceprint features, and determines whether the target speaker is present, thereby determining whether the model of the waveform synthesis stage needs to be optimized. During the voiceprint recognition feature embedding process, the embedding strength is linked to the amount of model parameter adjustment, and the embedding strength is adjusted. During model compression, the embedding strength is reduced to prevent the voiceprint recognition feature from affecting the sound quality of the transmitted audio. During model recovery, the embedding strength is increased to enhance the anti-interference ability of the voiceprint recognition feature. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 The present invention provides a flow chart of a low-delay real-time voice conversion method. DETAILED DESCRIPTION

[0069] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are for illustrative purposes only, and those skilled in the art will readily appreciate other obvious variations. The basic principles of the present invention defined in the following description may be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the present invention.

[0070] It is to be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the elements may be multiple, and the term "one" should not be understood as a limitation on the quantity.

[0071] Example 1:

[0072] refer to Figure 1 The present invention provides a low-latency real-time voice conversion method and system, comprising the following steps:

[0073] Step 1: Upon receiving a speech conversion task, the speech processing flow in the speech conversion task is divided into multiple independent stages for parallel processing through an asynchronous pipeline architecture. Specifically, the speech processing flow is split into the recording frame segmentation stage, the feature extraction stage, the speech conversion model inference stage, and the waveform synthesis stage.

[0074] Assign independent threads to each independent stage and use multi-core CPU or GPU to execute speech processing tasks in parallel;

[0075] Each independent thread uses a lock-free queue or ring buffer to transfer data, reducing thread data blocking.

[0076] Step 2: quantify the sensitivity of the time delay of each independent stage to the total delay of speech conversion and evaluate the impact of the time delay of each independent stage on the output sound quality, including the following steps:

[0077] Measure the processing time and total latency of each individual stage;

[0078] Adjust the processing time for each stage individually and remeasure the total latency;

[0079] Calculate the delay sensitivity coefficient for each individual stage And according to the size of the sensitivity coefficient, they are arranged in descending order to form a sensitivity sequence A=(L i |i=1, 2, ..., n);

[0080] Where, ΔT total 、T total , ΔT pi 、T pi They represent the total delay time variation, total delay time, the variation of the processing time of the ith independent stage, and the processing time of the ith independent stage, respectively. n represents the number of independent stages.

[0081] The evaluation of the impact of time delay of each independent stage on the output sound quality includes:

[0082] Quantify the impact of reduced processing time at each individual stage on the quality of the output audio, specifically:

[0083] Establish the mass decay function: Among them, k i represents the attenuation coefficient; Q max,i Indicates the theoretical maximum sound quality of the ith independent stage, that is, the highest MOS score; Q i represents the sound quality of the i-th independent stage;

[0084] The quality of the output audio is tested at different processing times, and k is determined by the linear regression algorithm. i ;

[0085] Arrange the sound quality of each independent stage in ascending order according to the value to form an attenuation coefficient sequence B = (Q1, Q2, ..., Q n ).

[0086] Step 3: Based on the sensitivity and the impact on the output sound quality, the processing time of each independent stage is optimized to balance the total delay of speech conversion and the final output sound quality. This includes the following steps: building a delayed speech optimization model to jointly optimize the total delay and output sound quality;

[0087] The delayed speech optimization model is

[0088] Among them, T target , Q total , Q target Respectively represent the target total delay, output sound quality, and target output sound quality; T min , Q min Represent the processing time threshold and sound quality threshold respectively; α and β represent the delay time weight and score weight respectively;

[0089] According to the arrangement of the sensitivity coefficients in the sensitivity sequence, the corresponding independent stages are selected in turn as the stages to be optimized;

[0090] The speech optimization model is delayed through an iterative optimization algorithm to find the optimal processing time for the stage to be optimized. Specifically:

[0091] The solution of the delay optimization speech optimization model is solved using the gradient descent algorithm, including:

[0092] Get the gradient of the loss function

[0093] Perform iterative processing:

[0094] Where η represents the learning rate, T pi (t+1), T pi(t) the processing time after iteration and the processing time before iteration, respectively;

[0095] When the maximum number of iterations is reached or the change in the loss function is less than the preset change threshold γ, the iteration is stopped;

[0096] Get the processing time value at this time as the optimal processing time for the stage to be optimized;

[0097] The optimized processing time of all independent stages is added together to obtain the optimized total delay time.

[0098] For example, the processing time of the original recording frame stage is 5ms, and the sound quality (MOS value) is 4.8; the processing time of the feature extraction stage is 10ms, and the sound quality (MOS value) is 4.6; the processing time of the speech conversion model thrust stage is 30ms, and the sound quality (MOS value) is 4.5; the processing time of the waveform synthesis stage is 15ms, and the sound quality (MOS value) is 4.7; the target total delay time T target =200ms, Q min =4.0;

[0099] The calculation results show that L1 = 0.5, L2 = 0.8, L3 = 1.5, and L4 = 1.2. Therefore, the sensitivity sequence A = (L3, L4, L2, L1).

[0100] According to the sensitivity sequence, the processing time of the speech conversion model inference stage, waveform synthesis stage, feature extraction stage and recording frame stage is optimized in turn;

[0101] Through an iterative optimization algorithm, the optimal processing time for the speech conversion model inference phase is 20ms, and the processing time for the feature extraction phase is 15ms, while the other two independent phases remain unchanged.

[0102] The inference time of the speech conversion model was reduced from 30ms to 20ms; the total latency was reduced by 1.5×(30-20) / 30=5%, and the sound quality was reduced from 4.5 to 4.2; the feature extraction time was optimized from 10ms to 15ms, the latency increased by 0.8×(15-10) / 10=4%, and the sound quality was improved from 4.6 to 4.7; the total latency was reduced by 1% (total latency became 198ms); the final system output sound quality

[0103] Example 2:

[0104] To complete the tasks of this stage, the independent stage usually requires corresponding models, such as speech conversion models, feature extraction models, and waveform synthesis models.

[0105] Optimizing the processing time of each stage can also be achieved by optimizing the working parameters of the model in each stage. To this end, based on the first embodiment, we propose a technical solution:

[0106] By optimizing the parameters of the model within each individual stage, processing time is minimized while maintaining sound quality, including:

[0107] Get the parameters of the model and form the model parameter set C = {θ j |j=1, 2, 3..., m}; where θ j represents the jth parameter of the model; m represents the number of model parameters;

[0108] Building a joint optimization model Among them, α′ and β′ represent the delay time weight and the score weight of the model respectively;

[0109] Use Latin hypercube sampling or Bayesian optimization algorithm to explore the model parameter space, collect model parameters, and normalize the collected model parameters to form multiple model parameter combination vectors

[0110] j≥1;

[0111] Build model processing time function Among them, b1, b2, b3, and b4 represent parameter fitting coefficients;

[0112] Constructing model sound quality model Q i (θ)=Q max -c1(θ 1max -θ1)-c2log(θ3); where c1 and c2 represent the parameter attenuation coefficients respectively;

[0113] Update model parameters, including:

[0114] For the objective loss function L′ in the joint optimization model loss Find the gradient,

[0115] Iteratively update the model parameters, i.e.

[0116] When the number of iterations is reached, the model parameters are stopped from being updated, the optimal model parameters are obtained, and the parameters of the model in the independent stage are optimized to reduce the processing time of the corresponding independent stage.

[0117] For example, during the speech conversion model inference phase, the Transformer model is optimized.

[0118] The model parameters of the Transformer model include the number of model layers, the number of attention heads, the hidden layer dimension, the model weight quantization bit width, the dynamic batch size, the number of input speech frames, and the pruning rate;

[0119] The model parameters selected in this example are the number of model layers, the number of attention heads, the model weight quantization bit width, and the dynamic batch size. After normalization of the selected model parameters, a model parameter combination vector is constructed.

[0120] Through experimental measurement model, obtain T p3 and Q3, by sampling 100 sets of parameter combinations using Latin hypercube, and combining the measured T p3 and T p3 , fitting the model to process the time function and the sound quality model, and obtaining the parameter fitting coefficient and the parameter attenuation coefficient;

[0121] Constructing the initialization parameter vector

[0122] If the parameters of the baseline Transformer model are known, the normalized composition And T p3 =20ms, sound quality is 4.0;

[0123] The optimization process of the baseline Transformer model is as follows:

[0124] Reduce the number of model layers (corresponding to θ1): from 12 to 8, the processing time becomes 30-4×2.5=20ms, and the sound quality becomes 4.3;

[0125] The model weight quantization bit width (corresponding to θ2) is reduced from 32 to 8, the processing time is reduced to 20×0.6=12ms, and the sound quality is reduced to 4.0.

[0126] Dynamic batch size (corresponding to θ4): from 1 to 4, processing time becomes 12 / 3 = 4ms, and the sound quality is 4.0;

[0127] After the above optimization process, the processing time of the speech conversion model inference phase is reduced to 3ms<20ms.

[0128] Example 3:

[0129] In actual use, in order to protect user privacy or prevent the system from being used by non-target users, we propose a technical solution based on Example 2, namely, to build a two-stage watermark cascade optimization solution in the voice conversion system: in the third stage (voice conversion model inference stage) and the fourth stage (waveform synthesis stage) of the system, respectively, voiceprint features with recognition functions (voiceprint recognition watermarks) are embedded, model parameters are adjusted, and the embedding strength is mapped to the model parameters to coordinate parameter adjustment and embedding strength. The specific solution is as follows:

[0130] Add speaker recognition to the inference phase of the speech conversion model. This feature triggers model parameter optimization only when the target speaker is detected, reducing latency. Specifically, it includes:

[0131] Collect the speaker's voiceprint features and normalize them to form a voiceprint recognition feature vector;

[0132] Import the pre-trained voiceprint verification model and calculate the cosine similarity between the voiceprint recognition feature vector and the pre-stored target speaker's voiceprint feature vector;

[0133] If the cosine similarity is greater than the preset matching threshold, the speaker is determined to be the target speaker, and the joint optimization model is started to optimize the speech conversion model in the speech conversion model inference phase;

[0134] Otherwise, the joint optimization model is not started.

[0135] After starting the joint optimization model, obtain the voiceprint recognition feature vector identified in the speech conversion model inference phase;

[0136] If the biometric feature vector is determined to be the target speaker, the voiceprint recognition feature vector is stored in a buffer area, and the voiceprint recognition feature vector is embedded in the output speech, specifically including:

[0137] After the voice conversion model inference phase, the voiceprint recognition feature vector is added to the voice features converted, that is, X out =X vc +χW, where W represents the voiceprint recognition feature vector, X vc represents the output speech feature vector of the model during the inference phase of the speech conversion model, X out Represents the output speech feature vector of the speech conversion model inference stage; the dimension of W is the same as χ vc The dimensions of are the same; χ represents the embedding strength;

[0138] From X out Extract the voiceprint recognition feature vector and determine whether it is the target speaker. If so, use the fast vocoder LPCNet to synthesize the sound wave signal and output it; otherwise, use the high-fidelity vocoder WaveNet to synthesize the sound wave signal and output it.

[0139] In the process of adjusting the model parameters, the embedding strength is dynamically adapted according to the model parameters to balance the robustness of the embedded voiceprint recognition feature vector and the sound quality. Specifically:

[0140] Establishing the mapping relationship between χ and model parameters includes the following steps:

[0141] Assume that the adjustment formula of embedding strength is Among them, b k , δ represent the attenuation coefficient and robustness compensation of parameter changes on embedding strength respectively; Δθ j represents parameter changes; θ j,new ,θ j,base They represent the adjusted value of the model parameter and the maximum value allowed, respectively.

[0142] For example, let's take the optimization of the voice conversion model in the voice conversion model inference phase as an example. The parameters of the voice conversion model (Transformer) in the initial state are:

[0143] Let parameter θ1 be the model pruning rate p mod =0, parameter θ2 is the model weight quantization bit width 32bit; original embedding strength χ0 = 0.01, MOS = 4.5, detection rate 95%;

[0144] By compressing the model, processing time is reduced; after model compression, the pruning rate is 0.3 and the model weight quantization bit width is 8 bits;

[0145] Then, the embedding strength

[0146] If the detection rate requirement is 90%, then adjust the robustness compensation to 1.5, and the embedding strength χ = 0.00234;

[0147] After actual measurement and adjustment, the output sound quality of the model becomes 4.2, which meets the requirements.

[0148] The present invention also provides a low-delay real-time voice conversion system, comprising a processor, a memory connected to the processor, and a communication module, wherein the system is used to execute the low-delay real-time voice conversion method.

[0149] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the low-latency real-time speech conversion method.

[0150] The embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. The embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable medium mentioned above in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wire segments, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, electrical wire, optical fiber cable, RF, etc., or any suitable combination thereof.

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or portion of code that contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as combinations of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or operations, or can be implemented using a combination of dedicated hardware and computer instructions.

[0152] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments. Without departing from the principles, the implementation methods of the present invention may be subject to any changes or modifications.

Claims

1. A low-latency real-time voice conversion method, characterized in that: The method comprises: When receiving a voice conversion task, the voice processing flow in the voice conversion task is divided into multiple independent stages and processed in parallel through an asynchronous pipeline architecture; Quantify the sensitivity of the time delay of each independent stage to the total delay of speech conversion, and evaluate the impact of the time delay of each independent stage on the output sound quality; Based on sensitivity and an assessment of the impact on output sound quality, the processing time of each individual stage is optimized to balance the overall voice conversion latency and the final output sound quality.

2. A low-delay real-time voice conversion method according to claim 1, characterized in that The asynchronous pipeline architecture is used to divide the voice data into multiple independent stages for parallel processing, including: The speech processing process is divided into the recording frame stage, feature extraction stage, speech conversion model inference stage and waveform synthesis stage; Assign independent threads to each independent stage and use multi-core CPU or GPU to execute speech processing tasks in parallel; Each independent thread uses a lock-free queue or ring buffer to transfer data, reducing thread data blocking.

3. A low-delay real-time voice conversion method according to claim 2, characterized in that: The method of quantifying the sensitivity of the time delay of each individual stage to the overall delay of speech conversion comprises: Measure the processing time and total latency of each individual stage; Adjust the processing time for each stage individually and remeasure the total latency; Calculate the delay sensitivity coefficient for each individual stage And according to the size of the sensitivity coefficient, they are arranged in descending order to form a sensitivity sequence A=(L i |i=1, 2, ..., n); Where, ΔT total 、T total , ΔT pi 、T pi They represent the total delay time variation, total delay time, the variation of the processing time of the ith independent stage, and the processing time of the ith independent stage, respectively. n represents the number of independent stages. The evaluation of the impact of time delay of each independent stage on the output sound quality includes: Quantify the impact of reduced processing time at each individual stage on the quality of the output audio, specifically: Establish the mass decay function: Among them, k i represents the attenuation coefficient; Q max,i Indicates the theoretical maximum sound quality of the ith independent stage, that is, the highest MOS score; Q i represents the sound quality of the i-th independent stage; The quality of the output audio is tested at different processing times, and k is determined by the linear regression algorithm. i ; Arrange the sound quality of each independent stage in ascending order according to the value to form an attenuation coefficient sequence B = (Q1, Q2, ..., Q n ).

4. A low-delay real-time voice conversion method according to claim 3, characterized in that: Based on the sensitivity and the impact on the output sound quality, the processing time of each independent stage is optimized to balance the total voice conversion delay and the final output sound quality, including: Build a delayed speech optimization model to jointly optimize total delay and output sound quality; The delayed speech optimization model is Among them, T target , Q total , Q target Respectively represent the target total delay, output sound quality, and target output sound quality; T min , Q min Represent the processing time threshold and sound quality threshold respectively; α and β represent the delay time weight and score weight respectively; According to the arrangement of the sensitivity coefficients in the sensitivity sequence, the corresponding independent stages are selected in turn as the stages to be optimized; The speech optimization model is delayed through an iterative optimization algorithm to find the optimal processing time for the stage to be optimized. Specifically: The solution of the delay optimization speech optimization model is solved using the gradient descent algorithm, including: Get the gradient of the loss function Perform iterative processing: Where η represents the learning rate, T pi (t+1), T pi (t) the processing time after iteration and the processing time before iteration, respectively; When the maximum number of iterations is reached or the change in the loss function is less than the preset change threshold γ, the iteration is stopped; Get the processing time value at this time as the optimal processing time for the stage to be optimized; The optimized processing time of all independent stages is added together to obtain the optimized total delay time.

5. A low-delay real-time voice conversion method according to claim 4, characterized in that: The process of optimizing the processing time of each independent stage based on sensitivity and impact on output sound quality, balancing the total voice conversion delay and final output sound quality, also includes: By optimizing the parameters of the model in each independent stage, processing time is minimized while maintaining sound quality, including: Get the parameters of the model and form the model parameter set C = {θ j |j=1, 2, 3..., m}; where θ j represents the jth parameter of the model; m represents the number of model parameters; Building a joint optimization model Among them, α′ and β′ represent the delay time weight and the score weight of the model respectively; Use Latin hypercube sampling or Bayesian optimization algorithm to explore the model parameter space, collect model parameters, and normalize the collected model parameters to form multiple model parameter combination vectors Build model processing time function Among them, b1, b2, b3, and b4 represent parameter fitting coefficients; Constructing model sound quality model Q i (θ)=Q max -c1(θ 1max -θ1)-c2log(θ3); where c1 and c2 represent the parameter attenuation coefficients respectively; Update model parameters, including: For the objective loss function L′ in the joint optimization model loss Find the gradient, Iteratively update the model parameters, i.e. θ j (t+1)=θ j (t)-η·▽ θj L′ loss (θ(t)); When the number of iterations is reached, the model parameters are stopped from being updated, the optimal model parameters are obtained, and the parameters of the model in the independent stage are optimized to reduce the processing time of the corresponding independent stage.

6. A low-delay real-time voice conversion method according to claim 5, characterized in that: The process of optimizing the processing time of each independent stage based on sensitivity and impact on output sound quality, balancing the total voice conversion delay and final output sound quality, also includes: Add speaker recognition to the inference phase of the speech conversion model. This feature triggers model parameter optimization only when the target speaker is detected, reducing latency. Specifically, it includes: Collect the speaker's voiceprint features and normalize them to form a voiceprint recognition feature vector; Import the pre-trained voiceprint verification model and calculate the cosine similarity between the voiceprint recognition feature vector and the pre-stored target speaker's voiceprint feature vector; If the cosine similarity is greater than the preset matching threshold, the speaker is determined to be the target speaker, and the joint optimization model is started to optimize the speech conversion model in the speech conversion model inference phase; Otherwise, the joint optimization model is not started.

7. A low-delay real-time voice conversion method according to claim 6, characterized in that: The process of optimizing the processing time of each independent stage based on sensitivity and impact on output sound quality, balancing the total voice conversion delay and final output sound quality, also includes: Obtain the voiceprint recognition feature vector identified during the speech conversion model inference phase; If the biometric feature vector is determined to be the target speaker, the voiceprint recognition feature vector is stored in a buffer area, and the voiceprint recognition feature vector is embedded in the output speech, specifically including: After the voice conversion model inference phase, the voiceprint recognition feature vector is added to the voice features converted, that is, X out =X vc +χW, where W represents the voiceprint recognition feature vector, X vc represents the output speech feature vector of the model during the inference phase of the speech conversion model, X out Represents the output speech feature vector of the speech conversion model inference stage; the dimension of W is the same as χ vc The dimensions of are the same; χ represents the embedding strength; From X out Extract the voiceprint recognition feature vector and determine whether it is the target speaker. If so, use the fast vocoder LPCNet to synthesize the sound wave signal and output it; otherwise, use the high-fidelity vocoder WaveNet to synthesize the sound wave signal and output it.

8. A low-delay real-time voice conversion method according to claim 7, characterized in that: It also includes dynamic adaptation of embedding strength based on model parameters to balance the robustness of the embedded voiceprint recognition feature vector and sound quality, including: Establish a mapping relationship between χ and model parameters, including: Assume that the adjustment formula of embedding strength is Among them, b k , δ represent the attenuation coefficient and robustness compensation of parameter changes on embedding strength respectively; Δθ j represents parameter changes; θ j,new ,θ j,base They represent the adjusted value of the model parameter and the maximum value allowed, respectively.

9. A low-latency real-time speech conversion system, comprising a processor, a memory connected to the processor, and a communication module, characterized in that: The system is used to execute the low-latency real-time speech conversion method described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the low-latency real-time speech conversion method according to any one of claims 1 to 8.

Images