Zero sample speech synthesis method and device, computer equipment and storage medium

Abstract

The invention relates to a zero-sample speech synthesis method and device, computer equipment, a storage medium and a program product, and the method comprises the steps: obtaining a target coding feature according to a reference speech and a target text; inputting the target coding features into a stream matching model to obtain a conditional velocity field and an unconditional velocity field; inputting the target coding feature into a prior model to obtain a prior speech feature; obtaining a prior generation flow field according to the prior voice features and the standard Gaussian noise; calculating a KL divergence value between the priori generated flow field and a preset real generated flow field, and taking a moment when the KL divergence value is smaller than or equal to a preset KL divergence threshold as an initial moment of the priori generated flow field; fusing the conditional velocity field and the unconditional velocity field to obtain a fused velocity field; and inputting the fusion velocity field from the initial moment to the target moment and the priori generated flow field into an ordinary differential equation solver to obtain the target speech features, thereby improving the speech quality of the synthesized speech.

Description

Technical Field

[0001] This application relates to the field of speech signal processing technology, and in particular to a zero-sample speech synthesis method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Technology

[0002] Zero-shot speech synthesis based on the flow matching model is the mainstream technology in the current speech synthesis field. Its core is to achieve a fast mapping of "random noise → target speech" by relying on the conditional flow matching (CFM) framework and bidirectional contextual speech filling task, and solving non-autoregressive ordinary differential equations (ODE).

[0003] However, traditional zero-shot speech synthesis methods based on stream matching models produce synthesized speech with low quality. Summary of the Invention

[0004] Therefore, it is necessary to provide a zero-sample speech synthesis method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the speech quality of synthesized speech in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a zero-shot speech synthesis method, the method comprising:

[0006] Obtain reference speech and target text; based on the reference speech and target text, obtain target encoding features; the target text includes the reference text corresponding to the reference speech and the text corresponding to the synthesized speech;

[0007] The target encoded features are input into the stream matching model to obtain the conditional velocity field and the unconditional velocity field. The conditional velocity field represents that the rate of change of speech features depends on the reference speech, while the unconditional velocity field represents that the rate of change of speech features does not depend on the reference speech.

[0008] The target encoded features are input into the prior model to obtain prior speech features; based on the prior speech features and standard Gaussian noise, the prior generated flow field is obtained.

[0009] Calculate the KL divergence value between the prior generated flow field and the preset real generated flow field, and take the time when the KL divergence value is less than or equal to the preset KL divergence threshold as the initial time of the prior generated flow field.

[0010] The conditional velocity field and the unconditional velocity field are fused to obtain the fused velocity field; the fused velocity field from the initial time to the target time and the prior generated flow field are input into the ordinary differential equation solver to obtain the target speech features; the target speech features are used to generate synthesized speech.

[0011] In one embodiment, the conditional velocity field and the unconditional velocity field are fused to obtain a fused velocity field, including:

[0012] Generate an optimized velocity field scale based on the conditional and unconditional velocity fields.

[0013] Based on the velocity field optimization scale, the conditional velocity field and the unconditional velocity field are fused to obtain the fused velocity field.

[0014] In one embodiment, generating a velocity field optimization scale based on the conditional velocity field and the unconditional velocity field includes:

[0015] Calculate the dot product of the conditional velocity field and the unconditional velocity field, and obtain the dot product result;

[0016] The summation result is obtained by summing the square of the norm of the unconditional velocity field with the preset perturbation parameters.

[0017] The ratio of the dot product result to the summation result is used to obtain the velocity field optimization scale.

[0018] In one embodiment, the conditional velocity field and the unconditional velocity field are fused according to a velocity field optimization scale to obtain a fused velocity field, including:

[0019] Calculate the product of the optimized velocity field scale and the unconditional velocity field to obtain the product result;

[0020] The product result and the conditional velocity field are weighted and summed to obtain the fused velocity field.

[0021] In one embodiment, the prior generated flow field is obtained based on prior speech features and standard Gaussian noise, including:

[0022] For each time point, the current time point is multiplied by the prior speech features to obtain the first product result;

[0023] Multiply the difference between the target time and the current time by the standard Gaussian noise to obtain the second product result;

[0024] Add the first product result to the second product result to obtain the prior generated flow field.

[0025] In one embodiment, target encoded features are obtained based on reference speech and target text, including:

[0026] The reference speech is preprocessed to obtain its features; the target text is preprocessed to obtain its phoneme sequence.

[0027] The reference speech features are input into a linear layer to obtain the reference speech coding features; the phoneme sequence is input into a linear layer to obtain the target text coding features.

[0028] The reference speech coding features are concatenated with the target text coding features to obtain the concatenated coding features;

[0029] The concatenated coded features are input into the conditional feature encoder to obtain the target coded features.

[0030] Secondly, this application also provides a zero-sample speech synthesis device, the device comprising:

[0031] The target coding feature acquisition module is used to acquire reference speech and target text; and to obtain target coding features based on the reference speech and target text; the target text includes the reference text corresponding to the reference speech and the text corresponding to the synthesized speech.

[0032] The velocity field acquisition module is used to input the target encoded features into the stream matching model to obtain the conditional velocity field and the unconditional velocity field; the conditional velocity field represents that the speed of speech feature change depends on the reference speech, while the unconditional velocity field represents that the speed of speech feature change does not depend on the reference speech.

[0033] The prior generation flow field acquisition module is used to input the target encoded features into the prior model to obtain prior speech features; and to obtain the prior generated flow field based on the prior speech features and standard Gaussian noise.

[0034] The initial time determination module is used to calculate the KL divergence value between the prior generated flow field and the preset real generated flow field, and the time corresponding to the KL divergence value being less than or equal to the preset KL divergence threshold is taken as the initial time of the prior generated flow field.

[0035] The speech synthesis module is used to fuse the conditional velocity field and the unconditional velocity field to obtain the fused velocity field; the fused velocity field from the initial time to the target time and the prior generated flow field are input to the ordinary differential equation solver to obtain the target speech features; the target speech features are used to generate synthesized speech.

[0036] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps of the first aspect.

[0037] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method steps of the first aspect.

[0038] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the method steps of the first aspect.

[0039] The aforementioned zero-sample speech synthesis method, apparatus, computer equipment, computer-readable storage medium, and computer program product acquire reference speech and target text; obtain target encoded features based on the reference speech and target text; the target text includes reference text corresponding to the reference speech and text corresponding to the synthesized speech; input the target encoded features into a flow matching model to obtain a conditional velocity field and an unconditional velocity field; the conditional velocity field represents that the rate of change of speech features depends on the reference speech, and the unconditional velocity field represents that the rate of change of speech features does not depend on the reference speech; input the target encoded features into a priori model to obtain prior speech features; obtain a priori generated flow field based on the prior speech features and standard Gaussian noise; calculate the KL divergence value between the prior generated flow field and a preset real generated flow field, and take the time when the KL divergence value is less than or equal to a preset KL divergence threshold as the initial time of the prior generated flow field; fuse the conditional velocity field and the unconditional velocity field to obtain a fused velocity field; input the fused velocity field from the initial time to the target time and the prior generated flow field into an ordinary differential equation solver to obtain target speech features; the target speech features are used to generate synthesized speech. As can be seen from the above, this application calculates the KL divergence value between the prior generated flow field and the preset real generated flow field, and takes the time when the KL divergence value is less than or equal to the preset KL divergence threshold as the initial time of the prior generated flow field, ensuring that the prior generated flow field is sufficiently close to the real generated flow field at the initial time. Using the prior generated flow field at the initial time as the inference starting point of the ordinary differential equation solver, and skipping the inaccurate prediction of the initial generated flow field, can effectively avoid early errors, thereby improving the speech quality of the finally generated synthesized speech. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a diagram illustrating the application environment of a zero-sample speech synthesis method in one embodiment.

[0042] Figure 2 This is a flowchart illustrating a zero-sample speech synthesis method in one embodiment;

[0043] Figure 3 This is a schematic diagram of the process for obtaining the fused velocity field in one embodiment;

[0044] Figure 4 This is a structural block diagram of a zero-sample speech synthesis device in one embodiment;

[0045] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0048] The zero-sample speech synthesis method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server. Terminal 102 acquires reference speech and target text; based on the reference speech and target text, it obtains target encoding features; the target text includes the reference text corresponding to the reference speech and the text corresponding to the synthesized speech; the target encoding features are input into a flow matching model to obtain a conditional velocity field and an unconditional velocity field; the conditional velocity field represents that the rate of change of speech features depends on the reference speech, and the unconditional velocity field represents that the rate of change of speech features does not depend on the reference speech; the target encoding features are input into a priori model to obtain priori speech features; based on the priori speech features and standard Gaussian noise, a priori generated flow field is obtained; the KL divergence value between the priori generated flow field and the preset real generated flow field is calculated, and the time when the KL divergence value is less than or equal to the preset KL divergence threshold is taken as the initial time of the priori generated flow field; the conditional velocity field and the unconditional velocity field are fused to obtain a fused velocity field; the fused velocity field from the initial time to the target time and the priori generated flow field are input into an ordinary differential equation solver to obtain target speech features; the target speech features are used to generate synthesized speech. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and projection equipment. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0049] In one embodiment, such as Figure 2 As shown, a zero-sample speech synthesis method is provided. This embodiment applies this method to... Figure 1 Taking terminal 102 as an example, the method includes the following steps:

[0050] Step S210: Obtain reference speech and target text; obtain target encoding features based on reference speech and target text; target text includes reference text corresponding to reference speech and text corresponding to synthesized speech.

[0051] The reference speech refers to an audio sample provided by the user that contains a specific timbre and speaking style. For example, the reference speech could be 1-3 seconds of audio from an unknown speaker.

[0052] Here, reference text refers to text information that matches the reference speech. For example, if the reference audio is the speech information of "abcd" read by the target pronunciation object, then the reference text can be the textual information of "abcd".

[0053] Here, the text corresponding to the synthesized speech refers to the text content that the user wants the synthesized speech to be. For example, if the user wants the synthesized speech to read aloud a message of "efgh", then the text corresponding to the synthesized speech can be the text message "efgh".

[0054] In this embodiment of the application, feature information can be extracted from the reference speech and the target text respectively, and the extracted feature information can be fused and encoded to obtain the target encoded features.

[0055] Step S220: Input the target encoded features into the stream matching model to obtain the conditional velocity field and the unconditional velocity field; the conditional velocity field represents that the speed of speech feature change depends on the reference speech, and the unconditional velocity field represents that the speed of speech feature change does not depend on the reference speech.

[0056] The conditional velocity field refers to the process of generating speech by incorporating not only the information from the target text but also the feature information from the reference speech as a constraint. The conditional velocity field describes the dynamic process of speech features changing over time under these constraints. By introducing the conditional velocity field, speech that is stylistically more similar to the reference speech can be generated, thereby improving the naturalness and realism of the synthesized speech.

[0057] The unconditional velocity field refers to a process in which only the information of the target text is considered during speech generation, without introducing any additional reference speech features. The unconditional velocity field describes the dynamic process of speech features changing over time based solely on the target text information.

[0058] In this embodiment of the application, the flow matching model is trained so that it outputs conditional velocity fields and unconditional velocity fields based on target encoded features.

[0059] Step S230: Input the target encoded features into the prior model to obtain the prior speech features; based on the prior speech features and standard Gaussian noise, obtain the prior generated flow field.

[0060] In this embodiment, the prior model adopts the same network architecture as the stream matching model, but directly relies on conditional learning to map the target Mel spectrum in a single step. Specifically, in one forward propagation process, the target encoded features are directly mapped to a coarse estimate of the target Mel spectrum to obtain coarse Mel spectrum features (prior speech features).

[0061] The prior speech features and standard Gaussian noise are superimposed to obtain the prior generated flow field. Specifically, the prior generated flow field is a speech feature that changes dynamically over time. At the initial time (t=0), the prior generated flow field is standard Gaussian noise. At the target time (t=1), the prior generated flow field is the prior speech feature.

[0062] Step S240: Calculate the KL divergence value between the prior generated flow field and the preset real generated flow field, and take the time when the KL divergence value is less than or equal to the preset KL divergence threshold as the initial time of the prior generated flow field.

[0063] The preset real-generated flow field is obtained by superimposing real Mel-frequency spectral features and standard Gaussian noise. Specifically, the real Mel-frequency spectral features are extracted from real speech signals.

[0064] The KL divergence value can be used to represent the distance between two distributions.

[0065] The preset KL divergence threshold can be set according to actual needs.

[0066] In this embodiment of the application, the initial time of the priori generation of the flow field The solution formula is as follows:

[0067]

[0068] in, This represents the preset real generated flow field. Represents the true characteristics of the Mel spectrum. Indicates the prior generated flow field. Represents prior speech features. Represents the KL divergence value. Let y represent the expected value, and y represent the test set. This indicates the preset KL divergence threshold.

[0069] In this embodiment of the application, by Replace the starting point of inference with standard Gaussian noise to search. The value of t. When t < 0.001, the generated speech changes almost nothing. Therefore, Set as KL divergence value between the noise and standard Gaussian noise.

[0070] Step S250: The conditional velocity field and the unconditional velocity field are fused to obtain the fused velocity field; the fused velocity field from the initial time to the target time and the prior generated flow field are input into the ordinary differential equation solver to obtain the target speech features; the target speech features are used to generate synthesized speech.

[0071] In this embodiment, the conditional velocity field and the unconditional velocity field can be weighted and summed to obtain the fused velocity field. The ordinary differential equation solver is based on the initial time t= The fused velocity field and the prior generated flow field at the target time t=1 are used for iterative inference at multiple time steps to obtain the target speech features at the target time. The target speech features are then input into a vocoder to obtain synthesized speech.

[0072] The aforementioned zero-shot speech synthesis method involves: acquiring reference speech and target text; obtaining target encoding features based on the reference speech and target text; the target text including the reference text corresponding to the reference speech and the text corresponding to the synthesized speech; inputting the target encoding features into a flow matching model to obtain a conditional velocity field and an unconditional velocity field; the conditional velocity field representing that the rate of change of speech features depends on the reference speech, and the unconditional velocity field representing that the rate of change of speech features does not depend on the reference speech; inputting the target encoding features into a priori model to obtain prior speech features; obtaining a priori generated flow field based on the prior speech features and standard Gaussian noise; calculating the KL divergence value between the prior generated flow field and a preset real generated flow field, and taking the time when the KL divergence value is less than or equal to a preset KL divergence threshold as the initial time of the prior generated flow field; fusing the conditional velocity field and the unconditional velocity field to obtain a fused velocity field; inputting the fused velocity field from the initial time to the target time and the prior generated flow field into an ordinary differential equation solver to obtain target speech features; and using the target speech features to generate synthesized speech. As can be seen from the above, this application calculates the KL divergence value between the prior generated flow field and the preset real generated flow field, and takes the time when the KL divergence value is less than or equal to the preset KL divergence threshold as the initial time of the prior generated flow field, ensuring that the prior generated flow field is sufficiently close to the real generated flow field at the initial time. Using the prior generated flow field at the initial time as the inference starting point of the ordinary differential equation solver, and skipping the inaccurate prediction of the initial generated flow field, can effectively avoid early errors, thereby improving the speech quality of the finally generated synthesized speech.

[0073] In one embodiment, obtaining target encoded features based on reference speech and target text includes:

[0074] Step S212: Preprocess the reference speech to obtain reference speech features; preprocess the target text to obtain a phoneme sequence.

[0075] In this embodiment, silence segments are removed and amplitude is normalized (in the [-1,1] interval) on the reference speech. The 80-dimensional spectrum is extracted by short-time Fourier transform (STFT, frame length 20ms, frame shift 10ms) to obtain the features of the reference speech.

[0076] Mapping text to phoneme sequences (such as the English word "apple" → / ˈæpl / ) ensures fine-grained alignment between semantics and speech.

[0077] Step S214: Input the reference speech features into the linear layer to obtain the reference speech coding features; input the phoneme sequence into the linear layer to obtain the target text coding features.

[0078] In this embodiment of the application, a linear layer is used to perform linear transformations on the reference speech features and phoneme sequences respectively to obtain the reference speech coding features and the target text coding features.

[0079] Step S216: Concatenate the reference speech coding features with the target text coding features to obtain concatenated coding features;

[0080] Step S218: Input the concatenated coded features into the conditional feature encoder to obtain the target coded features.

[0081] In this embodiment, the conditional feature encoder processes the concatenated coded features and outputs the target coded features. Specifically, the dimension of the target coded features is B×T×512, where B is the batch size and T is the temporal length of the Mel spectrum, ensuring the integrity of long sequence dependency modeling.

[0082] In one embodiment, obtaining the prior generated flow field based on prior speech features and standard Gaussian noise includes:

[0083] Step S232: For each time point, multiply the current time point with the prior speech features to obtain the first product result;

[0084] Step S234: Multiply the difference between the target time and the current time by the standard Gaussian noise to obtain the second product result;

[0085] Step S236: Add the first product result to the second product result to obtain the prior generated flow field.

[0086] In this embodiment of the application, the expression for the priori generated flow field is:

[0087]

[0088] in, This represents standard Gaussian noise.

[0089] Correspondingly, the preset expression for the actual generated flow field is:

[0090]

[0091] In one embodiment, such as Figure 3As shown, the conditional velocity field and the unconditional velocity field are fused to obtain the fused velocity field, including:

[0092] Step S310: Generate the velocity field optimization scale based on the conditional velocity field and the unconditional velocity field.

[0093] In this embodiment, considering that traditional classifier-free guidance (CFG) methods use fixed-scale guidance, which cannot balance the conditional and unconditional velocity fields, and that when the directions of the conditional and unconditional velocity fields differ significantly, the trajectory of the generated flow field will deviate from the target timbre distribution, reducing the similarity of the timbre clones, an optimized velocity field scale is calculated for each time step based on the conditional and unconditional velocity fields.

[0094] Step S320: Based on the velocity field optimization scale, the conditional velocity field and the unconditional velocity field are fused to obtain the fused velocity field.

[0095] In this embodiment, a velocity field optimization scale is used to balance the weights of the conditional velocity field and the unconditional velocity field during inference, thereby obtaining a fused velocity field.

[0096] The embodiments of this application obtain a fused velocity field by optimizing the velocity field scale, which can prevent the generated flow field from deviating too much from the real trajectory during the inference process of the subsequent ordinary differential equation solver, thereby improving the timbre similarity of the synthesized speech.

[0097] In one embodiment, generating a velocity field optimization scale based on the conditional and unconditional velocity fields includes:

[0098] Step S312: Calculate the dot product of the conditional velocity field and the unconditional velocity field to obtain the dot product result;

[0099] Step S314: Summate the squared norm of the unconditional velocity field with the preset perturbation parameters to obtain the summation result;

[0100] Step S316: Calculate the ratio of the dot product result to the summation result to obtain the velocity field optimization scale.

[0101] In this embodiment of the application, the velocity field optimization scale The expression is as follows:

[0102]

[0103] in, Represents the conditional velocity field. Represents the unconditional velocity field. This indicates the preset disturbance parameters.

[0104] In one embodiment, the conditional velocity field and the unconditional velocity field are fused according to the velocity field optimization scale to obtain a fused velocity field, including:

[0105] Step S322: Calculate the product of the optimized velocity field scale and the unconditional velocity field to obtain the product result;

[0106] Step S324: The product result and the conditional velocity field are weighted and summed to obtain the fused velocity field.

[0107] In this embodiment of the application, the fused velocity field The expression is as follows:

[0108]

[0109] in, Indicates the weight.

[0110] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0111] In one exemplary embodiment, such as Figure 4 As shown, a zero-shot speech synthesis device is provided, the device comprising:

[0112] The target coding feature acquisition module 410 is used to acquire reference speech and target text; and to acquire target coding features based on the reference speech and target text; the target text includes the reference text corresponding to the reference speech and the text corresponding to the synthesized speech.

[0113] The velocity field acquisition module 420 is used to input the target encoded features into the stream matching model to obtain the conditional velocity field and the unconditional velocity field; the conditional velocity field represents that the speed of speech feature change depends on the reference speech, and the unconditional velocity field represents that the speed of speech feature change does not depend on the reference speech.

[0114] The prior generation flow field acquisition module 430 is used to input the target encoded features into the prior model to obtain prior speech features; and to obtain the prior generated flow field based on the prior speech features and standard Gaussian noise.

[0115] The initial time determination module 440 is used to calculate the KL divergence value between the prior generated flow field and the preset real generated flow field, and to take the time when the KL divergence value is less than or equal to the preset KL divergence threshold as the initial time of the prior generated flow field.

[0116] The speech synthesis module 450 is used to fuse the conditional velocity field and the unconditional velocity field to obtain the fused velocity field; the fused velocity field from the initial time to the target time and the prior generated flow field are input to the ordinary differential equation solver to obtain the target speech features; the target speech features are used to generate synthesized speech.

[0117] In one embodiment, the conditional velocity field and the unconditional velocity field are fused to obtain a fused velocity field, including:

[0118] Generate an optimized velocity field scale based on the conditional and unconditional velocity fields.

[0119] Based on the velocity field optimization scale, the conditional velocity field and the unconditional velocity field are fused to obtain the fused velocity field.

[0120] In one embodiment, generating a velocity field optimization scale based on the conditional velocity field and the unconditional velocity field includes:

[0121] Calculate the dot product of the conditional velocity field and the unconditional velocity field, and obtain the dot product result;

[0122] The summation result is obtained by summing the square of the norm of the unconditional velocity field with the preset perturbation parameters.

[0123] The ratio of the dot product result to the summation result is used to obtain the velocity field optimization scale.

[0124] In one embodiment, the conditional velocity field and the unconditional velocity field are fused according to a velocity field optimization scale to obtain a fused velocity field, including:

[0125] Calculate the product of the optimized velocity field scale and the unconditional velocity field to obtain the product result;

[0126] The product result and the conditional velocity field are weighted and summed to obtain the fused velocity field.

[0127] In one embodiment, the prior generated flow field is obtained based on prior speech features and standard Gaussian noise, including:

[0128] For each time point, the current time point is multiplied by the prior speech features to obtain the first product result;

[0129] Multiply the difference between the target time and the current time by the standard Gaussian noise to obtain the second product result;

[0130] Add the first product result to the second product result to obtain the prior generated flow field.

[0131] In one embodiment, target encoded features are obtained based on reference speech and target text, including:

[0132] The reference speech is preprocessed to obtain its features; the target text is preprocessed to obtain its phoneme sequence.

[0133] The reference speech features are input into a linear layer to obtain the reference speech coding features; the phoneme sequence is input into a linear layer to obtain the target text coding features.

[0134] The reference speech coding features are concatenated with the target text coding features to obtain the concatenated coding features;

[0135] The concatenated coded features are input into the conditional feature encoder to obtain the target coded features.

[0136] Each module in the aforementioned zero-sample speech synthesis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0137] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows. Figure 5 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores speech synthesis data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a zero-sample speech synthesis method.

[0138] Those skilled in the art will understand that Figure 5 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the aforementioned zero-sample speech synthesis method. The steps of the zero-sample speech synthesis method described here may be steps from the zero-sample speech synthesis methods of the various embodiments described above.

[0139] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, causes the processor to perform the steps of the zero-shot speech synthesis method described above. The steps of the zero-shot speech synthesis method described here may be steps from the zero-shot speech synthesis methods of the various embodiments described above.

[0140] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, causes the processor to perform the steps of the zero-shot speech synthesis method described above. The steps of the zero-shot speech synthesis method described here may be steps from the zero-shot speech synthesis methods of the various embodiments described above.

[0141] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0142] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0143] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0144] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A zero-shot speech synthesis method, characterized in that, The method includes: Obtain reference speech and target text; obtain target encoding features based on the reference speech and target text; the target text includes reference text corresponding to the reference speech and text corresponding to the synthesized speech; The target encoded features are input into the stream matching model to obtain a conditional velocity field and an unconditional velocity field; the conditional velocity field represents that the rate of change of speech features depends on the reference speech, and the unconditional velocity field represents that the rate of change of speech features does not depend on the reference speech. The target encoded features are input into the prior model to obtain prior speech features; based on the prior speech features and standard Gaussian noise, the prior generated flow field is obtained. Calculate the KL divergence value between the prior generated flow field and the preset actual generated flow field, and take the time when the KL divergence value is less than or equal to the preset KL divergence threshold as the initial time of the prior generated flow field. The conditional velocity field and the unconditional velocity field are fused to obtain a fused velocity field; the fused velocity field from the initial time to the target time and the prior generated flow field are input into an ordinary differential equation solver to obtain target speech features; the target speech features are used to generate the synthesized speech.

2. The method according to claim 1, characterized in that, The process of fusing the conditional velocity field and the unconditional velocity field to obtain a fused velocity field includes: Based on the conditional velocity field and the unconditional velocity field, generate an optimized velocity field scale; Based on the velocity field optimization scale, the conditional velocity field and the unconditional velocity field are fused to obtain the fused velocity field.

3. The method according to claim 2, characterized in that, The step of generating a velocity field optimization scale based on the conditional velocity field and the unconditional velocity field includes: Calculate the dot product of the conditional velocity field and the unconditional velocity field to obtain the dot product result; The squared norm of the unconditional velocity field is summed with the preset perturbation parameters to obtain the summation result; The ratio of the dot product result to the summation result is calculated to obtain the velocity field optimization scale.

4. The method according to claim 2, characterized in that, The step of fusing the conditional velocity field and the unconditional velocity field according to the velocity field optimization scale to obtain the fused velocity field includes: Calculate the product of the velocity field optimization scale and the unconditional velocity field to obtain the product result; The product result is weighted and summed with the conditional velocity field to obtain the fused velocity field.

5. The method according to any one of claims 1 to 4, characterized in that, The step of obtaining the prior generated flow field based on the prior speech features and standard Gaussian noise includes: For each time point, the current time point is multiplied by the prior speech features to obtain the first product result; The difference between the target time and the current time is multiplied by the standard Gaussian noise to obtain the second product result; The first product result is added to the second product result to obtain the prior generated flow field.

6. The method according to any one of claims 1 to 4, characterized in that, The step of obtaining target encoding features based on the reference speech and the target text includes: The reference speech is preprocessed to obtain reference speech features; the target text is preprocessed to obtain a phoneme sequence. The reference speech features are input into a linear layer to obtain reference speech coding features; the phoneme sequence is input into the linear layer to obtain target text coding features; The reference speech coding features are concatenated with the target text coding features to obtain concatenated coding features; The concatenated encoding features are input into the conditional feature encoder to obtain the target encoding features.

7. A zero-sample speech synthesis device, characterized in that, The device includes: A target coding feature acquisition module is used to acquire reference speech and target text; and to acquire target coding features based on the reference speech and the target text; the target text includes reference text corresponding to the reference speech and text corresponding to the synthesized speech. The velocity field acquisition module is used to input the target encoded features into the stream matching model to obtain a conditional velocity field and an unconditional velocity field; the conditional velocity field represents that the speed of speech feature change depends on the reference speech, and the unconditional velocity field represents that the speed of speech feature change does not depend on the reference speech. The prior generation flow field acquisition module is used to input the target encoded features into the prior model to obtain prior speech features; and to obtain the prior generated flow field based on the prior speech features and standard Gaussian noise. The initial time determination module is used to calculate the KL divergence value between the prior generated flow field and the preset real generated flow field, and to take the time when the KL divergence value is less than or equal to the preset KL divergence threshold as the initial time of the prior generated flow field. A speech synthesis module is used to fuse the conditional velocity field and the unconditional velocity field to obtain a fused velocity field; input the fused velocity field from the initial time to the target time and the prior generated flow field to an ordinary differential equation solver to obtain target speech features; the target speech features are used to generate the synthesized speech.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images