Speech synthesis method and device with conditional matching stream, equipment and medium

Abstract

The invention relates to artificial intelligence, and discloses a speech synthesis method and device with a conditional matching stream, equipment and a medium, and the method comprises the steps: receiving text information, coding the text information into a high-dimensional implicit vector, and obtaining a text information vector; label information is formed according to the alignment relation between the text information and the voice, and the relation between the acoustic features of the text information and the voice is predicted; performing matching in the voice data set according to the text information vector to obtain corresponding text-related voice, generating intermediate-state voice according to Gaussian noise, the text-related voice and the time step, performing noise prediction according to the intermediate-state voice and the time step by adopting a conditional matching flow model to obtain predicted noise, and outputting the predicted noise. Calculating the loss between the input Gaussian noise and the prediction noise, and combining the prediction noise with the intermediate state voice to extract and obtain the acoustic features of the text related voice; and generating a decoded speech waveform according to the obtained acoustic characteristics of the text-related speech by adopting an acoustic decoder. And the speech synthesis quality is improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and speech processing technology, and in particular to a speech synthesis method, device, equipment and medium with conditional matching flow. Background Art

[0002] Intelligent customer service or intelligent assistant is one of the most extensive and important ways to implement voice processing technology in actual scenarios. Voice interaction can provide voice prompts or playback of responses to user operations through the client, which requires voice synthesis. Speech synthesis is an important field in the field of artificial intelligence, but the time steps of the diffusion model used in existing speech synthesis require dozens to hundreds of steps to generate good audio. The speech synthesis efficiency is low and the speech synthesis effect is not good. Summary of the invention

[0003] The present invention provides an artificial intelligence speech synthesis method, device, computer equipment and medium with conditional matching flow to improve the efficiency and quality of speech synthesis.

[0004] In a first aspect, a speech synthesis method with a conditional matching stream is provided, comprising:

[0005] Receive text information, encode the text information into a high-dimensional latent vector, and obtain a text information vector;

[0006] The label information is formed according to the alignment relationship between the text information and the speech, and the relationship between the text information and the acoustic features of the speech is predicted;

[0007] According to the text information vector, the corresponding text-related speech is matched in the speech data set, and the intermediate speech is generated according to the Gaussian noise, the text-related speech and the time step. The conditional matching flow model is used to predict the noise according to the intermediate speech and the time step to obtain the predicted noise, and the loss between the input Gaussian noise and the predicted noise is calculated. The acoustic features of the text-related speech are obtained according to the predicted noise combined with the intermediate speech extraction;

[0008] An acoustic decoder is used to generate a decoded speech waveform according to the obtained acoustic features of the text-related speech.

[0009] In a second aspect, a speech synthesis device with a conditional matching stream is provided, comprising:

[0010] A text encoder is used to receive text information, encode the text information into a high-dimensional latent vector, and obtain a text information vector;

[0011] Duration predictor, used to form label information based on the alignment relationship between text information and speech, and predict the relationship between text information and acoustic features of speech;

[0012] An acoustic feature extraction module is used to match the text information vector in the speech data set to obtain the corresponding text-related speech, generate intermediate speech according to Gaussian noise, text-related speech and time step, use a conditional matching flow model to perform noise prediction according to the intermediate speech and time step to obtain predicted noise, calculate the loss between the input Gaussian noise and the predicted noise, and extract the acoustic features of the text-related speech based on the predicted noise combined with the intermediate speech;

[0013] The acoustic decoder is used to generate a decoded speech waveform according to the obtained acoustic features of the text-related speech.

[0014] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned speech synthesis method with conditional matching stream when executing the computer program.

[0015] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned speech synthesis method with conditional matching flow are implemented.

[0016] In the scheme implemented by the above-mentioned speech synthesis method, device, computer equipment and storage medium with conditional matching flow, text information can be received through the client, and the text information can be encoded into a high-dimensional latent vector to obtain a text information vector; label information is formed according to the alignment relationship between the text information and the speech, and the relationship between the acoustic features of the text information and the speech is predicted; the corresponding text-related speech is obtained by matching the text information vector in the speech data set, and the intermediate state speech is generated according to Gaussian noise, text-related speech and time step, and the conditional matching flow model is used to predict the noise according to the intermediate state speech and time step to obtain the predicted noise, calculate the loss between the input Gaussian noise and the predicted noise, and extract the acoustic features of the text-related speech according to the predicted noise combined with the intermediate state speech; the acoustic decoder is used to generate a decoded speech waveform according to the obtained acoustic features of the text-related speech. In the present invention, by applying the conditional matching flow model to speech synthesis, the corresponding parameters of the model are small, the processing speed is accelerated, and the modeling is more intuitive, which can reduce the reasoning steps of the model, and can effectively improve the quality of speech synthesis and improve the effect of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0018] Figure 1 It is a schematic diagram of an application environment of a speech synthesis method with conditional matching stream in one embodiment of the present invention;

[0019] Figure 2 It is a flow chart of a speech synthesis method with conditional matching stream in one embodiment of the present invention;

[0020] Figure 3 is a structural schematic diagram of a speech synthesis device with a conditional matching stream in one embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0022] Figure 5 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] The speech synthesis method with conditional matching stream provided by the embodiment of the present invention can be applied in Figure 1In an application environment, the client communicates with the server through a network. The server can receive text information through the client, encode the text information into a high-dimensional latent vector, and obtain a text information vector; form label information according to the alignment relationship between the text information and the speech, and predict the relationship between the acoustic features of the text information and the speech; match the text information vector in the speech data set to obtain the corresponding text-related speech, generate an intermediate state speech according to Gaussian noise, text-related speech and time step, use a conditional matching flow model to predict noise according to the intermediate state speech and time step, obtain predicted noise, calculate the loss between the input Gaussian noise and the predicted noise, and extract the acoustic features of the text-related speech according to the predicted noise combined with the intermediate state speech; use an acoustic decoder to generate a decoded speech waveform according to the obtained acoustic features of the text-related speech. In the present invention, by applying the conditional matching flow model to speech synthesis, the corresponding parameters of the model are small, the processing speed is accelerated, and the modeling is more intuitive, the reasoning steps of the model can be reduced, and the quality of speech synthesis can be effectively improved, and the effect of speech synthesis can be improved. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0025] See also Figure 2 As shown, Figure 2 A flow chart of a speech synthesis method with conditional matching stream provided by an embodiment of the present invention includes the following steps:

[0026] S10: Receive text information, encode the text information into a high-dimensional latent vector, and obtain a text information vector.

[0027] The speech synthesis method with conditional matching stream provided by the present invention can be applied to intelligent question-and-answer engines such as intelligent customer service or intelligent assistants in various application scenarios. The intelligent question-and-answer engine is usually implemented through a server, which can receive user questions in real time. For example, in the insurance application field, users often ask questions by phone or online chat, and often need to use intelligent question-and-answer engines to answer some insurance questions of customers in order to improve insurance promotion efficiency and user experience.

[0028] For example, a user question may be: "What is the commission of Ping An Ping An Fu?" After receiving the user question, the intelligent question-answering engine analyzes the user question and obtains the corresponding standard answer, and performs speech synthesis based on the text information of the corresponding standard answer. Among them, Ping An Ping An Fu refers to the name of an insurance product. It is worth noting that the standard answer can refer to the answer written by the expert to the question, or the intelligent answer generated by the previous intelligent question-answering, which is not limited here.

[0029] S20: forming label information according to the alignment relationship between the text information and the speech, and predicting the relationship between the text information and the acoustic features of the speech.

[0030] In which, due to the temporal nature of the speech signal, there is a left-to-right correspondence between the text information and the speech. Before step S20, that is, before forming the label information according to the alignment relationship between the text information and the speech and predicting the relationship between the text information and the acoustic features, the following is further included:

[0031] A pre-trained Automatic Speech Recognition (ASR) model is used to obtain the alignment relationship between text information and speech.

[0032] Specifically, this step includes: using a pre-trained automatic speech recognition model to extract the spectrum of the speech signal frame by frame, predicting the text unit corresponding to the spectrum of each frame of the speech signal extracted, and combining the temporal sequence of the speech signal from left to right to obtain the number of frames of the speech signal corresponding to each text unit as the alignment relationship between the text information and the speech.

[0033] In this implementation, the alignment relationship between text information and speech may also be acquired by using a Beam Search or CTC alignment method, which will not be described in detail here.

[0034] S30: According to the text information vector, the corresponding text-related speech is matched in the speech data set, and the intermediate speech is generated according to the Gaussian noise, the text-related speech and the time step. The conditional matching flow model is used to predict the noise according to the intermediate speech and the time step to obtain the predicted noise, and the loss between the input Gaussian noise and the predicted noise is calculated. The acoustic features of the text-related speech are obtained based on the predicted noise combined with the intermediate speech extraction.

[0035] Among them, the speech data set represents a real speech data set, and the Gaussian noise is randomly extracted Gaussian noise. According to the text information, the corresponding text-related information is matched in the speech data set, so that the final synthesized speech is related to the text information, thereby improving the reliability of speech synthesis. The steps of extracting acoustic features may include: speech preprocessing: speech preprocessing includes speech pre-emphasis, which can highlight high-frequency information by emphasizing high-frequency speech energy; framing: framing the preprocessed speech using an overlapping frame method; windowing: adding a time window to the framed speech; Fourier transform: performing discrete Fourier transform calculation on the speech in the time window to extract the frequency domain information of the speech in the time window. The discrete Fourier transform may be a fast Fourier transform.

[0036] Specifically, in step S30, the conditional matching flow model used in the noise prediction according to the intermediate state speech and time step adopts the Transformer-UNet architecture.

[0037] In this embodiment, the conditional matching flow model includes six sequentially connected Transformer modules, and the first Transformer module is connected to the second Transformer module, the second Transformer module is connected to the third Transformer module, the third Transformer module is connected to the third-to-last Transformer module, the third-to-last Transformer module is connected to the second-to-last Transformer module, and the second-to-last Transformer module is connected to the first-to-last Transformer module. The third-to-last Transformer module is the fourth Transformer module, the second-to-last Transformer module is the fifth Transformer module, and the first-to-last Transformer module is the sixth Transformer module. Each of the Transformer modules includes a convolution unit and a Transformer unit, and the Transformer modules with symmetrical head and tail numbers are connected to each other. Among them, the Transformer module with symmetrical head and tail numbers refers to the Transformer module corresponding to the sequence number when the sequence number is sorted in order and the corresponding Transformer module whose reverse sequence number is equal to the value corresponding to the sequence number when the sequence number is sorted in reverse order, then the first Transformer module in the forward direction is also connected to the first Transformer module in the reverse direction, and the second Transformer module in the forward direction is also connected to the second Transformer module in the reverse direction. In other embodiments, the conditional matching flow model may include two or more Transformer modules.

[0038] Specifically, in step S30, the intermediate state speech in the intermediate state speech generated according to Gaussian noise, text-related speech and time step is calculated using formula (1):

[0039] X t =tX0+(1-t)X1(1)

[0040] Where, X t represents the intermediate state speech, t represents the time step, X0 represents the text-related speech, and X1 represents Gaussian noise.

[0041] S40: using an acoustic decoder to generate a decoded speech waveform according to the acquired acoustic features of the text-related speech.

[0042] After step S40, that is, after using the acoustic decoder to generate a decoded speech waveform according to the obtained acoustic features of the text-related speech, the method further includes:

[0043] The similarity between the original waveform of the text-related speech obtained by matching the speech dataset and the decoded speech waveform is calculated to compare the loss between the original waveform of the text-related speech and the decoded speech waveform, so as to train the acoustic decoder, improve the accuracy of the acoustic decoder, and then improve the speech synthesis effect and quality.

[0044] It can be seen that in the above scheme, by applying the conditional matching flow model to speech synthesis, the model has fewer corresponding parameters, the processing speed is accelerated, and the modeling is more intuitive, which can reduce the reasoning steps of the model and effectively improve the quality of speech synthesis and the effect of speech synthesis.

[0045] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0046] In one embodiment, a speech synthesis device with a conditional matching stream is provided, and the speech synthesis device with a conditional matching stream corresponds one-to-one to the speech synthesis method with a conditional matching stream in the above embodiment. Figure 3 As shown, the speech synthesis device with conditional matching stream includes a text encoder 101, a duration predictor 102, an acoustic feature extraction module 103 and an acoustic decoder 104. The functional modules are described in detail as follows:

[0047] The text encoder 101 is used to receive text information, encode the text information into a high-dimensional latent vector, and obtain a text information vector;

[0048] A duration predictor 102, used to form label information according to the alignment relationship between the text information and the speech, and predict the relationship between the text information and the acoustic features of the speech;

[0049] The acoustic feature extraction module 103 is used to match the text information vector in the speech data set to obtain the corresponding text-related speech, generate the intermediate state speech according to the Gaussian noise, the text-related speech and the time step, use the conditional matching flow model to perform noise prediction according to the intermediate state speech and the time step to obtain the predicted noise, calculate the loss between the input Gaussian noise and the predicted noise, and extract the acoustic features of the text-related speech according to the predicted noise combined with the intermediate state speech;

[0050] The acoustic decoder 104 is used to generate a decoded speech waveform according to the obtained acoustic features of the text-related speech.

[0051] Among them, due to the temporal nature of the speech signal, there is a left-to-right correspondence between the text information and the speech. The speech data set represents a real speech data set, and the Gaussian noise is a randomly extracted Gaussian noise. The acoustic feature extraction module 103 matches the corresponding text-related information in the speech data set according to the text information, so that the final synthesized speech is related to the text information, thereby improving the reliability of speech synthesis. The acoustic feature extraction module 103 is used to extract acoustic features, and the steps of extracting acoustic features may include: speech preprocessing: speech preprocessing includes speech pre-emphasis, which can highlight high-frequency information by emphasizing high-frequency speech energy; framing: framing the preprocessed speech using an overlapping frame method; windowing: adding a time window to the framed speech; Fourier transform: performing discrete Fourier transform calculation on the speech in the time window to extract the frequency domain information of the speech in the time window. The discrete Fourier transform may be a fast Fourier transform.

[0052] In one embodiment, the duration predictor 102 is further configured to:

[0053] A pre-trained automatic speech recognition model is used to obtain the alignment relationship between text information and speech.

[0054] In one embodiment, the duration predictor 102 is specifically configured to:

[0055] A pre-trained automatic speech recognition model is used to extract the spectrum of the speech signal frame by frame, and the text unit corresponding to the spectrum of each frame of the speech signal is predicted. Combined with the temporal sequence of the speech signal from left to right, the number of frames of the speech signal corresponding to each text unit is obtained as the alignment relationship between the text information and the speech.

[0056] In one embodiment, in the acoustic feature extraction module 103, the conditional matching flow model adopts a Transformer-UNet architecture.

[0057] In one embodiment, in the acoustic feature extraction module 103, the conditional matching flow model includes a plurality of sequentially connected Transformer modules, each of which includes a convolution unit and a Transformer unit, and the Transformer modules with symmetrical head and tail numbers are connected to each other. Among them, the Transformer modules with symmetrical head and tail numbers refer to the Transformer modules corresponding to the sequence numbers when the sequence numbers are sorted in order and the corresponding Transformer modules whose reverse sequence numbers are equal to the values ​​corresponding to the sequence numbers when the sequence numbers are sorted in reverse order, then the Transformer module with the first consecutive number is also connected to the Transformer module with the first consecutive number, and the Transformer module with the second consecutive number is also connected to the Transformer module with the second consecutive number, and so on.

[0058] In one embodiment, the acoustic feature extraction module 103 is specifically used to:

[0059] The intermediate speech is calculated using formula (1):

[0060] X t =tX0+(1-t)X1(1)

[0061] Where, X t represents the intermediate state speech, t represents the time step, X0 represents the text-related speech, and X1 represents Gaussian noise.

[0062] In one embodiment, the acoustic decoder 104 is further configured to:

[0063] The similarity between the original waveform of the text-related speech obtained by matching the speech dataset and the decoded speech waveform is calculated to compare the loss between the original waveform of the text-related speech and the decoded speech waveform, so as to train the acoustic decoder, improve the accuracy of the acoustic decoder, and then improve the speech synthesis effect and quality.

[0064] The present invention provides a speech synthesis device with a conditional matching stream. By applying a conditional matching stream model to speech synthesis, the model has fewer corresponding parameters, the processing speed is accelerated, and the modeling is more intuitive, which can reduce the reasoning steps of the model and effectively improve the quality of speech synthesis and the effect of speech synthesis.

[0065] For the specific limitations of the speech synthesis device with conditional matching stream, please refer to the limitations of the intelligent question and answer method above, which will not be repeated here. The various modules in the above-mentioned speech synthesis device with conditional matching stream can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0066] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of a speech synthesis method with a conditional matching stream are implemented.

[0067] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, the client-side functions or steps of a speech synthesis method with conditional matching flow are implemented.

[0068] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0069] Receive text information, encode the text information into a high-dimensional latent vector, and obtain a text information vector;

[0070] The label information is formed according to the alignment relationship between the text information and the speech, and the relationship between the text information and the acoustic features of the speech is predicted;

[0071] According to the text information vector, the corresponding text-related speech is matched in the speech data set, and the intermediate speech is generated according to the Gaussian noise, the text-related speech and the time step. The conditional matching flow model is used to predict the noise according to the intermediate speech and the time step to obtain the predicted noise, and the loss between the input Gaussian noise and the predicted noise is calculated. The acoustic features of the text-related speech are obtained according to the predicted noise combined with the intermediate speech extraction;

[0072] An acoustic decoder is used to generate a decoded speech waveform according to the obtained acoustic features of the text-related speech.

[0073] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0074] Receive text information, encode the text information into a high-dimensional latent vector, and obtain a text information vector;

[0075] The label information is formed according to the alignment relationship between the text information and the speech, and the relationship between the text information and the acoustic features of the speech is predicted;

[0076] According to the text information vector, the corresponding text-related speech is matched in the speech data set, and the intermediate speech is generated according to the Gaussian noise, the text-related speech and the time step. The conditional matching flow model is used to predict the noise according to the intermediate speech and the time step to obtain the predicted noise, and the loss between the input Gaussian noise and the predicted noise is calculated. The acoustic features of the text-related speech are obtained according to the predicted noise combined with the intermediate speech extraction;

[0077] An acoustic decoder is used to generate a decoded speech waveform according to the obtained acoustic features of the text-related speech.

[0078] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0079] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0080] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0081] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A speech synthesis method with conditional matching stream, characterized in that: include: Receive text information, encode the text information into a high-dimensional latent vector, and obtain a text information vector; The label information is formed according to the alignment relationship between the text information and the speech, and the relationship between the text information and the acoustic features of the speech is predicted; According to the text information vector, the corresponding text-related speech is matched in the speech data set, and the intermediate speech is generated according to the Gaussian noise, the text-related speech and the time step. The conditional matching flow model is used to predict the noise according to the intermediate speech and the time step to obtain the predicted noise, and the loss between the input Gaussian noise and the predicted noise is calculated. The acoustic features of the text-related speech are obtained according to the predicted noise combined with the intermediate speech extraction; An acoustic decoder is used to generate a decoded speech waveform according to the obtained acoustic features of the text-related speech.

2. The speech synthesis method with conditional matching stream according to claim 1, characterized in that: Before forming label information according to the alignment relationship between the text information and the speech and predicting the relationship between the acoustic features of the text information and the speech, the method further includes: A pre-trained automatic speech recognition model is used to obtain the alignment relationship between text information and speech.

3. The speech synthesis method with conditional matching stream according to claim 2, characterized in that: The pre-trained automatic speech recognition model is used to obtain the alignment relationship between text information and speech, specifically: A pre-trained automatic speech recognition model is used to extract the spectrum of the speech signal frame by frame, and the text unit corresponding to the spectrum of each frame of the speech signal is predicted. Combined with the temporal sequence of the speech signal from left to right, the number of frames of the speech signal corresponding to each text unit is obtained as the alignment relationship between the text information and the speech.

4. The speech synthesis method with conditional matching stream according to claim 1, characterized in that: The conditional matching flow model used to predict noise according to intermediate speech and time step adopts the Transformer-UNet architecture.

5. The speech synthesis method with conditional matching stream according to claim 4, characterized in that: The conditional matching flow model used to predict noise according to intermediate speech and time step includes multiple Transformer modules connected in sequence, each of which includes a convolution unit and a Transformer unit, and the Transformer modules with symmetrical head and tail numbers are connected to each other.

6. The speech synthesis method with conditional matching stream according to claim 1, characterized in that: The intermediate state speech in the intermediate state speech generated according to Gaussian noise, text-related speech and time step is calculated by the following formula: X t =tX0+(1-t)X1 Where, X t represents the intermediate state speech, t represents the time step, X0 represents the text-related speech, and X1 represents Gaussian noise.

7. The speech synthesis method with conditional matching stream according to claim 1, characterized in that: After the acoustic decoder is used to generate a decoded speech waveform according to the obtained acoustic features of the text-related speech, the method further includes: Calculate the similarity between the original waveform of the text-related speech obtained by matching the speech dataset and the decoded speech waveform.

8. A speech synthesis device with conditional matching stream, characterized in that: include: A text encoder is used to receive text information, encode the text information into a high-dimensional latent vector, and obtain a text information vector; Duration predictor, used to form label information based on the alignment relationship between text information and speech, and predict the relationship between text information and acoustic features of speech; An acoustic feature extraction module is used to match the text information vector in the speech data set to obtain the corresponding text-related speech, generate intermediate speech according to Gaussian noise, text-related speech and time step, use a conditional matching flow model to perform noise prediction according to the intermediate speech and time step to obtain predicted noise, calculate the loss between the input Gaussian noise and the predicted noise, and extract the acoustic features of the text-related speech based on the predicted noise combined with the intermediate speech; The acoustic decoder is used to generate a decoded speech waveform according to the obtained acoustic features of the text-related speech.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the speech synthesis method with conditional matching stream as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the speech synthesis method with conditional matching stream as claimed in any one of claims 1 to 7 are implemented.

Images