A speech synthesis method, device, apparatus and storage medium thereof

Abstract

The embodiment of the application belongs to the technical field of speech synthesis, is applied to a text-to-speech scene, and relates to a speech synthesis method, device, equipment and storage medium thereof. When actual speech synthesis is performed, a pre-trained speech synthesis intelligent agent is used to perform speech synthesis, so that when actual text-to-speech synthesis is performed, more accurate text-acoustic sequence alignment effects can be obtained, noise introduction can be greatly avoided, the audio quality of synthesized speech is ensured, and moreover, a sequence alignment component based on a differentiable aligner can quickly realize speech synthesis. The speech synthesis method is applied to the financial customer service industry, and can provide customers with more high-quality and good sound quality service functions.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, and is applied in text-to-speech scenarios, and particularly to a speech synthesis method, apparatus, device and its storage medium. Background Technology

[0002] With the rapid development of the internet, especially the self-media and financial industries, traditional speech synthesis is increasingly moving towards AI-powered intelligent speech synthesis. This often requires the use of speech synthesis technology to output a specified speech as the pronunciation of another person.

[0003] Existing speech synthesis tasks fall into several mainstream approaches. The first approach uses instance standardization to decouple and extract content and timbre information from the input speech. The second approach breaks down speech into multiple features such as content, rhythm, pitch, and timbre, and then uses supervised techniques and random sampling to extract the corresponding features. However, the instance standardization approach cannot achieve true feature decoupling, resulting in poor synthesis performance in speech synthesis tasks. The second approach tends to have a lot of partial repetition in feature selection, leading to excessive encoding and decoding, which introduces more noise and cannot guarantee the quality of speech synthesis. Therefore, existing technologies cannot quickly achieve speech synthesis in text-to-speech conversion and are prone to introducing noise, resulting in poor synthesized speech quality and affecting the customer's dialogue experience. Summary of the Invention

[0004] The purpose of this application is to provide a speech synthesis method, apparatus, device and storage medium to solve the problem that the prior art cannot quickly achieve speech synthesis in text-to-speech synthesis and is prone to introducing noise, resulting in poor quality of synthesized speech and affecting the customer's dialogue experience.

[0005] To address the aforementioned technical problems, this application provides a speech synthesis method, employing the following technical solution:

[0006] A speech synthesis method includes the following steps:

[0007] Obtain the text to be synthesized speech;

[0008] The speech text to be synthesized is serialized to obtain a speech text sequence;

[0009] The speech-text sequence to be synthesized is input into a pre-trained speech synthesis agent to obtain the acoustic signal synthesized by the speech synthesis agent.

[0010] The acoustic signal is sent to a preset speech generator to output synthesized speech.

[0011] Furthermore, the speech synthesis agent includes a sequence alignment component based on a differentiable aligner, an acoustic feature generation component, and a streaming decoding processing component. The step of inputting the speech text sequence to be synthesized into the pre-trained speech synthesis agent to obtain the acoustic signal synthesized by the speech synthesis agent specifically includes:

[0012] The speech-text sequence to be synthesized is input into the sequence alignment component, and the speech-text sequence that is time-aligned with the target Mel spectrum is output, wherein the target Mel spectrum refers to the Mel spectrum corresponding to the speech to be synthesized;

[0013] The speech text sequence that is time-aligned with the target Mel spectrum is mapped through a preset first linear mapping layer to obtain a series of high-dimensional acoustic feature representations that follow a Gaussian distribution;

[0014] The series of high-dimensional acoustic feature representations that follow a Gaussian distribution are input into the acoustic feature generation component to generate acoustic features for the speech to be synthesized.

[0015] The acoustic features of the speech to be synthesized are input into the streaming decoding processing component to perform acoustic feature decoding and obtain the target acoustic signal.

[0016] Furthermore, the acoustic feature generation component includes a Mel acoustic feature extraction sub-component and a first decoding sub-component. The step of inputting the series of high-dimensional acoustic feature representations following a Gaussian distribution into the acoustic feature generation component to generate acoustic features of the speech to be synthesized specifically includes:

[0017] The acoustic feature extraction subcomponent of the Mel acoustic feature extraction is used to extract acoustic features from the series of high-dimensional acoustic feature representations that follow a Gaussian distribution, and the acoustic feature extraction results are obtained.

[0018] The acoustic feature extraction result is input into the first decoding sub-component for decoding processing to obtain the acoustic features of the speech to be synthesized;

[0019] The streaming decoding component includes a sampling sub-component and a second decoding sub-component. The step of inputting the acoustic features of the speech to be synthesized into the streaming decoding component, performing acoustic feature decoding, and obtaining the target acoustic signal specifically includes:

[0020] The acoustic features of the speech to be synthesized are input into the sampling sub-component for Gaussian sampling to obtain the sampled acoustic features;

[0021] The target acoustic signal is obtained by decoding the sampled acoustic features using the second decoding sub-component.

[0022] Furthermore, before performing the step of inputting the speech-text sequence to be synthesized into the pre-trained speech synthesis agent to obtain the acoustic signal synthesized by the speech synthesis agent, the method further includes:

[0023] Obtain batch reference voice recordings from the target user;

[0024] The acoustic feature generation component is trained for noise reduction using the batch of reference speech to obtain a noise-reduced acoustic feature generation component; and...

[0025] Obtain all acoustic features generated by the acoustic feature generation component after the noise reduction training is completed;

[0026] The streaming decoding component is trained to perform noise reduction training using all the acoustic features generated by the acoustic feature generation component that has completed the noise reduction training.

[0027] Furthermore, the acoustic feature generation component also includes a first encoding sub-component. The step of training the acoustic feature generation component to reduce noise using the batch reference speech to obtain a noise-reduced acoustic feature generation component specifically includes:

[0028] Step 501: Input the batch of reference speech into the Mel acoustic feature extraction sub-component to obtain the acoustic feature extraction results in Mel spectrum form corresponding to all reference speech;

[0029] Step 502: Perform a first noise-adding process on the acoustic feature extraction results in the form of the Mel spectrum corresponding to all reference speech to obtain the noisy acoustic feature extraction results.

[0030] Step 503: Use the first encoding sub-component to perform feature encoding on the noisy acoustic feature extraction result to obtain the acoustic feature encoding result;

[0031] Step 504: Input the acoustic feature encoding result into the first decoding sub-component for decoding processing to obtain the acoustic features corresponding to all the reference speech.

[0032] Step 505: Map the acoustic features corresponding to all the reference speech through a preset second linear mapping layer to obtain the high-dimensional acoustic feature representations corresponding to all the reference speech.

[0033] Step 506: Using the maximum likelihood estimation method, calculate the loss value of the high-dimensional acoustic feature representation corresponding to each of the reference speech segments compared to the acoustic feature extraction result in the Mel spectrum form;

[0034] Step 507: If the loss value exceeds the preset first loss threshold, adjust the encoding and decoding parameters of the acoustic feature generation component and / or adjust the mapping parameters of the second linear mapping layer, and re-execute steps 501 to 506 until the loss value does not exceed the first loss threshold, then the acoustic feature generation component training is complete.

[0035] Step 508: If the loss value does not exceed the first loss threshold, the trained acoustic feature generation component is obtained.

[0036] Furthermore, after performing the step of obtaining batch reference speech of the target speech user, the method further includes:

[0037] Step 601: Perform text serialization processing on the batch of reference speech to obtain the serialized text corresponding to each batch of reference speech;

[0038] Step 602: Obtain the temporal features corresponding to the batch of reference speech from the acoustic feature extraction results in the form of the Mel spectrum corresponding to all reference speech, and use them as the temporal features extracted from the Mel spectrum.

[0039] Step 603: Deploy the time-domain features as reference knowledge to the sequence alignment component to obtain a sequence alignment component with complete reference knowledge deployment;

[0040] Step 604: Input the serialized text corresponding to each of the batch reference speech into the sequence alignment component after the reference knowledge is deployed, and output the speech text sequence that is time-aligned with the target Mel spectrum, wherein the target Mel spectrum refers to the Mel spectrum corresponding to each of the batch reference speech;

[0041] Step 605: The speech text sequence that is time-aligned with the target Mel spectrum is mapped through the first linear mapping layer to obtain a series of high-dimensional acoustic feature representations that follow a Gaussian distribution;

[0042] Step 606: Extract the temporal features corresponding to the batch of reference speech from the series of high-dimensional acoustic feature representations that follow a Gaussian distribution, and use them as temporal features for adjustment by the sequence alignment component;

[0043] Step 607: Obtain the similarity between the temporal features adjusted by the sequence alignment component and the temporal features extracted by the Mel spectrum through a comparative calculation method.

[0044] Step 608: If the similarity does not exceed the preset similarity threshold, adjust the alignment parameters of the sequence alignment component and / or the mapping parameters of the first linear mapping layer, and re-execute steps 604 to 607 until the similarity exceeds the preset similarity threshold, then the training of the sequence alignment component is complete.

[0045] Step 609: If the similarity exceeds a preset similarity threshold, the trained sequence alignment component is obtained.

[0046] Furthermore, the streaming decoding component also includes a second encoding sub-component. The step of using all acoustic features generated by the acoustic feature generation component that has undergone noise reduction training to train the streaming decoding component for noise reduction, and obtaining a stream decoding component that has undergone noise reduction training, specifically includes:

[0047] Step 701: Perform a second noise-adding process on all acoustic features generated by the acoustic feature generation component after the noise reduction training is completed, to obtain all acoustic features after noise addition.

[0048] Step 702: Use the second coding sub-component to perform feature coding on all the acoustic features after adding noise, and obtain the acoustic feature coding result;

[0049] Step 703: Input the acoustic feature encoding result into the second decoding sub-component for decoding processing to obtain the acoustic signals corresponding to all reference speech.

[0050] Step 704: Using the maximum likelihood estimation method, calculate the loss value of the acoustic signal corresponding to each of the reference speech to all the acoustic features generated by the acoustic feature generation component after the noise reduction training is completed.

[0051] Step 705: If the loss value exceeds the preset second loss threshold, adjust the encoding and decoding parameters of the streaming decoding component and re-execute steps 701 to 704 until the loss value does not exceed the second loss threshold, then the training of the streaming decoding component is complete.

[0052] Step 706: If the loss value does not exceed the second loss threshold, the trained streaming decoding component is obtained.

[0053] To address the aforementioned technical problems, this application also provides a speech synthesis device, which employs the following technical solution:

[0054] A speech synthesis device, comprising:

[0055] The text acquisition module is used to acquire the text to be synthesized speech;

[0056] A serialization processing module is used to perform serialization processing on the speech text to be synthesized to obtain a speech text sequence to be synthesized.

[0057] The speech synthesis module is used to input the speech text sequence to be synthesized into a pre-trained speech synthesis agent to obtain the acoustic signal synthesized by the speech synthesis agent.

[0058] The voice output module is used to send the acoustic signal to a preset voice transmitter to output synthesized voice.

[0059] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0060] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the speech synthesis method described above.

[0061] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0062] A computer-readable storage medium storing computer-readable instructions that, when executed by a processor, implement the steps of the speech synthesis method described above.

[0063] Compared with the prior art, the embodiments of this application have the following main advantages:

[0064] The speech synthesis method described in this application uses a pre-trained speech synthesis agent to perform speech synthesis, ensuring that the actual text-to-speech synthesis achieves more accurate text-acoustic sequence alignment while minimizing noise introduction and guaranteeing the audio quality of the synthesized speech. Furthermore, the use of a sequence alignment component based on a differentiable aligner enables rapid speech synthesis. Applying this speech synthesis method to the financial customer service industry can provide customers with higher quality audio services. Attached Figure Description

[0065] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0067] Figure 2 This is a flowchart of an embodiment of the speech synthesis method according to this application;

[0068] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 203 shown;

[0069] Figure 4 This is a flowchart of a specific embodiment of training the speech synthesis agent in the speech synthesis method described in this application;

[0070] Figure 5 yes Figure 4 A flowchart of a specific embodiment of step 402 shown;

[0071] Figure 6 This is a flowchart of a specific embodiment of training the sequence alignment component in the speech synthesis method described in this application;

[0072] Figure 7 yes Figure 4 A flowchart of a specific embodiment of step 404 shown;

[0073] Figure 8 This is a schematic diagram of the structure of one embodiment of the speech synthesis apparatus according to this application;

[0074] Figure 9 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0076] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0077] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0078] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0079] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0080] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0081] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0082] It should be noted that the speech synthesis method provided in this application embodiment is generally executed by a terminal device, and correspondingly, the speech synthesis device is generally located in the terminal device.

[0083] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0084] Continue to refer to Figure 2 A flowchart of an embodiment of the speech synthesis method according to this application is shown. The speech synthesis method includes the following steps:

[0085] Step 201: Obtain the text to be synthesized speech.

[0086] In this embodiment, the text to be synthesized refers to a piece of text content to be converted into speech. For example, in the field of fintech, the pre-set text content of the intelligent customer service when answering customer questions can be converted into synthesized speech through the speech synthesis method provided in this application. This speech synthesis method can also be used in teaching assistance scenarios such as intelligent recitation, intelligent storytelling, and intelligent reading, such as audiobooks.

[0087] Step 202: Serialize the speech text to be synthesized to obtain a speech text sequence.

[0088] Specifically, the process of serializing the speech text to be synthesized to obtain a speech text sequence is performed by sequentially sorting the text according to the order of the characters in the speech text to be synthesized, thereby obtaining a speech text sequence to be synthesized, which facilitates subsequent speech synthesis based on the sequence.

[0089] Step 203: Input the speech-text sequence to be synthesized into the pre-trained speech synthesis agent to obtain the acoustic signal synthesized by the speech synthesis agent.

[0090] In this embodiment, the speech synthesis agent is, for example, a speech synthesis component or a speech synthesis model.

[0091] In this embodiment, the speech synthesis agent includes a sequence alignment component based on a differentiable aligner, an acoustic feature generation component, and a streaming decoding processing component. The sequence alignment component based on a differentiable aligner is, for example, the alignment module Differentiable Algner of EFficityTTS2. EFficityTTS2, or EFTS2, is an efficient and high-performance speech synthesis model structure. Its alignment module Differentiable Algner is a fully differentiable internal alignment module. Compared with some other non-differentiable or external alignment modules, such as MAS search in VITS and MFA in FastSpeech, the alignment module of EFTS2 is accurate and computationally efficient.

[0092] Continue to refer to Figure 3 , Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 203 shown includes:

[0093] Step 301: Input the speech-text sequence to be synthesized into the sequence alignment component and output the speech-text sequence that is time-aligned with the target Mel spectrum, wherein the target Mel spectrum refers to the Mel spectrum corresponding to the speech to be synthesized;

[0094] Specifically, in an acoustic Mel spectrum, the horizontal axis is the time axis, used to represent the change of sound frequency within a time period, and the vertical axis is the frequency axis, used to characterize the change of sound frequency value; it can also be understood that the acoustic Mel spectrum contains both time-domain and frequency-domain features. The target Mel spectrum, for example, is the acoustic Mel spectrum corresponding to the speech text to be synthesized, output as Zhang San's speech.

[0095] The sequence alignment component aligns the text sequence to be synthesized, making the subsequently synthesized speech more closely resemble the real target voice, resulting in a more human-like, fluent, and natural sound.

[0096] Step 302: The speech text sequence that is time-aligned with the target Mel spectrum is mapped through a preset first linear mapping layer to obtain a series of high-dimensional acoustic feature representations that follow a Gaussian distribution;

[0097] Specifically, by mapping the speech text sequence that is time-aligned with the target Mel spectrum through a preset first linear mapping layer, a series of high-dimensional acoustic feature representations that follow a Gaussian distribution are obtained, which facilitates the subsequent processing steps to obtain the acoustic Mel spectrum.

[0098] Step 303: Input the series of high-dimensional acoustic feature representations that follow a Gaussian distribution into the acoustic feature generation component to generate acoustic features of the speech to be synthesized;

[0099] Specifically, the acoustic feature generation component includes a Mel spectrum acoustic feature extraction subcomponent and a first decoding subcomponent. The Mel spectrum acoustic feature extraction subcomponent is used to extract Mel spectrum acoustic features from the high-dimensional feature representation, and the first decoding subcomponent is used to decode the extracted acoustic features to obtain the acoustic features of the speech to be synthesized.

[0100] In this embodiment, the step of inputting the series of high-dimensional acoustic feature representations following a Gaussian distribution into the acoustic feature generation component to generate acoustic features of the speech to be synthesized specifically includes: performing acoustic feature extraction on the series of high-dimensional acoustic feature representations following a Gaussian distribution through the Mel acoustic feature extraction sub-component to obtain acoustic feature extraction results; and inputting the acoustic feature extraction results into the first decoding sub-component for decoding processing to obtain the acoustic features of the speech to be synthesized.

[0101] Step 304: Input the acoustic features of the speech to be synthesized into the streaming decoding processing component to perform acoustic feature decoding and obtain the target acoustic signal.

[0102] Specifically, the streaming decoding component includes a sampling sub-component and a second decoding sub-component. The sampling sub-component is used to sample the acoustic features of the speech to be synthesized to obtain more comprehensive acoustic features, and the second decoding sub-component is used to decode the acoustic features into an audio signal.

[0103] In this embodiment, the step of inputting the acoustic features of the speech to be synthesized into the streaming decoding component for acoustic feature decoding to obtain the target acoustic signal specifically includes: inputting the acoustic features of the speech to be synthesized into the sampling sub-component for Gaussian sampling to obtain the sampled acoustic features; and performing acoustic feature decoding on the sampled acoustic features through the second decoding sub-component to obtain the target acoustic signal. Specifically, Gaussian sampling refers to the selection of acoustic feature points that follow a Gaussian distribution during sampling.

[0104] In this embodiment, before performing the step of inputting the speech-text sequence to be synthesized into a pre-trained speech synthesis agent to obtain the acoustic signal synthesized by the speech synthesis agent, the method further includes a method for training the speech synthesis agent. Training the speech synthesis agent makes the speech synthesized by the agent more natural and closer to the pronunciation of the target object, providing customers with higher quality audio services.

[0105] Continue to refer to Figure 4 , Figure 4 This is a flowchart of a specific embodiment of training the speech synthesis agent in the speech synthesis method described in this application, including:

[0106] Step 401: Obtain batch reference voice recordings of the target voice user;

[0107] Step 402: The acoustic feature generation component is trained for noise reduction using the batch reference speech to obtain a noise-reduced acoustic feature generation component; and...

[0108] Step 403: Obtain all acoustic features generated by the acoustic feature generation component after the noise reduction training is completed;

[0109] In this embodiment, the acoustic feature generation component further includes a first encoding sub-component. Specifically, the acoustic feature generation component includes a Mel acoustic feature extraction sub-component, a first encoding sub-component, and a first decoding sub-component. Before actual application, the acoustic feature generation component is first trained to reduce noise, so that noise interference can be eliminated as much as possible during subsequent actual speech synthesis, resulting in better audio quality of synthesized speech.

[0110] Continue to refer to Figure 5 , Figure 5 yes Figure 4 A flowchart of a specific embodiment of step 402 shown includes:

[0111] Step 501: Input the batch of reference speech into the Mel acoustic feature extraction sub-component to obtain the acoustic feature extraction results in Mel spectrum form corresponding to all reference speech;

[0112] Step 502: Perform a first noise-adding process on the acoustic feature extraction results in the form of the Mel spectrum corresponding to all reference speech to obtain the noisy acoustic feature extraction results.

[0113] Step 503: Use the first encoding sub-component to perform feature encoding on the noisy acoustic feature extraction result to obtain the acoustic feature encoding result;

[0114] Step 504: Input the acoustic feature encoding result into the first decoding sub-component for decoding processing to obtain the acoustic features corresponding to all the reference speech.

[0115] Step 505: Map the acoustic features corresponding to all the reference speech through a preset second linear mapping layer to obtain the high-dimensional acoustic feature representations corresponding to all the reference speech.

[0116] By using a linear mapping layer, the acoustic features corresponding to all reference speech are converted into high-dimensional acoustic feature representations, which facilitates comparison with the acoustic feature extraction results in the form of the Mel spectrum.

[0117] Step 506: Using the maximum likelihood estimation method, calculate the loss value of the high-dimensional acoustic feature representation corresponding to each of the reference speech segments compared to the acoustic feature extraction result in the Mel spectrum form;

[0118] Step 507: If the loss value exceeds the preset first loss threshold, adjust the encoding and decoding parameters of the acoustic feature generation component and / or adjust the mapping parameters of the second linear mapping layer, and re-execute steps 501 to 506 until the loss value does not exceed the first loss threshold, then the acoustic feature generation component training is complete.

[0119] Step 508: If the loss value does not exceed the first loss threshold, the trained acoustic feature generation component is obtained.

[0120] Essentially, the acoustic feature generation component undergoes noise reduction training. First, noise is added to the acoustic feature extraction results in the form of the Mel spectrum corresponding to all reference speech samples to obtain noisy acoustic feature extraction results. Then, encoding and decoding processes are performed. Finally, the loss value between the decoded result and the acoustic feature extraction result before noise addition is calculated. The acoustic feature generation component is adjusted based on the loss value, ultimately obtaining a trained acoustic feature generation component. This means that the trained acoustic feature generation component learns automatic noise recognition and elimination during acoustic feature generation, ensuring reduced noise introduction during actual speech synthesis.

[0121] In this embodiment, after performing the step of obtaining batch reference speech of the target speech user, the speech synthesis method further includes a sequence alignment component training method.

[0122] Continue to refer to Figure 6 , Figure 6 This is a flowchart of a specific embodiment of training the sequence alignment component in the speech synthesis method described in this application, including:

[0123] Step 601: Perform text serialization processing on the batch of reference speech to obtain the serialized text corresponding to each batch of reference speech;

[0124] Step 602: Obtain the temporal features corresponding to the batch of reference speech from the acoustic feature extraction results in the form of the Mel spectrum corresponding to all reference speech, and use them as the temporal features extracted from the Mel spectrum.

[0125] Step 603: Deploy the time-domain features as reference knowledge to the sequence alignment component to obtain a sequence alignment component with complete reference knowledge deployment;

[0126] Step 604: Input the serialized text corresponding to each of the batch reference speech into the sequence alignment component after the reference knowledge is deployed, and output the speech text sequence that is time-aligned with the target Mel spectrum, wherein the target Mel spectrum refers to the Mel spectrum corresponding to each of the batch reference speech;

[0127] Step 605: The speech text sequence that is time-aligned with the target Mel spectrum is mapped through the first linear mapping layer to obtain a series of high-dimensional acoustic feature representations that follow a Gaussian distribution;

[0128] Step 606: Extract the temporal features corresponding to the batch of reference speech from the series of high-dimensional acoustic feature representations that follow a Gaussian distribution, and use them as temporal features for adjustment by the sequence alignment component;

[0129] Step 607: Obtain the similarity between the temporal features adjusted by the sequence alignment component and the temporal features extracted by the Mel spectrum through a comparative calculation method.

[0130] Step 608: If the similarity does not exceed the preset similarity threshold, adjust the alignment parameters of the sequence alignment component and / or the mapping parameters of the first linear mapping layer, and re-execute steps 604 to 607 until the similarity exceeds the preset similarity threshold, then the training of the sequence alignment component is complete.

[0131] Step 609: If the similarity exceeds a preset similarity threshold, the trained sequence alignment component is obtained.

[0132] By training the sequence alignment component, when actual speech synthesis is performed, the alignment effect can be guaranteed when the speech text sequence to be synthesized is input into the sequence alignment component and the output speech text sequence is time-aligned with the target Mel spectrum.

[0133] Step 404: Use all the acoustic features generated by the acoustic feature generation component that has completed the noise reduction training to perform noise reduction training on the streaming decoding component, and obtain the streaming decoding component that has completed the noise reduction training.

[0134] In this embodiment, the streaming decoding component further includes a second encoding sub-component, meaning the streaming decoding component comprises a sampling sub-component, a second encoding sub-component, and a second decoding sub-component. By training the streaming decoding component to reduce noise, noise interference can be minimized during subsequent speech synthesis, resulting in better audio quality of the synthesized speech.

[0135] Continue to refer to Figure 7 , Figure 7 yes Figure 4 A flowchart of a specific embodiment of step 404 shown includes:

[0136] Step 701: Perform a second noise-adding process on all acoustic features generated by the acoustic feature generation component after the noise reduction training is completed, to obtain all acoustic features after noise addition.

[0137] Step 702: Use the second coding sub-component to perform feature coding on all the acoustic features after adding noise, and obtain the acoustic feature coding result;

[0138] Step 703: Input the acoustic feature encoding result into the second decoding sub-component for decoding processing to obtain the acoustic signals corresponding to all reference speech.

[0139] Step 704: Using the maximum likelihood estimation method, calculate the loss value of the acoustic signal corresponding to each of the reference speech to all the acoustic features generated by the acoustic feature generation component after the noise reduction training is completed.

[0140] Step 705: If the loss value exceeds the preset second loss threshold, adjust the encoding and decoding parameters of the streaming decoding component and re-execute steps 701 to 704 until the loss value does not exceed the second loss threshold, then the training of the streaming decoding component is complete.

[0141] Step 706: If the loss value does not exceed the second loss threshold, the trained streaming decoding component is obtained.

[0142] Essentially, the streaming decoding component undergoes noise reduction training. First, all acoustic features generated by the trained acoustic feature generation component are subjected to a second noise-adding process to obtain noisy acoustic features. Then, these features are encoded and decoded. Finally, the loss value of the acoustic signal corresponding to each reference speech is calculated compared to the loss value of all acoustic features generated by the trained acoustic feature generation component. The streaming decoding component is adjusted based on this loss value. Ultimately, a trained streaming decoding component is obtained, enabling it to automatically identify and eliminate noise before decoding actual audio signals, thus minimizing noise introduction during actual speech synthesis.

[0143] Step 204: Send the acoustic signal to a preset speech generator to output synthesized speech.

[0144] In this embodiment, the preset speech source refers to a speech output component or device, such as a speaker or audio system. Specifically, the step of sending the acoustic signal to the preset speech source and outputting synthesized speech includes: decoding the acoustic signal and outputting the decoding result through the speech output component to obtain the synthesized speech.

[0145] In this embodiment, the speech synthesis agent is trained by combining batch speech data of the target object before speech synthesis. Specifically, the sequence alignment component is trained for alignment, the acoustic feature generation component is trained for noise reduction, and the streaming decoding component is trained for noise reduction. This ensures that when performing text-to-speech synthesis, more accurate text-acoustic sequence alignment can be obtained, and the introduction of noise can be avoided to a great extent, thus ensuring the audio quality of the synthesized speech.

[0146] This application trains the speech synthesis agent using batch speech data of the target object before actual speech synthesis. Specifically, it trains the sequence alignment component for alignment, the acoustic feature generation component for noise reduction, and the streaming decoding component for noise reduction. This ensures that during actual text-to-speech synthesis, more accurate text-acoustic sequence alignment is achieved while minimizing noise introduction and guaranteeing the audio quality of the synthesized speech. Furthermore, the use of a sequence alignment component based on a differentiable aligner enables rapid speech synthesis. Applying this speech synthesis method to the financial customer service industry can provide customers with higher quality audio services.

[0147] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0148] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0149] In this embodiment, before actual speech synthesis, the speech synthesis agent is trained using batch speech data of the target object. Specifically, the sequence alignment component is trained for alignment, the acoustic feature generation component is trained for noise reduction, and the streaming decoding component is trained for noise reduction. This ensures that during actual text-to-speech synthesis, a more accurate text-acoustic sequence alignment effect can be obtained, and the introduction of noise can be greatly avoided, thus ensuring the audio quality of the synthesized speech. Moreover, by using a sequence alignment component based on a differentiable aligner, speech synthesis can be achieved quickly. Applying this speech synthesis method to the financial customer service industry can provide customers with higher quality and better audio service functions.

[0150] Further reference Figure 8 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a speech synthesis device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0151] like Figure 8 As shown, the speech synthesis device 800 described in this embodiment includes: a text acquisition module 801, a serialization processing module 802, a speech synthesis module 803, and a speech output module 804. Wherein:

[0152] Text acquisition module 801 is used to acquire the text to be synthesized speech;

[0153] The serialization processing module 802 is used to perform serialization processing on the speech text to be synthesized to obtain the speech text sequence to be synthesized.

[0154] The speech synthesis module 803 is used to input the speech text sequence to be synthesized into the pre-trained speech synthesis agent to obtain the acoustic signal synthesized by the speech synthesis agent.

[0155] The voice output module 804 is used to send the acoustic signal to a preset voice transmitter and output synthesized voice.

[0156] This application trains the speech synthesis agent using batch speech data of the target object before actual speech synthesis. Specifically, it trains the sequence alignment component for alignment, the acoustic feature generation component for noise reduction, and the streaming decoding component for noise reduction. This ensures that during actual text-to-speech synthesis, more accurate text-acoustic sequence alignment is achieved while minimizing noise introduction and guaranteeing the audio quality of the synthesized speech. Furthermore, the use of a sequence alignment component based on a differentiable aligner enables rapid speech synthesis. Applying this speech synthesis method to the financial customer service industry can provide customers with higher quality audio services.

[0157] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0158] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0159] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 9 , Figure 9 This is a basic structural block diagram of the computer device in this embodiment.

[0160] The computer device 9 includes a memory 9a, a processor 9b, and a network interface 9c that are interconnected via a system bus. It should be noted that only the computer device 9 with components 9a-9c is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital digital processors (DSPs), embedded devices, etc.

[0161] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0162] The memory 9a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 9a may be an internal storage unit of the computer device 9, such as the hard disk or memory of the computer device 9. In other embodiments, the memory 9a may also be an external storage device of the computer device 9, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD) card, flash memory card, etc. of the computer device 9. Of course, the memory 9a may include both the internal storage unit and the external storage device of the computer device 9. In this embodiment, the memory 9a is typically used to store the operating system and various application software installed on the computer device 9, such as computer-readable instructions for a speech synthesis method. In addition, the memory 9a can also be used to temporarily store various types of data that have been output or will be output.

[0163] In some embodiments, the processor 9b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 9b is typically used to control the overall operation of the computer device 9. In this embodiment, the processor 9b is used to execute computer-readable instructions stored in the memory 9a or to process data, such as executing computer-readable instructions for the speech synthesis method.

[0164] The network interface 9c may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 9 and other electronic devices.

[0165] The computer device proposed in this embodiment belongs to the field of speech synthesis technology and is applied in text-to-speech scenarios. This application trains the speech synthesis agent using batch speech data of the target object before actual speech synthesis. Specifically, it trains the sequence alignment component for alignment, the acoustic feature generation component for noise reduction, and the streaming decoding component for noise reduction. This ensures that during actual text-to-speech synthesis, more accurate text-acoustic sequence alignment is achieved, while significantly minimizing noise introduction and guaranteeing the audio quality of the synthesized speech. Furthermore, the use of a sequence alignment component based on a differentiable aligner enables rapid speech synthesis. Applying this speech synthesis method to the financial customer service industry can provide customers with higher quality audio services.

[0166] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the speech synthesis method described above.

[0167] The computer-readable storage medium proposed in this embodiment belongs to the field of speech synthesis technology and is applied in text-to-speech scenarios. This application trains the speech synthesis agent using batch speech data of the target object before actual speech synthesis. Specifically, it trains the sequence alignment component for alignment, the acoustic feature generation component for noise reduction, and the streaming decoding component for noise reduction. This ensures that during actual text-to-speech synthesis, more accurate text-acoustic sequence alignment is achieved while minimizing noise introduction and guaranteeing the audio quality of the synthesized speech. Furthermore, the use of a sequence alignment component based on a differentiable aligner enables rapid speech synthesis. Applying this speech synthesis method to the financial customer service industry can provide customers with higher quality audio services.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0169] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A speech synthesis method characterized by, The method comprises the following steps: obtaining a text to be synthesized; serializing the text to be synthesized to obtain a text sequence to be synthesized; inputting the text sequence to be synthesized into a pre-trained speech synthesis agent to obtain an acoustic signal synthesized by the speech synthesis agent, wherein the speech synthesis agent comprises a sequence alignment component based on a differentiable aligner, an acoustic feature generation component, and a streaming decoding processing component, and the step of inputting the text sequence to be synthesized into the pre-trained speech synthesis agent to obtain the acoustic signal synthesized by the speech synthesis agent specifically comprises: inputting the text sequence to be synthesized into the sequence alignment component to output a text sequence time-aligned with a target mel spectrum, wherein the target mel spectrum refers to a mel spectrum corresponding to the synthesized speech; mapping the text sequence time-aligned with the target mel spectrum through a preset first linear mapping layer to obtain a series of high-dimensional acoustic feature representations subject to a Gaussian distribution; inputting the series of high-dimensional acoustic feature representations subject to the Gaussian distribution into the acoustic feature generation component to generate acoustic features of the synthesized speech; inputting the acoustic features of the synthesized speech into the streaming decoding processing component to perform acoustic feature decoding to obtain a target acoustic signal; sending the acoustic signal to a preset speech sounder to output synthesized speech.

2. The speech synthesis method according to claim 1, characterized by, The acoustic feature generation component comprises a Mel acoustic feature extraction sub-component and a first decoding sub-component, and the step of inputting the series of high-dimensional acoustic feature representations subject to the Gaussian distribution into the acoustic feature generation component to generate acoustic features of the synthesized speech specifically comprises: performing acoustic feature extraction on the series of high-dimensional acoustic feature representations subject to the Gaussian distribution through the Mel acoustic feature extraction sub-component to obtain an acoustic feature extraction result; inputting the acoustic feature extraction result into the first decoding sub-component to perform decoding processing to obtain the acoustic features of the synthesized speech. The streaming decoding component comprises a sampling sub-component and a second decoding sub-component, and the step of inputting the acoustic features of the synthesized speech into the streaming decoding component to perform acoustic feature decoding to obtain a target acoustic signal specifically comprises: inputting the acoustic features of the synthesized speech into the sampling sub-component to perform Gaussian sampling to obtain sampled acoustic features; performing acoustic feature decoding on the sampled acoustic features through the second decoding sub-component to obtain the target acoustic signal.

3. The speech synthesis method of claim 2, wherein, Before performing the step of inputting the text sequence to be synthesized into the pre-trained speech synthesis agent to obtain the acoustic signal synthesized by the speech synthesis agent, the method further comprises: obtaining a batch of reference speech of a target speech user; performing noise reduction training on the acoustic feature generation component through the batch of reference speech to obtain a noise reduction trained acoustic feature generation component; and obtaining all acoustic features generated by the noise reduction trained acoustic feature generation component. All acoustic features generated by the acoustic feature generation component after the noise reduction training are used to perform noise reduction training on the streaming decoding component, to obtain a streaming decoding component after noise reduction training.

4. The speech synthesis method according to claim 3, characterized by, The acoustic feature generation component further comprises a first encoding subcomponent, and the step of performing noise reduction training on the acoustic feature generation component by using the batch of reference speeches comprises: Step 501: input the batch of reference speeches into the Mel acoustic feature extraction subcomponent to obtain acoustic feature extraction results in the form of Mel spectrum corresponding to all reference speeches respectively; Step 502: perform first noise adding processing on the acoustic feature extraction results in the form of Mel spectrum corresponding to all reference speeches respectively to obtain acoustic feature extraction results after noise adding; Step 503: perform feature encoding on the acoustic feature extraction results after noise adding by using the first encoding subcomponent to obtain acoustic feature encoding results; Step 504: input the acoustic feature encoding results into the first decoding subcomponent to perform decoding processing to obtain acoustic features corresponding to all reference speeches respectively; Step 505: map the acoustic features corresponding to all reference speeches respectively through a preset second linear mapping layer to obtain high-dimensional acoustic feature representations corresponding to all reference speeches respectively; Step 506: calculate loss values of the high-dimensional acoustic feature representations corresponding to all reference speeches respectively compared with the acoustic feature extraction results in the form of Mel spectrum by using a maximum likelihood estimation method; Step 507: if the loss values exceed a preset first loss threshold, adjust encoding and decoding parameters of the acoustic feature generation component and / or adjust mapping parameters of the second linear mapping layer, and re-execute steps 501 to 506 until the loss values do not exceed the first loss threshold, and then the acoustic feature generation component is trained to be completed; Step 508: if the loss values do not exceed the first loss threshold, obtain an acoustic feature generation component after training.

5. The speech synthesis method of claim 4, wherein, After the step of obtaining the batch of reference speeches of the target speech user, the method further comprises: Step 601: perform text serialization processing on the batch of reference speeches to obtain serialized texts corresponding to the batch of reference speeches respectively; Step 602: obtain time domain features corresponding to the batch of reference speeches respectively from the acoustic feature extraction results in the form of Mel spectrum corresponding to all reference speeches respectively as time domain features extracted by Mel spectrum; Step 603: deploy the time domain features as reference knowledge of the sequence alignment component to the sequence alignment component to obtain a sequence alignment component after deployment of reference knowledge; Step 604: input the serialized texts corresponding to the batch of reference speeches respectively into the sequence alignment component after deployment of reference knowledge to output speech text sequences aligned with target Mel spectrum in time, wherein the target Mel spectrum refers to Mel spectrum corresponding to the batch of reference speeches respectively. Step 605, mapping the speech text sequence aligned in time with the target mel spectrum through the first linear mapping layer to obtain a series of high-dimensional acoustic feature representations subject to Gaussian distribution; Step 606, extracting the time-domain features corresponding to the batch of reference speeches respectively from the series of high-dimensional acoustic feature representations subject to Gaussian distribution as the time-domain features adjusted by the sequence alignment component; Step 607, obtaining the similarity between the time-domain features adjusted by the sequence alignment component and the time-domain features extracted from the Mel spectrum through a comparison calculation method; Step 608, if the similarity does not exceed a preset similarity threshold, adjusting the alignment parameters of the sequence alignment component and / or the mapping parameters of the first linear mapping layer, and re-executing steps 604 to 607 until the similarity exceeds the preset similarity threshold, then the sequence alignment component training is completed; Step 609, if the similarity exceeds the preset similarity threshold, obtaining the trained sequence alignment component.

6. The speech synthesis method of claim 3, wherein, The streaming decoding component further includes a second encoding subcomponent, and the step of training the streaming decoding component through all acoustic features generated by the noise reduction trained acoustic feature generation component includes: Step 701, performing second noise adding processing on all acoustic features generated by the noise reduction trained acoustic feature generation component to obtain all acoustic features after noise adding; Step 702, performing feature encoding on all acoustic features after noise adding through the second encoding subcomponent to obtain acoustic feature encoding results; Step 703, inputting the acoustic feature encoding results into the second decoding subcomponent for decoding processing to obtain acoustic signals corresponding to all reference speeches respectively; Step 704, calculating loss values of acoustic signals corresponding to all reference speeches respectively compared with all acoustic features generated by the noise reduction trained acoustic feature generation component through a maximum likelihood estimation method; Step 705, if the loss value exceeds a preset second loss threshold, adjusting the encoding and decoding parameters of the streaming decoding component, and re-executing steps 701 to 704 until the loss value does not exceed the second loss threshold, then the streaming decoding component training is completed; Step 706, if the loss value does not exceed the second loss threshold, obtaining the trained streaming decoding component.

7. A speech synthesis apparatus characterized by comprising: The text acquisition module is configured to acquire a to-be-synthesized speech text. The serialization processing module is configured to perform serialization processing on the to-be-synthesized speech text to obtain a to-be-synthesized speech text sequence. The speech synthesis module is configured to input the to-be-synthesized speech text sequence into a pre-trained speech synthesis intelligent agent to obtain an acoustic signal synthesized by the speech synthesis intelligent agent, wherein the speech synthesis intelligent agent includes a sequence alignment component based on a differentiable aligner, an acoustic feature generation component, and a streaming decoding processing component, and the step of inputting the to-be-synthesized speech text sequence into the pre-trained speech synthesis intelligent agent to obtain the acoustic signal synthesized by the speech synthesis intelligent agent includes: ​ inputting a speech text sequence to be synthesized into the sequence alignment component, outputting a speech text sequence time-aligned with a target mel-spectrogram, wherein the target mel-spectrogram refers to a mel-spectrogram corresponding to the speech to be synthesized; mapping the speech text sequence time-aligned with the target mel-spectrogram through a preset first linear mapping layer to obtain a series of high-dimensional acoustic feature representations subject to Gaussian distribution; inputting the series of high-dimensional acoustic feature representations subject to Gaussian distribution into the acoustic feature generation component to generate acoustic features of the speech to be synthesized; inputting the acoustic features of the speech to be synthesized into the streaming decoding processing component to perform acoustic feature decoding to obtain a target acoustic signal; a speech output module configured to send the acoustic signal to a preset speech sounder to output synthesized speech.

8. A computer device, comprising: A computer readable storage medium having computer readable instructions stored thereon, wherein the computer readable instructions, when executed by a processor, implement the steps of the speech synthesis method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer readable storage medium having computer readable instructions stored thereon, wherein the computer readable instructions, when executed by a processor, implement the steps of the speech synthesis method according to any one of claims 1 to 6.

Images