Speech synthesis method and apparatus, electronic device, and storage medium

Abstract

The embodiment of the application provides a speech synthesis method and device, electronic equipment and storage medium, and belongs to the field of artificial intelligence and financial technology. The method comprises the following steps: obtaining sample speech data, sample text data, first sample synthesized speech data and sample synthesized spectrum data; performing model optimization on an original spectrum synthesis model based on the sample speech data, the sample text data, the first sample synthesized speech data and the sample synthesized spectrum data to obtain a target spectrum synthesis model; performing model optimization on an original speech synthesis model according to the sample synthesized spectrum data and the first sample synthesized speech data to obtain a target speech synthesis model; performing spectrum synthesis on reference speech data and target text data through the target spectrum synthesis model to obtain target spectrum data; and performing speech synthesis on the target spectrum data through the target speech synthesis model to obtain target synthesized speech data. The embodiment of the application can improve the efficiency of synthesized speech.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and financial technology, and particularly relates to a speech synthesis method and device, an electronic device and a storage medium. BACKGROUND

[0002] Speech synthesis is a technology of converting text input into speech output, which can be applied to various scenarios. For example, in the financial field, it is used to synthesize natural speech in scenarios such as intelligent assistants and virtual anchors.

[0003] Currently, a speech synthesis model is mainly trained using speech data of a specific speaker, and then speech synthesis is performed according to text data. However, when it is necessary to synthesize speech of other speakers, the speech synthesis model needs to be retrained, which affects the efficiency of speech synthesis.

[0004] Therefore, how to improve the efficiency of speech synthesis has become a technical problem to be solved. SUMMARY

[0005] The main purpose of the embodiments of the present application is to provide a speech synthesis method and device, an electronic device and a storage medium, which aims to improve the efficiency of speech synthesis.

[0006] To achieve the above purpose, a first aspect of the embodiments of the present application provides a speech synthesis method, which comprises:

[0007] obtaining sample speech data and sample text data; wherein the sample speech data has a sample reference speaker feature;

[0008] performing speech synthesis according to the sample reference speaker feature and the sample text data to obtain a speech synthesis sample; wherein the speech synthesis sample comprises first sample synthesized speech data and sample synthesized spectrum data;

[0009] performing spectrum synthesis on the sample speech data and the sample text data through a preset original spectrum synthesis model to obtain target sample spectrum data;

[0010] performing speech synthesis on the target sample spectrum data through a preset original speech synthesis model to obtain second sample synthesized speech data;

[0011] performing model optimization on the original spectrum synthesis model according to the sample synthesized spectrum data and the target sample spectrum data to obtain a target spectrum synthesis model;

[0012] performing model optimization on the original speech synthesis model according to the sample synthesized spectrum data, the first sample synthesized speech data and the second sample synthesized speech data to obtain a target speech synthesis model;

[0013] obtaining reference speech data of a target object; wherein the reference speech data has a target reference speaker feature of the target object;

[0014] performing spectrum synthesis on the reference speech data and pre-obtained target text data through the target spectrum synthesis model to obtain target spectrum data;

[0015] performing speech synthesis on the target spectrum data through the target speech synthesis model to obtain target synthesized speech data; wherein the target synthesized speech data has the target reference speaker feature.

[0016] In some embodiments, the original spectrum synthesis model includes an original speaker feature extraction model, an original phoneme feature extraction model, and an original mel spectrum generation model; and the performing spectrum synthesis on the sample speech data and the sample text data through the preset original spectrum synthesis model to obtain target sample spectrum data includes:

[0017] performing spectrum conversion on the sample speech data to obtain original sample spectrum data;

[0018] performing phoneme conversion on the sample text data to obtain sample phoneme data;

[0019] performing feature extraction on the original sample spectrum data through the original speaker feature extraction model to obtain sample speaker feature;

[0020] performing phoneme feature coding on the sample phoneme data through the original phoneme feature extraction model to obtain sample phoneme coding data;

[0021] splicing the sample speaker feature and the sample phoneme coding data to obtain sample synthesized phoneme data;

[0022] performing spectrum generation on the sample synthesized phoneme data through the original mel spectrum generation model to obtain target sample spectrum data.

[0023] In some embodiments, the model optimization on the original spectrum synthesis model according to the sample synthesized spectrum data and the target sample spectrum data to obtain a target spectrum synthesis model includes:

[0024] model adjustment on the original speaker feature extraction model based on the target sample spectrum data to obtain a target speaker feature extraction model;

[0025] spectrum loss calculation on the sample synthesized spectrum data and the target sample spectrum data to obtain spectrum loss data;

[0026] Parameter optimization is performed on the original phoneme feature extraction model according to the spectrum loss data, to obtain a target phoneme feature extraction model.

[0027] Parameter optimization is performed on the original mel spectrum generation model according to the spectrum loss data, to obtain a target mel spectrum generation model.

[0028] Model fusion is performed on the target phoneme feature extraction model, the target mel spectrum generation model and a target speaker feature extraction model, to obtain the target spectrum synthesis model.

[0029] In some embodiments, the original spectrum synthesis model further includes an optimized speaker feature extraction model; model adjustment is performed on the original speaker feature extraction model based on the target sample spectrum data, to obtain a target speaker feature extraction model, including:

[0030] Feature extraction is performed on the target sample spectrum data by the optimized speaker feature extraction model, to obtain synthesized speaker features;

[0031] Classification loss calculation is performed based on the synthesized speaker features, to obtain classification loss data;

[0032] Contrast loss calculation is performed on the sample speaker features and the synthesized speaker features, to obtain contrast loss data;

[0033] Parameter optimization is performed on the original speaker feature extraction model based on the classification loss data and the contrast loss data, to obtain the target speaker feature extraction model.

[0034] In some embodiments, the original speech synthesis model includes an original diffusion model and an original vocoder; speech synthesis is performed on target sample spectrum data by a preset original speech synthesis model, to obtain second sample synthesized speech data, including:

[0035] Spectrum noise reduction processing is performed on the target sample spectrum data based on the original diffusion model, to obtain noise reduction spectrum data;

[0036] Speech synthesis is performed on the noise reduction spectrum data by the original vocoder, to obtain the second sample synthesized speech data.

[0037] In some embodiments, model optimization is performed on the original speech synthesis model based on the sample synthesized spectrum data, the first sample synthesized speech data and the second sample synthesized speech data, to obtain a target speech synthesis model, including:

[0038] Noise reduction loss calculation is performed based on the noise reduction spectrum data and the sample synthesized spectrum data, to obtain noise reduction loss data;

[0039] perform a synthesis loss calculation based on the first sample synthesized speech data and the second sample synthesized speech data, to obtain speech synthesis loss data;

[0040] perform parameter optimization on the original diffusion model based on the noise reduction loss data, to obtain a target diffusion model;

[0041] perform parameter optimization on the original vocoder based on the speech synthesis loss data, to obtain a target vocoder;

[0042] perform model fusion on the target diffusion model and the target vocoder, to obtain the target speech synthesis model.

[0043] In some embodiments, the reference speech data of the target object is obtained by:

[0044] performing denoising processing on the pre-acquired original speech data, to obtain denoised speech data;

[0045] performing human voice detection on the denoised speech data, to obtain human voice speech segment information;

[0046] performing speech extraction on the denoised speech data based on the human voice speech segment information, to obtain the reference speech data.

[0047] To achieve the above-mentioned purposes, a second aspect of the embodiments of the present application proposes a speech synthesis device, the device comprising:

[0048] a sample data acquisition module, configured to acquire sample speech data and sample text data; wherein the sample speech data has a sample reference speaker feature;

[0049] a speech synthesis sample generation module, configured to perform speech synthesis according to the sample reference speaker feature and the sample text data, to obtain a speech synthesis sample; wherein the speech synthesis sample comprises first sample synthesized speech data and sample synthesized spectrum data;

[0050] a sample spectrum synthesis module, configured to perform spectrum synthesis on the sample speech data and the sample text data through a preset original spectrum synthesis model, to obtain target sample spectrum data;

[0051] a sample speech synthesis module, configured to perform speech synthesis on the target sample spectrum data through a preset original speech synthesis model, to obtain second sample synthesized speech data;

[0052] a spectrum synthesis model optimization module, configured to perform model optimization on the original spectrum synthesis model according to the sample synthesized spectrum data and the target sample spectrum data, to obtain a target spectrum synthesis model;

[0053] The voice synthesis model optimization module is configured to perform model optimization on the original voice synthesis model according to the sample spectrum data, the first sample synthesized voice data and the second sample synthesized voice data, to obtain a target voice synthesis model.

[0054] The reference voice data acquisition module is configured to acquire reference voice data of a target object, wherein the reference voice data has a target reference speaker feature of the target object.

[0055] The target spectrum synthesis module is configured to perform spectrum synthesis on the reference voice data and target text data acquired in advance by using the target spectrum synthesis model, to obtain target spectrum data.

[0056] The target voice synthesis module is configured to perform voice synthesis on the target spectrum data by using the target voice synthesis model, to obtain target synthesized voice data, wherein the target synthesized voice data has the target reference speaker feature.

[0057] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0058] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0059] The voice synthesis method and device, the electronic equipment and the storage medium provided by the present application obtain sample voice data and sample text data, wherein the sample voice data has a sample reference speaker feature; voice synthesis is performed according to the sample reference speaker feature and the sample text data to obtain a voice synthesis sample, wherein the voice synthesis sample includes first sample synthesized voice data and sample synthesized spectrum data; an original spectrum synthesis model is trained based on the sample voice data, the sample text data, the first sample synthesized voice data and the sample synthesized spectrum data to obtain a target spectrum synthesis model; the original voice synthesis model is trained based on the sample voice data, the sample text data, the first sample synthesized voice data and the sample synthesized spectrum data to obtain a target voice synthesis model, thereby improving the generalization ability of the model, enabling the model to quickly learn the features of any speaker without repeated training, and thereby performing spectrum synthesis on reference voice data and pre-acquired target text data through the target spectrum synthesis model to obtain target spectrum data; voice synthesis is performed on the target spectrum data through the target voice synthesis model to obtain target synthesized voice data, so that the target synthesized voice data has the target reference speaker feature in the reference voice data, and the efficiency of voice synthesis is improved. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is a flowchart of the voice synthesis method provided by the present application;

[0061] Figure 2 is a flowchart of step S103 in Figure 1

[0062] Figure 3 is a flowchart of step S104 in Figure 1

[0063] Figure 4 is a flowchart of step S105 in Figure 1

[0064] Figure 5 is a flowchart of step S401 in Figure 4

[0065] Figure 6 is a flowchart of step S106 in Figure 1

[0066] Figure 7 is a flowchart of step S107 in Figure 1

[0067] Figure 8 is a structural schematic diagram of the voice synthesis device provided by the present application;

[0068] Figure 9 ​​​​​​Fig. 1 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0070] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", and the like in the specification and claims and the above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0072] First, the terms involved in the present application are analyzed:

[0073] Artificial intelligence (AI): is a new technical science that studies, develops and extends human intelligence, and aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems.

[0074] Speech synthesis is a technology that converts text or symbols into speech. The speech synthesis process involves knowledge and technology in multiple fields such as speech signal processing, acoustic modeling, and speech coding. It is widely used in voice assistants, virtual character dubbing, and accessible reading.

[0075] Any speaker speech synthesis, as a high-level form of speech synthesis, aims to synthesize speech with the characteristics of a specified speaker (such as tone, intonation, etc.) based on input text and the characteristics of the speaker. The synthesized speech is almost indistinguishable from the speech of a real speaker. Any speaker speech synthesis technology has wide application prospects in personalized voice interaction, voice cloning, entertainment creation, etc.

[0076] In the financial field, speech synthesis can be applied to synthesize natural speech in intelligent assistants, virtual anchors, and other scenarios.

[0077] At present, the speech synthesis technology mainly aims at a single speaker, mainly uses the speech data of a specific speaker to train the speech synthesis model, and then performs speech synthesis according to the text data. However, when the speech of other speakers needs to be synthesized, the speech synthesis model needs to be retrained, which brings additional computational overhead and affects the efficiency of speech synthesis.

[0078] Based on this, the embodiment of the application provides a speech synthesis method and device, an electronic device and a storage medium, aiming to improve the efficiency of speech synthesis.

[0079] The speech synthesis method and device, the electronic device and the storage medium provided by the embodiment of the application are specifically explained by the following embodiments, and first, the speech synthesis method in the embodiment of the application is described.

[0080] The embodiment of the application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Theory, method, technology and application system.

[0081] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc. Several major directions.

[0082] The speech synthesis method provided by the embodiment of the application relates to the field of artificial intelligence and financial technology. The speech synthesis method provided by the embodiment of the application can be applied in a terminal, can also be applied in a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms; the software can be an application that implements the speech synthesis method, etc., but is not limited to the above forms.

[0083] The application is operable in a multitude of generic or specific computer system environments or configurations. Examples of well known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0084] It should be noted that in each specific embodiment of the present application, when it is necessary to process relevant data related to the identity or characteristics of the user according to user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to normally operate will be obtained.

[0085] Figure 1 is an optional flowchart of the speech synthesis method provided by the embodiments of the present application, Figure 1 The method in the above method can include but is not limited to including steps S101 to S109.

[0086] Step S101, obtaining sample voice data and sample text data; wherein the sample voice data has a sample reference speaker feature;

[0087] Step S102, performing speech synthesis according to the sample reference speaker feature and the sample text data to obtain a speech synthesis sample; wherein the speech synthesis sample includes first sample synthesized voice data and sample synthesized spectrum data;

[0088] Step S103, performing spectrum synthesis on the sample voice data and the sample text data through a preset original spectrum synthesis model to obtain target sample spectrum data;

[0089] Step S104, performing speech synthesis on the target sample spectrum data through a preset original speech synthesis model to obtain second sample synthesized voice data;

[0090] Step S105, model optimization is performed on the original spectrum synthesis model according to the sample synthesis spectrum data and the target sample spectrum data, to obtain a target spectrum synthesis model;

[0091] Step S106, model optimization is performed on the original speech synthesis model according to the sample synthesis spectrum data, the first sample synthesis speech data and the second sample synthesis speech data, to obtain a target speech synthesis model;

[0092] Step S107, reference speech data of the target object is obtained; the reference speech data has a target reference speaker feature of the target object;

[0093] Step S108, spectrum synthesis is performed on the reference speech data and the target text data obtained in advance by the target spectrum synthesis model, to obtain target spectrum data;

[0094] Step S109, speech synthesis is performed on the target spectrum data by the target speech synthesis model, to obtain target synthesis speech data; the target synthesis speech data has the target reference speaker feature.

[0095] The steps S101 to S109 shown in the embodiments of the present application are performed by obtaining sample speech data and sample text data; the sample speech data has a sample reference speaker feature; speech synthesis is performed according to the sample reference speaker feature and the sample text data, to obtain speech synthesis samples; the speech synthesis samples include first sample synthesis speech data and sample synthesis spectrum data; the original spectrum synthesis model is trained based on the sample speech data, the sample text data, the first sample synthesis speech data and the sample synthesis spectrum data, to obtain a target spectrum synthesis model; the original speech synthesis model is trained based on the sample speech data, the sample text data, the first sample synthesis speech data and the sample synthesis spectrum data, to obtain a target speech synthesis model, thereby improving the generalization ability of the model, enabling the model to quickly learn the features of any speaker without repeated training, so that spectrum synthesis is performed on the reference speech data and the target text data obtained in advance by the target spectrum synthesis model, to obtain target spectrum data; speech synthesis is performed on the target spectrum data by the target speech synthesis model, to obtain target synthesis speech data, so that the target synthesis speech data has the target reference speaker feature in the reference speech data, and the efficiency of speech synthesis is improved.

[0096] In step S101 of some embodiments, the sample voice data and the sample text data are pre-labeled training samples, the sample voice data has a sample reference speaker feature, and the original spectrum synthesis model and the original voice synthesis model are trained based on the sample voice data, so that the spectrum synthesis model and the voice synthesis model have the ability to learn the speaker feature from any voice data, and perform voice synthesis based on the learned speaker feature and specific text data, thereby improving the efficiency of voice synthesis.

[0097] It should be noted that the number of training samples is multiple.

[0098] In step S102 of some embodiments, the voice synthesis sample also belongs to a part of the training samples, and the voice synthesis sample including the first sample synthesized voice data and the sample synthesized spectrum data is obtained by performing voice synthesis based on the sample reference speaker feature and the sample text data. The voice synthesis sample can be used as a gold standard to evaluate the performance and accuracy of the model, thereby improving the learning ability of the spectrum synthesis model and the voice synthesis model.

[0099] In some embodiments, step S101 and step S102 can be combined as:

[0100] obtaining training samples; wherein the training samples include sample voice data, sample text data, and a voice synthesis sample, the sample voice data has a sample reference speaker feature, the voice synthesis sample is obtained by performing voice synthesis based on the sample reference speaker feature and the sample text data, and the voice synthesis sample includes first sample synthesized voice data and sample synthesized spectrum data.

[0101] In some embodiments, the original spectrum synthesis model includes an original speaker feature extraction model, an original phoneme feature extraction model, an original mel spectrum generation model, and an optimized speaker feature extraction model.

[0102] The optimized speaker feature extraction model is only used to optimize the original speaker feature extraction model in the training process, and the optimized speaker feature extraction model is not used in the use process.

[0103] In some embodiments, the backbone network of the original speaker feature extraction model and the optimized speaker feature extraction model adopts ResNet50, and the original two-dimensional convolution of ResNet50 is replaced with Depthwise convolution and pointwise convolution, and a CBAM attention mechanism is added to achieve more accurate feature extraction.

[0104] The original phoneme feature extraction model adopts an encoder of a transformer, and a variance adaptor in the Fastspeech2 is used to predict the interval, fundamental frequency and energy of the audio.

[0105] The original mel-spectrogram generation model adopts a decoder of a transformer.

[0106] The original speaker feature extraction model and the optimized speaker feature extraction model are initialized by using a pre-trained model, and specifically, the original speaker feature extraction model and the optimized speaker feature extraction model are trained as an encoder of speaker recognition.

[0107] The original phoneme feature extraction model and the original mel-spectrogram generation model are constructed by using a random parameter initialization method.

[0108] Please refer to Figure 2 In some embodiments, the step S103 can include, but is not limited to, steps S201 to S206:

[0109] In step S201, the sample speech data is converted into a spectrum to obtain original sample spectrum data.

[0110] In step S202, the sample text data is converted into a phoneme to obtain sample phoneme data.

[0111] In step S203, the original sample spectrum data is extracted by using the original speaker feature extraction model to obtain sample speaker features.

[0112] In step S204, the sample phoneme data is encoded by using the original phoneme feature extraction model to obtain sample phoneme encoded data.

[0113] In step S205, the sample speaker features and the sample phoneme encoded data are spliced to obtain sample synthesized phoneme data.

[0114] In step S206, the sample synthesized phoneme data is generated by using the original mel-spectrogram generation model to obtain target sample spectrum data.

[0115] The steps S201 to S206 shown in the embodiments of the present application convert the sample voice data into original sample spectrum data and convert the sample text data into sample phoneme data, so as to convert the training sample into a form easy for the model to read, and accurately extract the sample speaker feature from the original sample spectrum data through the original speaker feature extraction model, accurately extract the sample phoneme coding data from the sample phoneme data through the original phoneme feature extraction model, splice the sample speaker feature and the sample phoneme coding data to obtain sample synthesized phoneme data, and generate the spectrum through the original mel spectrum generation model to obtain the target sample spectrum data, which can ensure that the spectrum data not only retains the speaker feature, but also reflects the voice performance of the text content, thereby improving the realism and naturalness of voice synthesis.

[0116] Before step S201 of some embodiments, the voice synthesis method can further include but is not limited to including:

[0117] The sample voice data is subjected to denoising processing to obtain denoised voice sample data.

[0118] The denoised voice sample data is subjected to human voice detection to obtain sample human voice segment information.

[0119] The denoised voice sample data is subjected to voice extraction based on the sample human voice segment information to obtain the final sample voice data.

[0120] The background noise in the sample voice data can be effectively eliminated, and the effective human voice part in the voice can be accurately identified, thereby enhancing the reliability of the data and improving the processing efficiency of the voice data.

[0121] In step S201 of some embodiments, the sample voice data is converted from time domain to frequency domain representation through short-time Fourier transform, and then the frequency domain representation is filtered through a mel filter to obtain the original sample spectrum data.

[0122] Specifically, the original sample spectrum data can be mel spectrum data, mel inverse spectrum data, etc., without being limited thereto.

[0123] In step S202 of some embodiments, a phoneme conversion tool such as a Seq2Seq model, a convolutional neural network (CNN) model, etc. can be used to convert each word in the text data into a corresponding phoneme representation, and a voice text alignment tool can be used to calculate the duration of the phoneme representation, thereby obtaining the sample phoneme data.

[0124] In step S203 of some embodiments, the sample speaker feature includes but is not limited to a voiceprint feature, a timbre feature, a tone feature, etc., which can reflect the personality of a specific speaker.

[0125] It should be noted that there is a difference between the sample speaker feature and the sample reference speaker feature, so the original speaker feature extraction model needs to be optimized to make the original speaker feature extraction model as accurate as possible to extract the speaker feature in the speech data.

[0126] In step S205 of some embodiments, since the sample speaker feature and the sample phoneme encoding data are both in vector form, the sample speaker feature and the sample phoneme encoding data can be spliced according to a preset splicing order.

[0127] For example:

[0128] The sample speaker feature and the sample phoneme encoding data are added as vectors, that is:

[0129] Sample synthesized phoneme data = [sample speaker feature, sample phoneme encoding data];

[0130] It should be noted that the sample phoneme encoding feature has a time attribute, and the sample synthesized phoneme data obtained after data splicing also has a time attribute, so the sample synthesized phoneme data can be converted into frequency spectrum by, for example, a fast Fourier transform method, thereby obtaining target sample spectrum data.

[0131] In some embodiments, the original speech synthesis model includes an original diffusion model and an original vocoder;

[0132] The original diffusion model uses a pre-trained diffusion model to perform noise reduction, de-voicing, etc. on the spectrum data, which can improve the quality and clarity of the spectrum data and help improve the quality of speech synthesis.

[0133] The original vocoder can be a pre-trained vocoder, such as a MelGAN model, a WaveGlow model, a WaveNet model, etc., which can synthesize speech data based on spectrum data.

[0134] Referring to Figure 3 In some embodiments, step S104 can include but is not limited to steps S301 to S302:

[0135] Step S301, performing spectrum noise reduction processing on the target sample spectrum data based on the original diffusion model to obtain noise-reduced spectrum data;

[0136] Step S302, performing speech synthesis on the noise-reduced spectrum data by the original vocoder to obtain second sample synthesized speech data.

[0137] The steps S301 to S302 shown in the embodiments of the present application effectively remove the background noise and redundant information in the spectrum data by performing spectrum noise reduction processing on the target sample spectrum data, and further ensure the naturalness and accuracy of the synthesized speech by using the original vocoder to perform speech synthesis on the noise-reduced spectrum data.

[0138] In step S301 of some embodiments, the noise and accent parts in the spectrum data are determined by analyzing the target sample spectrum data, and the noise-reduced spectrum data is obtained by performing diffusion processing on the target sample spectrum data based on the noise and accent parts in the target sample spectrum data through the original diffusion model.

[0139] Specifically, the diffusion processing can be filtering processing, time domain transformation processing, etc., without being limited thereto.

[0140] Please refer to Figure 4 In some embodiments, step S105 can include but is not limited to steps S401 to S405:

[0141] Step S401, based on the target sample spectrum data, the original speaker feature extraction model is adjusted to obtain a target speaker feature extraction model;

[0142] Step S402, the sample synthesized spectrum data and the target sample spectrum data are subjected to spectrum loss calculation to obtain spectrum loss data;

[0143] Step S403, according to the spectrum loss data, the original phoneme feature extraction model is subjected to parameter optimization to obtain a target phoneme feature extraction model;

[0144] Step S404, according to the spectrum loss data, the original mel-spectrum generation model is subjected to parameter optimization to obtain a target mel-spectrum generation model;

[0145] Step S405, the target phoneme feature extraction model, the target mel-spectrum generation model and the target speaker feature extraction model are subjected to model fusion to obtain a target spectrum synthesis model.

[0146] The steps S401 to S405 shown in the embodiments of the present application obtain the target speaker feature extraction model, the target phoneme feature extraction model and the target mel-spectrum generation model by respectively optimizing the original speaker feature extraction model, the original phoneme feature extraction model and the original mel-spectrum generation model according to the target sample spectrum data, and finally fuse them into the target spectrum synthesis model, thereby improving the performance of the spectrum synthesis model and providing a solid foundation for subsequent arbitrary speaker speech synthesis.

[0147] It should be noted that when the original speaker feature extraction model in the original spectrum synthesis model is optimized, the optimized speaker feature extraction model needs to be used.

[0148] Please refer to Figure 5 In some embodiments, step S401 can further include, but is not limited to, steps S501 to S504:

[0149] In step S501, the target sample spectrum data is extracted by the optimized speaker feature extraction model to obtain synthesized speaker features.

[0150] In step S502, classification loss data is obtained by calculating the classification loss based on the synthesized speaker features.

[0151] In step S503, the sample speaker features and the synthesized speaker features are compared to calculate the contrast loss, and the contrast loss data is obtained.

[0152] In step S504, the original speaker feature extraction model is optimized based on the classification loss data and the contrast loss data to obtain the target speaker feature extraction model.

[0153] The steps S501 to S504 shown in the embodiments of the present application can more accurately capture the features of the speaker by extracting the features of the target sample spectrum data by the optimized speaker feature extraction model, and can obtain the classification loss data and the contrast loss data by calculating the loss based on the synthesized speaker features and the sample speaker features. The target speaker feature extraction model is obtained by optimizing the parameters of the original speaker feature extraction model based on the classification loss data and the contrast loss data, so that the target speaker feature extraction model can more accurately extract the speaker features, improve the robustness of the model, and provide a solid foundation for subsequent arbitrary speaker voice synthesis.

[0154] It should be noted that the synthesized speaker features are obtained by introducing the optimized speaker feature extraction model to extract the features of the target sample spectrum data, so as to optimize the original speaker feature extraction model based on the synthesized speaker features, to research and optimize the changes of the speaker features before and after the spectrum synthesis, and to ensure the quality and accuracy of the voice synthesis.

[0155] In step S502 of some embodiments, the synthesized speaker features are classified by a pre-set classification head, and the classification loss data is obtained by calculating the classification loss of the classified speaker class labels.

[0156] The classification head is used for speaker prediction on the synthesized speaker feature to obtain a speaker corresponding speaker class label, each speaker class label corresponds to a specific speaker, and the specific speaker belongs to one of all speakers in the training sample, so as to determine whether the speaker currently performing voice synthesis and the speaker in the sample voice data are the same person, and the speaker extraction capability of the original speaker feature extraction model can be optimized.

[0157] In step S503 of some embodiments, specifically, the loss function of the contrast loss calculation can be represented as:

[0158]

[0159] Wherein, q is the sample speaker feature, k + is the current sample voice data, k i is other sample voice data, K represents the number of sample voice data, and τ is a preset control factor.

[0160] Before step S504 of some embodiments, the voice synthesis method further comprises:

[0161] Based on the classification loss data and the contrast loss data, the speaker feature extraction model is adjusted in parameters.

[0162] Specifically, the back propagation method can be used to adjust the speaker feature extraction model in parameters based on the classification loss data and the contrast loss data.

[0163] In step S504 of some embodiments, the momentum update method can be used to optimize the parameters of the original speaker feature extraction model based on the classification loss data and the contrast loss data, and obtain the target speaker feature extraction model.

[0164] In step S402 of some embodiments, by performing spectrum loss calculation on the sample synthesized spectrum data and the target sample spectrum data, the model learning ability of the original phoneme feature extraction model and the original mel spectrum generation model can be evaluated, and then the original phoneme feature extraction model and the original mel spectrum generation model are adjusted in parameters based on the spectrum loss data. Wherein, the spectrum loss calculation can be calculated by using the mean square error loss function, the cross entropy loss function, the likelihood loss function, the adversarial loss function and the like, which are not limited thereto.

[0165] In step S403 of some embodiments, the gradient descent method can be used to optimize the parameters of the original phoneme feature extraction model based on the spectrum loss data, and obtain the target phoneme feature extraction model.

[0166] In step S404 of some embodiments, a gradient descent method can be used to perform parameter optimization on the original mel-spectrum generation model according to the spectrum loss data, to obtain a target mel-spectrum generation model.

[0167] Please refer to Figure 6 In some embodiments, step S106 includes but is not limited to steps S601 to S605:

[0168] In step S601, noise reduction loss calculation is performed based on the noise reduction spectrum data and the sample synthesized spectrum data, to obtain noise reduction loss data.

[0169] In step S602, synthesis loss calculation is performed based on the first sample synthesized speech data and the second sample synthesized speech data, to obtain speech synthesis loss data.

[0170] In step S603, parameter optimization is performed on the original diffusion model based on the noise reduction loss data, to obtain a target diffusion model.

[0171] In step S604, parameter optimization is performed on the original vocoder based on the speech synthesis loss data, to obtain a target vocoder.

[0172] In step S605, model fusion is performed on the target diffusion model and the target vocoder, to obtain a target speech synthesis model.

[0173] The steps S601 to S605 shown in the embodiments of the present application perform noise reduction loss calculation on the noise reduction spectrum data and the sample synthesized spectrum data to obtain noise reduction loss data, and perform synthesis loss calculation on the first sample synthesized speech data and the second sample synthesized speech data to obtain speech synthesis loss data, then perform parameter optimization on the original diffusion model based on the noise reduction loss data to obtain a target diffusion model, which improves the noise reduction capability of the diffusion model, and perform parameter optimization on the original vocoder based on the speech synthesis loss data to obtain a target vocoder, which improves the speech synthesis capability, and finally perform model fusion on the target diffusion model and the target vocoder to obtain a target speech synthesis model, which improves the overall performance of speech synthesis, thereby improving the efficiency and quality of speech synthesis.

[0174] In step S601 of some embodiments, noise reduction loss calculation is performed on the noise reduction spectrum data and the sample synthesized spectrum data, which can evaluate the noise reduction capability of the original diffusion model, and then perform parameter adjustment on the original diffusion model based on the noise reduction loss data. The noise reduction loss calculation can use loss functions such as mean square error loss function, cross-entropy loss function, likelihood loss function, and adversarial loss function, but is not limited thereto.

[0175] In step S602 of some embodiments, the synthesis loss calculation is performed on the first sample synthesized speech data and the second sample synthesized speech data, which can evaluate the speech synthesis capability of the original vocoder, and then the original vocoder is adjusted based on the speech synthesis loss data. The synthesis loss calculation can be calculated by using a mean square error loss function, a cross-entropy loss function, a likelihood loss function, an adversarial loss function, etc., and is not limited thereto.

[0176] Referring to Figure 7 In some embodiments, step S107 can include, but is not limited to, steps S701-S703.

[0177] In step S701, the pre-acquired original speech data is denoised to obtain denoised speech data.

[0178] In step S702, the denoised speech data is voice detected to obtain voice speech segment information.

[0179] In step S703, the denoised speech data is speech extracted based on the voice speech segment information to obtain reference speech data.

[0180] The steps S701-S703 shown in the embodiments of the present application can effectively eliminate the background noise in the original speech data and accurately identify the effective voice part in the speech by denoising, voice detecting and speech extracting the original speech data, thereby enhancing the reliability of the data and improving the processing efficiency of the speech data.

[0181] In some embodiments, step S702 can include, but is not limited to, the following steps:

[0182] The denoised speech data is segmented to obtain denoised speech segment data.

[0183] The denoised speech segment data is feature extracted to obtain speech segment features.

[0184] The speech segment features are voice detected to obtain voice speech segments.

[0185] The voice speech segment information is determined based on the voice speech segments.

[0186] Specifically, each speech segment feature is analyzed to obtain a voice confidence degree representing whether the speech segment feature contains voice, and the voice confidence degree is compared with a preset confidence threshold to obtain a speech segment type, which includes a voice speech segment and a silent speech segment.

[0187] Specifically, the de-noised voice segment data with a human voice confidence greater than or equal to a confidence threshold is determined as a human voice segment, and the de-noised voice segment data with a human voice confidence less than the confidence threshold is determined as a mute voice segment.

[0188] Further, the de-noised voice segment data is filtered based on the voice segment type to filter out all human voice segments.

[0189] Time information of all human voice segments is extracted to obtain human voice segment information.

[0190] In step S703 of some embodiments, voice extraction is performed on the de-noised voice data based on the time information in the human voice segment information to obtain reference voice data, wherein the reference voice data is voice data containing human voice, irrelevant noise or mute segments are removed, and the reliability of the data is enhanced.

[0191] In some embodiments, the target spectrum synthesis model includes a target speaker feature extraction model, a target phoneme feature extraction model, and a target mel spectrum generation model, and step S108 includes but is not limited to the following steps:

[0192] The reference voice data is converted into a spectrum to obtain original reference spectrum data;

[0193] The target text data is converted into a phoneme to obtain target phoneme data;

[0194] The original reference spectrum data is extracted by the target speaker feature extraction model to obtain target reference speaker features;

[0195] The target phoneme data is phoneme feature coded by the target phoneme feature extraction model to obtain target phoneme coded data;

[0196] The target reference speaker features and the target phoneme coded data are spliced to obtain target synthesized phoneme data;

[0197] The sample synthesized phoneme data is generated by the target mel spectrum generation model to obtain target spectrum data.

[0198] It should be noted that the specific implementation of step S108 is the same as the implementation shown in steps S201 to S206, and will not be repeated here.

[0199] In some embodiments, the target voice synthesis model includes a target diffusion model and a target vocoder, and step S109 includes but is not limited to the following steps:

[0200] The target spectrum data is spectrum denoising processed based on the target diffusion model to obtain target de-noised spectrum data;

[0201] The target denoising spectrum data is subjected to speech synthesis by a target vocoder to obtain target synthesized speech data.

[0202] It should be noted that the specific implementation of step S109 is the same as the implementation shown in steps S301 to S302, and will not be described again.

[0203] Please refer to Figure 8 The embodiment of the application further provides a speech synthesis device, which can implement the speech synthesis method, and the device comprises:

[0204] The sample data acquisition module 801 is configured to acquire sample speech data and sample text data, wherein the sample speech data has a sample reference speaker feature.

[0205] The speech synthesis sample generation module 802 is configured to perform speech synthesis according to the sample reference speaker feature and the sample text data to obtain a speech synthesis sample, wherein the speech synthesis sample comprises first sample synthesized speech data and sample synthesized spectrum data.

[0206] The sample spectrum synthesis module 803 is configured to perform spectrum synthesis on the sample speech data and the sample text data by using a preset original spectrum synthesis model to obtain target sample spectrum data.

[0207] The sample speech synthesis module 804 is configured to perform speech synthesis on the target sample spectrum data by using a preset original speech synthesis model to obtain second sample synthesized speech data.

[0208] The spectrum synthesis model optimization module 805 is configured to perform model optimization on the original spectrum synthesis model according to the sample synthesized spectrum data and the target sample spectrum data to obtain a target spectrum synthesis model.

[0209] The speech synthesis model optimization module 806 is configured to perform model optimization on the original speech synthesis model according to the sample synthesized spectrum data, the first sample synthesized speech data and the second sample synthesized speech data to obtain a target speech synthesis model.

[0210] The reference speech data acquisition module 807 is configured to acquire reference speech data of a target object, wherein the reference speech data has a target reference speaker feature of the target object.

[0211] The target spectrum synthesis module 808 is configured to perform spectrum synthesis on the reference speech data and target text data acquired in advance by using a target spectrum synthesis model to obtain target spectrum data.

[0212] The target speech synthesis module 809 is configured to perform speech synthesis on the target spectrum data by using a target speech synthesis model to obtain target synthesized speech data, wherein the target synthesized speech data has the target reference speaker feature.

[0213] The specific implementation of the voice synthesis device is basically the same as the specific embodiment of the voice synthesis method described above, and will not be repeated here.

[0214] The embodiments of the present application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the voice synthesis method described above when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0215] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0216] The processor 901 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0217] The memory 902 can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the voice synthesis method of the embodiments of the present application.

[0218] The input / output interface 903 is used to realize information input and output.

[0219] The communication interface 904 is used to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.).

[0220] The bus 905 transmits information between various components (such as the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0221] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 for communication within the device.

[0222] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the voice synthesis method.

[0223] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0224] The voice synthesis method and device, the electronic device and the storage medium provided by the embodiment of the present application are provided. The sample voice data and the sample text data are obtained. The sample voice data has a sample reference speaker feature. The voice synthesis is performed according to the sample reference speaker feature and the sample text data to obtain a voice synthesis sample. The voice synthesis sample includes first sample synthesized voice data and sample synthesized spectrum data. The original spectrum synthesis model is trained based on the sample voice data, the sample text data, the first sample synthesized voice data and the sample synthesized spectrum data to obtain a target spectrum synthesis model. The original voice synthesis model is trained based on the sample voice data, the sample text data, the first sample synthesized voice data and the sample synthesized spectrum data to obtain a target voice synthesis model, thereby improving the generalization ability of the model, and the characteristics of any speaker can be quickly learned without repeated training, so that the target spectrum synthesis model is used to perform spectrum synthesis on the reference voice data and the target text data obtained in advance to obtain target spectrum data. The target voice synthesis model is used to perform voice synthesis on the target spectrum data to obtain target synthesized voice data, so that the target synthesized voice data has the target reference speaker feature in the reference voice data, and the efficiency of voice synthesis is improved.

[0225] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0226] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0227] The apparatus embodiments described above are merely exemplary, and the units described as separate units can or can not be physically separate, i.e., can be located in one place, or can be distributed over multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0228] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof.

[0229] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims of the foregoing drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so

[0230] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0231] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0232] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0233] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0234] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0235] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A speech synthesis method characterized by, The method comprises: obtaining sample voice data and sample text data; wherein the sample voice data has sample reference speaker characteristics; performing voice synthesis according to the sample reference speaker characteristics and the sample text data to obtain a voice synthesis sample; wherein the voice synthesis sample comprises first sample synthesized voice data and sample synthesized spectrum data; performing spectrum synthesis on the sample voice data and the sample text data by using a preset original spectrum synthesis model to obtain target sample spectrum data; performing voice synthesis on the target sample spectrum data by using a preset original voice synthesis model to obtain second sample synthesized voice data; performing model optimization on the original spectrum synthesis model according to the sample synthesized spectrum data and the target sample spectrum data to obtain a target spectrum synthesis model; performing model optimization on the original voice synthesis model according to the sample synthesized spectrum data, the first sample synthesized voice data and the second sample synthesized voice data to obtain a target voice synthesis model; obtaining reference voice data of a target object; wherein the reference voice data has target reference speaker characteristics of the target object; performing spectrum synthesis on the reference voice data and target text data obtained in advance by using the target spectrum synthesis model to obtain target spectrum data; performing voice synthesis on the target spectrum data by using the target voice synthesis model to obtain target synthesized voice data; wherein the target synthesized voice data has the target reference speaker characteristics; The original spectrum synthesis model comprises an original speaker characteristic extraction model, an original phoneme characteristic extraction model and an original mel spectrum generation model; the spectrum synthesis on the sample voice data and the sample text data by using the preset original spectrum synthesis model to obtain the target sample spectrum data comprises: performing spectrum conversion on the sample voice data to obtain original sample spectrum data; performing phoneme conversion on the sample text data to obtain sample phoneme data; performing characteristic extraction on the original sample spectrum data by using the original speaker characteristic extraction model to obtain sample speaker characteristics; performing phoneme characteristic coding on the sample phoneme data by using the original phoneme characteristic extraction model to obtain sample phoneme coding data; splicing the sample speaker characteristics and the sample phoneme coding data to obtain sample synthesized phoneme data; performing spectrum generation on the sample synthesized phoneme data by using the original mel spectrum generation model to obtain the target sample spectrum data.

2. The method of claim 1, wherein, The model optimization on the original spectrum synthesis model according to the sample synthesized spectrum data and the target sample spectrum data to obtain a target spectrum synthesis model comprises: performing model adjustment on the original speaker characteristic extraction model based on the target sample spectrum data to obtain a target speaker characteristic extraction model; performing spectrum loss calculation on the sample synthesized spectrum data and the target sample spectrum data to obtain spectrum loss data; performing parameter optimization on the original phoneme characteristic extraction model according to the spectrum loss data to obtain a target phoneme characteristic extraction model; Parameter optimization is performed on the original mel spectrum generation model according to the spectrum loss data, so as to obtain a target mel spectrum generation model; The target phoneme feature extraction model, the target mel spectrum generation model and a target speaker feature extraction model are fused to obtain the target spectrum synthesis model.

3. The method of claim 2, wherein, The original spectrum synthesis model further comprises an optimized speaker feature extraction model; the original speaker feature extraction model is adjusted based on the target sample spectrum data to obtain a target speaker feature extraction model, comprising: Feature extraction is performed on the target sample spectrum data by the optimized speaker feature extraction model to obtain synthesized speaker features; Classification loss calculation is performed based on the synthesized speaker features to obtain classification loss data; Comparative loss calculation is performed on the sample speaker features and the synthesized speaker features to obtain comparative loss data; The original speaker feature extraction model is parameter optimized based on the classification loss data and the comparative loss data to obtain the target speaker feature extraction model.

4. The method of claim 1, wherein, The original speech synthesis model comprises an original diffusion model and an original vocoder; the target sample spectrum data is synthesized by a preset original speech synthesis model to obtain second sample synthesized speech data, comprising: The target sample spectrum data is subjected to spectrum noise reduction processing based on the original diffusion model to obtain noise reduction spectrum data; The noise reduction spectrum data is subjected to speech synthesis by the original vocoder to obtain the second sample synthesized speech data.

5. The method of claim 4, wherein, The original speech synthesis model is model optimized based on the sample synthesized spectrum data, the first sample synthesized speech data and the second sample synthesized speech data to obtain a target speech synthesis model, comprising: Noise reduction loss calculation is performed based on the noise reduction spectrum data and the sample synthesized spectrum data to obtain noise reduction loss data; Synthesis loss calculation is performed based on the first sample synthesized speech data and the second sample synthesized speech data to obtain speech synthesis loss data; The original diffusion model is parameter optimized based on the noise reduction loss data to obtain a target diffusion model; The original vocoder is parameter optimized based on the speech synthesis loss data to obtain a target vocoder; The target diffusion model and the target vocoder are fused to obtain the target speech synthesis model.

6. The method according to any one of claims 1 to 5, characterized in that, The reference speech data of the target object is obtained, comprising: The original speech data obtained in advance is subjected to noise reduction processing to obtain noise reduction speech data; The noise reduction speech data is subjected to human voice detection to obtain human voice speech segment information; The noise reduction speech data is subjected to speech extraction based on the human voice speech segment information to obtain the reference speech data.

7. A speech synthesis apparatus characterized by comprising: The device comprises: A sample data acquisition module is configured to acquire sample speech data and sample text data; wherein the sample speech data has sample reference speaker features; The speech synthesis sample generation module is configured to perform speech synthesis according to the sample reference speaker feature and the sample text data, and obtain speech synthesis samples; wherein the speech synthesis samples include first sample synthesized speech data and sample synthesized spectrum data; The sample spectrum synthesis module is configured to perform spectrum synthesis on the sample speech data and the sample text data by using a preset original spectrum synthesis model, and obtain target sample spectrum data; The sample speech synthesis module is configured to perform speech synthesis on the target sample spectrum data by using a preset original speech synthesis model, and obtain second sample synthesized speech data; The spectrum synthesis model optimization module is configured to perform model optimization on the original spectrum synthesis model according to the sample synthesized spectrum data and the target sample spectrum data, and obtain a target spectrum synthesis model; The speech synthesis model optimization module is configured to perform model optimization on the original speech synthesis model according to the sample synthesized spectrum data, the first sample synthesized speech data and the second sample synthesized speech data, and obtain a target speech synthesis model; The reference speech data acquisition module is configured to acquire reference speech data of a target object; wherein the reference speech data has a target reference speaker feature of the target object; The target spectrum synthesis module is configured to perform spectrum synthesis on the reference speech data and pre-acquired target text data by using the target spectrum synthesis model, and obtain target spectrum data; The target speech synthesis module is configured to perform speech synthesis on the target spectrum data by using the target speech synthesis model, and obtain target synthesized speech data; wherein the target synthesized speech data has the target reference speaker feature; The original spectrum synthesis model includes an original speaker feature extraction model, an original phoneme feature extraction model and an original mel spectrum generation model; the spectrum synthesis on the sample speech data and the sample text data by using the preset original spectrum synthesis model to obtain target sample spectrum data includes: performing spectrum conversion on the sample speech data to obtain original sample spectrum data; performing phoneme conversion on the sample text data to obtain sample phoneme data; performing feature extraction on the original sample spectrum data by using the original speaker feature extraction model to obtain sample speaker features; performing phoneme feature coding on the sample phoneme data by using the original phoneme feature extraction model to obtain sample phoneme coding data; splicing the sample speaker features and the sample phoneme coding data to obtain sample synthesized phoneme data; performing spectrum generation on the sample synthesized phoneme data by using the original mel spectrum generation model to obtain the target sample spectrum data.

8. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method of any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the method of any one of claims 1 to 6.

Images