Speech synthesis method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a speech synthesis method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence and finance. The method comprises the following steps: acquiring sample voice data, sample text data, first sample synthetic voice data and sample synthetic spectrum data; performing model optimization on the original spectrum synthesis model based on the sample voice data, the sample text data, the first sample synthesis voice data and the sample synthesis spectrum data to obtain a target spectrum synthesis model; performing model optimization on the original speech synthesis model according to the sample synthesis spectrum data and the first sample synthesis speech data to obtain a target speech synthesis model; performing frequency spectrum synthesis on the reference voice data and the target text data through a target frequency spectrum synthesis model to obtain target frequency spectrum data; and performing speech synthesis on the target spectrum data through the target speech synthesis model to obtain target synthesized speech data. According to the embodiment of the invention, the speech synthesis efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and financial technology, and in particular to a speech synthesis method and device, an electronic device and a storage medium. Background Art

[0002] Speech synthesis is a technology that converts text input into speech output. It can be applied in a variety of scenarios. For example, in the financial field, it is used to synthesize natural speech in scenarios such as smart assistants and virtual anchors.

[0003] At present, the speech synthesis model is mainly trained using the speech data of a specific speaker, and then speech synthesis is performed based on text data. However, when the speech of other speakers needs to be synthesized, the speech synthesis model needs to be retrained, which affects the efficiency of speech synthesis.

[0004] Therefore, how to improve the efficiency of speech synthesis has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a speech synthesis method and device, an electronic device and a storage medium, aiming to improve the efficiency of speech synthesis.

[0006] To achieve the above object, a first aspect of an embodiment of the present application provides a speech synthesis method, the method comprising:

[0007] Acquire sample speech data and sample text data; wherein the sample speech data has sample reference speaker characteristics;

[0008] Performing speech synthesis according to the sample reference speaker characteristics and the sample text data to obtain a speech synthesis sample; wherein the speech synthesis sample includes first sample synthesized speech data and sample synthesized spectrum data;

[0009] Performing spectrum synthesis on the sample speech data and the sample text data through a preset original spectrum synthesis model to obtain target sample spectrum data;

[0010] Performing speech synthesis on the target sample spectrum data through a preset original speech synthesis model to obtain second sample synthesized speech data;

[0011] Optimizing the original spectrum synthesis model according to the sample synthesis spectrum data and the target sample spectrum data to obtain a target spectrum synthesis model;

[0012] Optimizing the original speech synthesis model according to the sample synthesis spectrum data, the first sample synthesis speech data and the second sample synthesis speech data to obtain a target speech synthesis model;

[0013] Acquire reference speech data of a target object; wherein the reference speech data has target reference speaker characteristics of the target object;

[0014] Performing spectrum synthesis on the reference speech data and the pre-acquired target text data by using the target spectrum synthesis model to obtain target spectrum data;

[0015] The target spectrum data is subjected to speech synthesis by the target speech synthesis model to obtain target synthesized speech data; wherein the target synthesized speech data has the target reference speaker characteristics.

[0016] In some embodiments, the original spectrum synthesis model includes an original speaker feature extraction model, an original phoneme feature extraction model, and an original mel spectrum generation model; the spectrum synthesis of the sample speech data and the sample text data by the preset original spectrum synthesis model to obtain the target sample spectrum data includes:

[0017] Performing spectrum conversion on the sample speech data to obtain original sample spectrum data;

[0018] Performing phoneme conversion on the sample text data to obtain sample phoneme data;

[0019] Performing feature extraction on the original sample spectrum data by using the original speaker feature extraction model to obtain sample speaker features;

[0020] Performing phoneme feature encoding on the sample phoneme data by using the original phoneme feature extraction model to obtain sample phoneme encoding data;

[0021] Performing data splicing on the sample speaker feature and the sample phoneme encoding data to obtain sample synthesized phoneme data;

[0022] The spectrum of the sample synthesized phoneme data is generated by using the original mel-spectrogram generation model to obtain target sample spectrum data.

[0023] In some embodiments, the optimizing the original spectrum synthesis model according to the sample synthesis spectrum data and the target sample spectrum data to obtain the target spectrum synthesis model includes:

[0024] Adjusting the original speaker feature extraction model based on the target sample spectrum data to obtain a target speaker feature extraction model;

[0025] Performing spectrum loss calculation on the sample synthetic spectrum data and the target sample spectrum data to obtain spectrum loss data;

[0026] Optimizing the parameters of the original phoneme feature extraction model according to the spectrum loss data to obtain a target phoneme feature extraction model;

[0027] Optimizing the parameters of the original Mel-spectrogram generation model according to the spectrum loss data to obtain a target Mel-spectrogram generation model;

[0028] The target phoneme feature extraction model, the target Mel-spectrogram generation model and the target speaker feature extraction model are fused to obtain the target spectrum synthesis model.

[0029] In some embodiments, the original spectrum synthesis model further includes optimizing a speaker feature extraction model; and the step of adjusting the original speaker feature extraction model based on the target sample spectrum data to obtain a target speaker feature extraction model includes:

[0030] Extracting features from the target sample spectrum data using the optimized speaker feature extraction model to obtain synthetic speaker features;

[0031] Calculate classification loss based on the synthesized speaker features to obtain classification loss data;

[0032] Performing contrast loss calculation on the sample speaker feature and the synthesized speaker feature to obtain contrast loss data;

[0033] The parameters of the original speaker feature extraction model are optimized based on the classification loss data and the contrast loss data to obtain the target speaker feature extraction model.

[0034] In some embodiments, the original speech synthesis model includes an original diffusion model and an original vocoder; the step of performing speech synthesis on the target sample spectrum data by using the preset original speech synthesis model to obtain the second sample synthesized speech data includes:

[0035] Performing spectrum denoising processing on the target sample spectrum data based on the original diffusion model to obtain denoised spectrum data;

[0036] The noise reduction spectrum data is subjected to speech synthesis by the original vocoder to obtain the second sample synthesized speech data.

[0037] In some embodiments, the optimizing the original speech synthesis model according to the sample synthesis spectrum data, the first sample synthesis speech data and the second sample synthesis speech data to obtain a target speech synthesis model includes:

[0038] Calculate the noise reduction loss based on the noise reduction spectrum data and the sample synthetic spectrum data to obtain noise reduction loss data;

[0039] Calculate synthesis loss based on the first sample synthesized speech data and the second sample synthesized speech data to obtain speech synthesis loss data;

[0040] Optimizing the parameters of the original diffusion model based on the noise reduction loss data to obtain a target diffusion model;

[0041] Optimizing the parameters of the original vocoder based on the speech synthesis loss data to obtain a target vocoder;

[0042] The target diffusion model and the target vocoder are subjected to model fusion to obtain the target speech synthesis model.

[0043] In some embodiments, obtaining reference speech data of the target object includes:

[0044] Performing denoising processing on the original voice data acquired in advance to obtain denoised voice data;

[0045] Performing human voice detection on the denoised speech data to obtain human voice segment information;

[0046] The denoised voice data is subjected to voice extraction based on the human voice segment information to obtain the reference voice data.

[0047] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech synthesis device, the device comprising:

[0048] A sample data acquisition module, used to acquire sample voice data and sample text data; wherein the sample voice data has sample reference speaker characteristics;

[0049] A speech synthesis sample generation module, used for performing speech synthesis according to the sample reference speaker characteristics and the sample text data to obtain a speech synthesis sample; wherein the speech synthesis sample includes first sample synthesis speech data and sample synthesis spectrum data;

[0050] A sample spectrum synthesis module, used for performing spectrum synthesis on the sample speech data and the sample text data through a preset original spectrum synthesis model to obtain target sample spectrum data;

[0051] A sample speech synthesis module is used to perform speech synthesis on the target sample spectrum data through a preset original speech synthesis model to obtain second sample synthesized speech data;

[0052] A spectrum synthesis model optimization module, used for optimizing the original spectrum synthesis model according to the sample synthesis spectrum data and the target sample spectrum data to obtain a target spectrum synthesis model;

[0053] A speech synthesis model optimization module, used for optimizing the original speech synthesis model according to the sample synthesis spectrum data, the first sample synthesis speech data and the second sample synthesis speech data to obtain a target speech synthesis model;

[0054] A reference speech data acquisition module, used to acquire reference speech data of a target object; wherein the reference speech data has target reference speaker characteristics of the target object;

[0055] A target spectrum synthesis module, used for performing spectrum synthesis on the reference speech data and the pre-acquired target text data through the target spectrum synthesis model to obtain target spectrum data;

[0056] The target speech synthesis module is used to perform speech synthesis on the target spectrum data through the target speech synthesis model to obtain target synthesized speech data; wherein the target synthesized speech data has the target reference speaker characteristics.

[0057] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0058] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0059] The speech synthesis method and device, electronic device and storage medium proposed in the present application obtain sample speech data and sample text data; wherein the sample speech data has sample reference speaker characteristics; speech synthesis is performed according to the sample reference speaker characteristics and the sample text data to obtain a speech synthesis sample; wherein the speech synthesis sample includes first sample synthesized speech data and sample synthesized spectrum data; an original spectrum synthesis model is trained based on the sample speech data, the sample text data, the first sample synthesized speech data and the sample synthesized spectrum data to obtain a target spectrum synthesis model; the original speech synthesis model is trained based on the sample speech data, the sample text data, the first sample synthesized speech data and the sample synthesized spectrum data to obtain a target speech synthesis model, thereby improving the generalization ability of the model, and being able to quickly learn the characteristics of any speaker without repeated training, so that the reference speech data and the pre-acquired target text data are spectrum synthesized by the target spectrum synthesis model to obtain the target spectrum data; speech synthesis is performed on the target spectrum data by the target speech synthesis model to obtain the target synthesized speech data, so that the target synthesized speech data has the target reference speaker characteristics in the reference speech data, thereby improving the efficiency of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart of a speech synthesis method provided in an embodiment of the present application;

[0061] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.

[0062] Figure 3 yes Figure 1 Flow chart of step S104 in FIG.

[0063] Figure 4 yes Figure 1 Flow chart of step S105 in FIG.

[0064] Figure 5 yes Figure 4 Flow chart of step S401 in FIG.

[0065] Figure 6 yes Figure 1 Flow chart of step S106 in FIG.

[0066] Figure 7 yes Figure 1 Flow chart of step S107 in FIG.

[0067] Figure 8 is a structural schematic diagram of a speech synthesis device provided in an embodiment of the present application;

[0068] Fig. 9It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0070] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0072] First, some nouns involved in this application are analyzed:

[0073] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems.

[0074] Speech synthesis is a technology that converts text or symbols into speech. The speech synthesis process involves knowledge and technologies in multiple fields such as speech signal processing, acoustic modeling, and speech coding. It is widely used in voice assistants, virtual character dubbing, barrier-free reading and other fields.

[0075] As an advanced form of speech synthesis, any-speaker speech synthesis aims to synthesize speech with the characteristics of the speaker based on the input text and the characteristics of the specified speaker (such as timbre, intonation, etc.). The synthesized speech sounds almost the same as the real speaker's speech. Any-speaker speech synthesis technology has broad application prospects in the fields of personalized voice interaction, voice cloning, entertainment creation, etc.

[0076] In the financial field, speech synthesis can be used to synthesize natural speech in scenarios such as smart assistants and virtual anchors.

[0077] At present, speech synthesis technology is mainly aimed at a single speaker, mainly using the speech data of a specific speaker to train the speech synthesis model, and then perform speech synthesis based on text data. However, when it is necessary to synthesize the speech of other speakers, the speech synthesis model needs to be retrained, which brings additional computing overhead and affects the efficiency of speech synthesis.

[0078] Based on this, the embodiments of the present application provide a speech synthesis method and device, an electronic device and a storage medium, aiming to improve the efficiency of speech synthesis.

[0079] The speech synthesis method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the speech synthesis method in the embodiments of the present application is described.

[0080] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0081] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0082] The speech synthesis method provided in the embodiment of the present application relates to the field of artificial intelligence and financial technology. The speech synthesis method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speech synthesis method, etc., but is not limited to the above forms.

[0083] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0084] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0085] Figure 1 is an optional flowchart of the speech synthesis method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S109.

[0086] Step S101, obtaining sample speech data and sample text data; wherein the sample speech data has sample reference speaker characteristics;

[0087] Step S102, performing speech synthesis according to the sample reference speaker characteristics and the sample text data to obtain a speech synthesis sample; wherein the speech synthesis sample includes first sample synthesized speech data and sample synthesized spectrum data;

[0088] Step S103, performing spectrum synthesis on the sample speech data and the sample text data through a preset original spectrum synthesis model to obtain target sample spectrum data;

[0089] Step S104, performing speech synthesis on the target sample spectrum data through a preset original speech synthesis model to obtain second sample synthesized speech data;

[0090] Step S105, optimizing the original spectrum synthesis model according to the sample synthesis spectrum data and the target sample spectrum data to obtain a target spectrum synthesis model;

[0091] Step S106, optimizing the original speech synthesis model according to the sample synthesis spectrum data, the first sample synthesis speech data and the second sample synthesis speech data to obtain a target speech synthesis model;

[0092] Step S107, obtaining reference speech data of the target object; wherein the reference speech data has target reference speaker characteristics of the target object;

[0093] Step S108, performing spectrum synthesis on the reference speech data and the pre-acquired target text data through a target spectrum synthesis model to obtain target spectrum data;

[0094] Step S109, performing speech synthesis on the target spectrum data through a target speech synthesis model to obtain target synthesized speech data; wherein the target synthesized speech data has characteristics of a target reference speaker.

[0095] In the steps S101 to S109 shown in the embodiment of the present application, sample speech data and sample text data are obtained; wherein the sample speech data has sample reference speaker characteristics; speech synthesis is performed according to the sample reference speaker characteristics and the sample text data to obtain a speech synthesis sample; wherein the speech synthesis sample includes first sample synthesized speech data and sample synthesized spectrum data; an original spectrum synthesis model is trained based on the sample speech data, the sample text data, the first sample synthesized speech data and the sample synthesized spectrum data to obtain a target spectrum synthesis model; the original speech synthesis model is trained based on the sample speech data, the sample text data, the first sample synthesized speech data and the sample synthesized spectrum data to obtain a target speech synthesis model, thereby improving the generalization ability of the model, and being able to quickly learn the characteristics of any speaker without repeated training, thereby performing spectrum synthesis on the reference speech data and the pre-acquired target text data through the target spectrum synthesis model to obtain the target spectrum data; speech synthesis is performed on the target spectrum data through the target speech synthesis model to obtain the target synthesized speech data, so that the target synthesized speech data has the target reference speaker characteristics in the reference speech data, thereby improving the efficiency of speech synthesis.

[0096] In step S101 of some embodiments, the sample speech data and the sample text data are pre-labeled training samples, and the sample speech data have sample reference speaker characteristics, which are used to train the original spectrum synthesis model and the original speech synthesis model, so that the spectrum synthesis model and the speech synthesis model are capable of learning speaker characteristics through any speech data, and performing speech synthesis based on the learned speaker characteristics and specific text data, thereby improving the efficiency of speech synthesis.

[0097] It should be noted that there are multiple training samples.

[0098] In step S102 of some embodiments, the speech synthesis sample is also part of the training sample. By performing speech synthesis according to the sample reference speaker characteristics and sample text data, a speech synthesis sample including first sample synthesized speech data and sample synthesized spectrum data is obtained. The speech synthesis sample can be used as a gold standard to evaluate the performance and accuracy of the model, thereby improving the learning ability of the spectrum synthesis model and the speech synthesis model.

[0099] In some embodiments, step S101 and step S102 may be combined as follows:

[0100] Acquire training samples; wherein the training samples include sample speech data, sample text data, and speech synthesis samples, the sample speech data has sample reference speaker characteristics, the speech synthesis samples are obtained by performing speech synthesis based on the sample reference speaker characteristics and the sample text data, and the speech synthesis samples include first sample synthesized speech data and sample synthesized spectrum data.

[0101] In some embodiments, the original spectrum synthesis model includes an original speaker feature extraction model, an original phoneme feature extraction model, an original mel spectrum generation model, and an optimized speaker feature extraction model;

[0102] Among them, the optimized speaker feature extraction model is only used to optimize the original speaker feature extraction model during the training process; and the optimized speaker feature extraction model is not used during use.

[0103] In some embodiments, the backbone networks of the original speaker feature extraction model and the optimized speaker feature extraction model both use ResNet50, and the original two-dimensional convolution of ResNet50 is replaced by Depthwise convolution and pointwise convolution, and a CBAM attention mechanism is added to achieve more accurate feature extraction;

[0104] The original phoneme feature extraction model uses the transformer encoder and the Variance adaptor in Fastspeech2 to predict the interval, fundamental frequency, and energy of the audio.

[0105] The original Mel spectrum generation model uses the transformer decoder.

[0106] The original speaker feature extraction model and the optimized speaker feature extraction model are initialized with parameters using a pre-trained model. Specifically, the original speaker feature extraction model and the optimized speaker feature extraction model can be trained as encoders for speaker recognition.

[0107] The original phoneme feature extraction model and the original mel-spectrogram generation model are constructed by randomly initializing parameters.

[0108] See also Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S206:

[0109] Step S201, performing spectrum conversion on the sample speech data to obtain original sample spectrum data;

[0110] Step S202, performing phoneme conversion on the sample text data to obtain sample phoneme data;

[0111] Step S203, extracting features from the original sample spectrum data using the original speaker feature extraction model to obtain sample speaker features;

[0112] Step S204, performing phoneme feature encoding on the sample phoneme data through the original phoneme feature extraction model to obtain sample phoneme encoding data;

[0113] Step S205, concatenating the sample speaker features and the sample phoneme encoding data to obtain sample synthesized phoneme data;

[0114] Step S206, generating a spectrum for the sample synthesized phoneme data through the original mel-spectrogram generation model to obtain target sample spectrum data.

[0115] Steps S201 to S206 shown in the embodiment of the present application convert the sample speech data into original sample spectrum data and convert the sample text data into sample phoneme data, thereby converting the training sample into a form that is easy for the model to read, and accurately extracting sample speaker features from the original sample spectrum data through the original speaker feature extraction model, accurately extracting sample phoneme encoding data from the sample phoneme data through the original phoneme feature extraction model, splicing the sample speaker features and the sample phoneme encoding data to obtain sample synthetic phoneme data, and performing spectrum generation on the sample synthetic phoneme data through the original Mel-spectrogram generation model to obtain target sample spectrum data, which can ensure that the spectrum data reflects the voice performance of the text content while retaining the speaker characteristics, thereby improving the realism and naturalness of speech synthesis.

[0116] Before step S201 in some embodiments, the speech synthesis method may also include but is not limited to:

[0117] Performing denoising processing on the sample speech data to obtain denoised speech sample data;

[0118] Performing voice detection on the denoised speech sample data to obtain sample voice segment information;

[0119] Based on the sample human voice segment information, voice extraction is performed on the denoised voice sample data to obtain the final sample voice data.

[0120] It can effectively eliminate background noise in sample speech data and accurately identify the effective human voice part in the speech, thereby enhancing the reliability of the data and improving the processing efficiency of speech data.

[0121] In step S201 of some embodiments, the sample speech data is converted from the time domain to the frequency domain by performing short-time Fourier transform on the sample speech data, and the frequency domain representation is filtered by a Mel filter to obtain the original sample spectrum data.

[0122] Specifically, the original sample spectrum data may be mel spectrum data, mel cepstrum data, etc., but is not limited thereto.

[0123] In step S202 of some embodiments, a phoneme conversion tool, such as a Seq2Seq model, a convolutional neural network (CNN) model, etc., can be used to convert each word in the text data into a corresponding phoneme representation, and the duration of the phoneme representation is calculated using a speech-to-text alignment tool to obtain sample phoneme data.

[0124] In step S203 of some embodiments, the sample speaker characteristics include, but are not limited to: voiceprint characteristics, timbre characteristics, intonation characteristics, etc., which can reflect the personality of a specific speaker.

[0125] It should be noted that there are differences between the sample speaker features and the sample reference speaker features. Therefore, it is necessary to optimize the original speaker feature extraction model so that the original speaker feature extraction model can extract the speaker features in the speech data as accurately as possible.

[0126] In step S205 of some embodiments, since both the sample speaker features and the sample phoneme encoding data are in vector form, the sample speaker features and the sample phoneme encoding data may be concatenated according to a preset concatenation order.

[0127] For example:

[0128] The sample speaker features and sample phoneme encoding data are vectorized and added, that is:

[0129] Sample synthesized phoneme data = [sample speaker characteristics, sample phoneme encoding data];

[0130] It should be noted that the sample phoneme encoding feature has a time attribute, and the sample synthetic phoneme data obtained after data splicing also has a time attribute. Therefore, the sample synthetic phoneme data can be spectrally converted by, for example, the fast Fourier transform method to obtain the target sample spectrum data.

[0131] In some embodiments, the original speech synthesis model includes an original diffusion model and an original vocoder;

[0132] Among them, the original diffusion model uses a pre-trained diffusion model to perform operations such as noise reduction and de-emphasis on spectral data, which can improve the quality and clarity of spectral data and help improve the quality of speech synthesis.

[0133] The original vocoder may be a pre-trained vocoder, such as a MelGAN model, a WaveGlow model, a WaveNet model, etc., which can synthesize speech data based on spectrum data.

[0134] See also Figure 3 In some embodiments, step S104 may include but is not limited to steps S301 to S302:

[0135] Step S301, performing spectrum denoising processing on the target sample spectrum data based on the original diffusion model to obtain denoised spectrum data;

[0136] Step S302: Perform speech synthesis on the noise reduction spectrum data through the original vocoder to obtain second sample synthesized speech data.

[0137] In steps S301 to S302 shown in the embodiment of the present application, by performing spectral noise reduction processing on the target sample spectral data, background noise and redundant information in the spectral data are effectively removed, and then the original vocoder is used to perform speech synthesis on the denoised spectral data, thereby further ensuring the naturalness and accuracy of the synthesized speech.

[0138] In step S301 of some embodiments, the target sample spectrum data is analyzed to determine the noise and accent parts in the spectrum data, and the target sample spectrum data is diffused based on the noise and accent parts in the target sample spectrum data through an original diffusion model to obtain noise-reduced spectrum data.

[0139] Specifically, the diffusion processing may be filtering processing, time domain transformation processing, etc., but is not limited thereto.

[0140] See also Figure 4 In some embodiments, step S105 may include but is not limited to steps S401 to S405:

[0141] Step S401, adjusting the original speaker feature extraction model based on the target sample spectrum data to obtain a target speaker feature extraction model;

[0142] Step S402, performing spectrum loss calculation on the sample synthetic spectrum data and the target sample spectrum data to obtain spectrum loss data;

[0143] Step S403, optimizing the parameters of the original phoneme feature extraction model according to the spectrum loss data to obtain a target phoneme feature extraction model;

[0144] Step S404, optimizing the parameters of the original Mel-spectrogram generation model according to the spectrum loss data to obtain a target Mel-spectrogram generation model;

[0145] Step S405 , performing model fusion on the target phoneme feature extraction model, the target mel-spectrogram generation model and the target speaker feature extraction model to obtain a target spectrum synthesis model.

[0146] In steps S401 to S405 shown in the embodiment of the present application, the original speaker feature extraction model, the original phoneme feature extraction model, and the original mel-spectrogram generation model are optimized according to the target sample spectrum data to obtain the target speaker feature extraction model, the target phoneme feature extraction model, and the target mel-spectrogram generation model, and finally merge them into a target spectrum synthesis model, thereby improving the performance of the spectrum synthesis model and providing a solid foundation for subsequent speech synthesis of any speaker.

[0147] It should be noted that when optimizing the original speaker feature extraction model in the original spectrum synthesis model, the optimized speaker feature extraction model is required.

[0148] See also Figure 5 In some embodiments, step S401 may also include but is not limited to steps S501 to S504:

[0149] Step S501, extracting features from the target sample spectrum data by optimizing the speaker feature extraction model to obtain synthetic speaker features;

[0150] Step S502, calculating classification loss based on the synthesized speaker features to obtain classification loss data;

[0151] Step S503, performing contrast loss calculation on the sample speaker features and the synthesized speaker features to obtain contrast loss data;

[0152] Step S504 , optimizing the parameters of the original speaker feature extraction model based on the classification loss data and the contrast loss data to obtain a target speaker feature extraction model.

[0153] In steps S501 to S504 shown in the embodiment of the present application, by optimizing the speaker feature extraction model to extract features from the target sample spectrum data, the speaker's features can be captured more accurately, and loss calculation is performed based on the synthesized speaker features and the sample speaker features to obtain classification loss data and contrast loss data; based on the classification loss data and the contrast loss data, the parameters of the original speaker feature extraction model are optimized to obtain the target speaker feature extraction model, so that the target speaker feature extraction model can extract speaker features more accurately, improve the robustness of the model, and provide a solid foundation for subsequent arbitrary speaker speech synthesis.

[0154] It should be noted that by introducing an optimized speaker feature extraction model to extract features from the target sample spectrum data, synthetic speaker features are obtained, and then the original speaker feature extraction model is optimized based on the synthetic speaker features, so that the changes in speaker features before and after spectrum synthesis can be studied and optimized, thereby ensuring the quality and accuracy of speech synthesis.

[0155] In step S502 of some embodiments, the synthesized speaker features are classified by a preset classification head, and the classification loss is calculated for the classified speaker category labels to obtain classification loss data;

[0156] Among them, the classification head is used to perform speaker prediction on the synthesized speaker features to obtain the speaker category label corresponding to the speaker. Each speaker category label corresponds to a specific speaker, and the specific speaker belongs to one of all the speakers in the training sample. In this way, it is determined whether the speaker currently performing speech synthesis is the same person as the speaker in the sample speech data, and the speaker extraction capability of the initial speaker feature extraction model can be optimized.

[0157] In step S503 of some embodiments, specifically, the loss function of the contrast loss calculation can be expressed as:

[0158]

[0159] Among them, q is the sample speaker feature, k + is the current sample speech data, k i are other sample speech data, K represents the number of sample speech data, and τ is a preset control factor.

[0160] Before step S504 in some embodiments, the speech synthesis method further includes:

[0161] Parameters of the optimized speaker feature extraction model are adjusted based on the classification loss data and the contrast loss data.

[0162] Specifically, a back-propagation method may be used to adjust parameters of the optimized speaker feature extraction model based on classification loss data and contrast loss data.

[0163] In step S504 of some embodiments, a momentum update method may be used to optimize parameters of the original speaker feature extraction model based on the classification loss data and the contrast loss data to obtain a target speaker feature extraction model.

[0164] In step S402 of some embodiments, by performing spectrum loss calculation on the sample synthetic spectrum data and the target sample spectrum data, the model learning ability of the original phoneme feature extraction model and the original Mel spectrum generation model can be evaluated, and then the parameters of the original phoneme feature extraction model and the original Mel spectrum generation model can be adjusted based on the spectrum loss data. The spectrum loss calculation can be calculated using a loss function such as a mean square error loss function, a cross entropy loss function, a likelihood loss function, an adversarial loss function, etc., but is not limited thereto.

[0165] In step S403 of some embodiments, a gradient descent method may be used to optimize parameters of the original phoneme feature extraction model based on the spectrum loss data to obtain a target phoneme feature extraction model.

[0166] In step S404 of some embodiments, a gradient descent method may be used to optimize parameters of the original mel-spectrogram generation model according to the spectrum loss data to obtain a target mel-spectrogram generation model.

[0167] See also Figure 6 In some embodiments, step S106 includes but is not limited to steps S601 to S605:

[0168] Step S601, performing noise reduction loss calculation based on the noise reduction spectrum data and the sample synthetic spectrum data to obtain noise reduction loss data;

[0169] Step S602, performing synthesis loss calculation based on the first sample synthesized speech data and the second sample synthesized speech data to obtain speech synthesis loss data;

[0170] Step S603, optimizing the parameters of the original diffusion model based on the noise reduction loss data to obtain a target diffusion model;

[0171] Step S604, optimizing parameters of the original vocoder based on the speech synthesis loss data to obtain a target vocoder;

[0172] Step S605, performing model fusion on the target diffusion model and the target vocoder to obtain a target speech synthesis model.

[0173] In steps S601 to S605 shown in the embodiment of the present application, noise reduction loss data is obtained by performing noise reduction loss calculation on noise reduction spectrum data and sample synthesis spectrum data, and synthesis loss calculation is performed on first sample synthesis speech data and second sample synthesis speech data to obtain speech synthesis loss data. Then, parameters of the original diffusion model are optimized based on the noise reduction loss data to obtain a target diffusion model, thereby improving the noise reduction capability of the diffusion model. Meanwhile, parameters of the original vocoder are optimized based on the speech synthesis loss data to obtain a target vocoder, thereby improving the speech synthesis capability. Finally, the target diffusion model and the target vocoder are model-fused to obtain a target speech synthesis model, thereby improving the overall performance of speech synthesis, thereby improving the efficiency and quality of speech synthesis.

[0174] In step S601 of some embodiments, the noise reduction loss is calculated by the noise reduction spectrum data and the sample synthetic spectrum data, so that the noise reduction ability of the original diffusion model can be evaluated, and then the parameters of the original diffusion model can be adjusted based on the noise reduction loss data. The noise reduction loss calculation can be calculated using a mean square error loss function, a cross entropy loss function, a likelihood loss function, an adversarial loss function, and other loss functions, but is not limited thereto.

[0175] In step S602 of some embodiments, the synthesis loss calculation is performed by using the first sample synthesized speech data and the second sample synthesized speech data, so that the speech synthesis capability of the original vocoder can be evaluated, and then the parameters of the original vocoder can be adjusted based on the speech synthesis loss data. The synthesis loss calculation can be performed using a loss function such as a mean square error loss function, a cross entropy loss function, a likelihood loss function, and an adversarial loss function, but is not limited thereto.

[0176] See also Figure 7 In some embodiments, step S107 may include but is not limited to steps S701 to S703:

[0177] Step S701, performing denoising processing on the original voice data acquired in advance to obtain denoised voice data;

[0178] Step S702, performing human voice detection on the denoised speech data to obtain human voice segment information;

[0179] Step S703: perform voice extraction on the denoised voice data based on the human voice segment information to obtain reference voice data.

[0180] Steps S701 to S703 shown in the embodiment of the present application can effectively eliminate background noise in the original voice data and accurately identify the effective voice part in the voice by performing denoising, voice detection and voice extraction on the original voice data, thereby enhancing the reliability of the data and improving the processing efficiency of the voice data.

[0181] In some embodiments, step S702 may include but is not limited to the following steps:

[0182] Segmenting the denoised speech data to obtain denoised speech segment data;

[0183] Extract features from the denoised speech segment data to obtain speech segment features;

[0184] Performing human voice detection on the features of the speech segment to obtain a human voice speech segment;

[0185] The human voice segment information is determined based on the human voice segment.

[0186] Specifically, each speech segment feature is analyzed to obtain a voice confidence level that characterizes whether the speech segment feature contains a human voice, and the voice confidence level is compared with a preset confidence threshold to obtain a speech segment type, which includes a human voice segment and a silent speech segment.

[0187] Specifically, the denoised speech segment data whose human voice confidence is greater than or equal to the confidence threshold is determined as a human voice segment, and the denoised speech segment data whose human voice confidence is less than the confidence threshold is determined as a silent speech segment;

[0188] Then, the denoised speech segment data is segmented based on the speech segment type to screen out all human voice segments;

[0189] The time information of all vocal segments is extracted to obtain the vocal speech segment information.

[0190] In step S703 of some embodiments, based on the time information in the human voice segment information, voice extraction is performed on the denoised voice data to obtain reference voice data, wherein the reference voice data is voice data containing human voice, and irrelevant noise or silence segments are removed, thereby enhancing the reliability of the data.

[0191] In some embodiments, the target spectrum synthesis model includes: a target speaker feature extraction model, a target phoneme feature extraction model and a target mel-spectrogram generation model. Step S108 includes but is not limited to the following steps:

[0192] Performing spectrum conversion on the reference speech data to obtain original reference spectrum data;

[0193] Performing phoneme conversion on the target text data to obtain target phoneme data;

[0194] The target speaker feature extraction model is used to extract features from the original reference spectrum data to obtain target reference speaker features;

[0195] Performing phoneme feature encoding on target phoneme data through a target phoneme feature extraction model to obtain target phoneme encoding data;

[0196] Concatenate the target reference speaker characteristics and the target phoneme encoding data to obtain the target synthesized phoneme data;

[0197] The target mel-spectrogram generation model is used to generate the spectrum of the sample synthesized phoneme data to obtain the target spectrum data.

[0198] It should be noted that the specific implementation of the above step S108 is the same as the implementation shown in steps S201 to S206, and will not be repeated here.

[0199] In some embodiments, the target speech synthesis model includes: a target diffusion model and a target vocoder, and step S109 includes but is not limited to the following steps:

[0200] Performing spectrum denoising on the target spectrum data based on the target diffusion model to obtain target denoised spectrum data;

[0201] The target denoised spectrum data is subjected to speech synthesis by a target vocoder to obtain target synthesized speech data.

[0202] It should be noted that the specific implementation of the above step S109 is the same as the implementation shown in steps S301 to S302, and will not be repeated here.

[0203] See also Figure 8 The present application also provides a speech synthesis device, which can implement the above-mentioned speech synthesis method, and the device includes:

[0204] The sample data acquisition module 801 is used to acquire sample speech data and sample text data; wherein the sample speech data has sample reference speaker characteristics;

[0205] The speech synthesis sample generation module 802 is used to perform speech synthesis according to the sample reference speaker characteristics and the sample text data to obtain a speech synthesis sample; wherein the speech synthesis sample includes first sample synthesis speech data and sample synthesis spectrum data;

[0206] The sample spectrum synthesis module 803 is used to perform spectrum synthesis on the sample speech data and the sample text data through a preset original spectrum synthesis model to obtain target sample spectrum data;

[0207] The sample speech synthesis module 804 is used to perform speech synthesis on the target sample spectrum data through a preset original speech synthesis model to obtain second sample synthesized speech data;

[0208] The spectrum synthesis model optimization module 805 is used to optimize the original spectrum synthesis model according to the sample synthesis spectrum data and the target sample spectrum data to obtain the target spectrum synthesis model;

[0209] The speech synthesis model optimization module 806 is used to optimize the original speech synthesis model according to the sample synthesis spectrum data, the first sample synthesis speech data and the second sample synthesis speech data to obtain a target speech synthesis model;

[0210] A reference speech data acquisition module 807 is used to acquire reference speech data of a target object; wherein the reference speech data has target reference speaker characteristics of the target object;

[0211] A target spectrum synthesis module 808 is used to perform spectrum synthesis on the reference speech data and the pre-acquired target text data through a target spectrum synthesis model to obtain target spectrum data;

[0212] The target speech synthesis module 809 is used to perform speech synthesis on the target spectrum data through a target speech synthesis model to obtain target synthesized speech data; wherein the target synthesized speech data has the characteristics of a target reference speaker.

[0213] The specific implementation of the speech synthesis device is basically the same as the specific embodiment of the above-mentioned speech synthesis method, and will not be repeated here.

[0214] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned speech synthesis method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0215] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0216] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0217] The memory 902 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 may store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the speech synthesis method of the embodiment of this application;

[0218] Input / output interface 903, used to implement information input and output;

[0219] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0220] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0221] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0222] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned speech synthesis method is implemented.

[0223] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0224] The speech synthesis method and device, electronic device and storage medium provided by the embodiments of the present application obtain sample speech data and sample text data; wherein the sample speech data has sample reference speaker characteristics; speech synthesis is performed according to the sample reference speaker characteristics and the sample text data to obtain a speech synthesis sample; wherein the speech synthesis sample includes first sample synthesized speech data and sample synthesized spectrum data; an original spectrum synthesis model is trained based on the sample speech data, the sample text data, the first sample synthesized speech data and the sample synthesized spectrum data to obtain a target spectrum synthesis model; the original speech synthesis model is trained based on the sample speech data, the sample text data, the first sample synthesized speech data and the sample synthesized spectrum data to obtain a target speech synthesis model, thereby improving the generalization ability of the model, and being able to quickly learn the characteristics of any speaker without repeated training, so that the reference speech data and the pre-acquired target text data are spectrum synthesized by the target spectrum synthesis model to obtain the target spectrum data; speech synthesis is performed on the target spectrum data by the target speech synthesis model to obtain the target synthesized speech data, so that the target synthesized speech data has the target reference speaker characteristics in the reference speech data, thereby improving the efficiency of speech synthesis.

[0225] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0226] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0227] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0228] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0229] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0230] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0231] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0232] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0233] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0234] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0235] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Acquire sample speech data and sample text data; wherein the sample speech data has sample reference speaker characteristics; Performing speech synthesis according to the sample reference speaker characteristics and the sample text data to obtain a speech synthesis sample; wherein the speech synthesis sample includes first sample synthesized speech data and sample synthesized spectrum data; Performing spectrum synthesis on the sample speech data and the sample text data through a preset original spectrum synthesis model to obtain target sample spectrum data; Performing speech synthesis on the target sample spectrum data through a preset original speech synthesis model to obtain second sample synthesized speech data; Optimizing the original spectrum synthesis model according to the sample synthesis spectrum data and the target sample spectrum data to obtain a target spectrum synthesis model; Optimizing the original speech synthesis model according to the sample synthesis spectrum data, the first sample synthesis speech data and the second sample synthesis speech data to obtain a target speech synthesis model; Acquire reference speech data of a target object; wherein the reference speech data has target reference speaker characteristics of the target object; Performing spectrum synthesis on the reference speech data and the pre-acquired target text data by using the target spectrum synthesis model to obtain target spectrum data; The target spectrum data is subjected to speech synthesis by the target speech synthesis model to obtain target synthesized speech data; wherein the target synthesized speech data has the target reference speaker characteristics.

2. The method according to claim 1, characterized in that: The original spectrum synthesis model includes an original speaker feature extraction model, an original phoneme feature extraction model and an original mel spectrum generation model; the sample speech data and the sample text data are spectrum synthesized by the preset original spectrum synthesis model to obtain target sample spectrum data, including: Performing spectrum conversion on the sample speech data to obtain original sample spectrum data; Performing phoneme conversion on the sample text data to obtain sample phoneme data; Performing feature extraction on the original sample spectrum data by using the original speaker feature extraction model to obtain sample speaker features; Performing phoneme feature encoding on the sample phoneme data by using the original phoneme feature extraction model to obtain sample phoneme encoding data; Performing data splicing on the sample speaker feature and the sample phoneme encoding data to obtain sample synthesized phoneme data; The spectrum of the sample synthesized phoneme data is generated by using the original mel-spectrogram generation model to obtain target sample spectrum data.

3. The method according to claim 2, characterized in that The optimizing the original spectrum synthesis model according to the sample synthesis spectrum data and the target sample spectrum data to obtain the target spectrum synthesis model includes: Adjusting the original speaker feature extraction model based on the target sample spectrum data to obtain a target speaker feature extraction model; Performing spectrum loss calculation on the sample synthetic spectrum data and the target sample spectrum data to obtain spectrum loss data; Optimizing the parameters of the original phoneme feature extraction model according to the spectrum loss data to obtain a target phoneme feature extraction model; Optimizing the parameters of the original Mel-spectrogram generation model according to the spectrum loss data to obtain a target Mel-spectrogram generation model; The target phoneme feature extraction model, the target Mel-spectrogram generation model and the target speaker feature extraction model are fused to obtain the target spectrum synthesis model.

4. The method according to claim 3, characterized in that The original spectrum synthesis model also includes an optimized speaker feature extraction model; the original speaker feature extraction model is adjusted based on the target sample spectrum data to obtain a target speaker feature extraction model, including: Extracting features from the target sample spectrum data using the optimized speaker feature extraction model to obtain synthetic speaker features; Calculate classification loss based on the synthesized speaker features to obtain classification loss data; Performing contrast loss calculation on the sample speaker feature and the synthesized speaker feature to obtain contrast loss data; The parameters of the original speaker feature extraction model are optimized based on the classification loss data and the contrast loss data to obtain the target speaker feature extraction model.

5. The method according to claim 1, characterized in that: The original speech synthesis model includes an original diffusion model and an original vocoder; the speech synthesis of the target sample spectrum data is performed by the preset original speech synthesis model to obtain the second sample synthesized speech data, including: Performing spectrum denoising processing on the target sample spectrum data based on the original diffusion model to obtain denoised spectrum data; The noise reduction spectrum data is subjected to speech synthesis by the original vocoder to obtain the second sample synthesized speech data.

6. The method according to claim 5, characterized in that The step of optimizing the original speech synthesis model according to the sample synthesis spectrum data, the first sample synthesis speech data and the second sample synthesis speech data to obtain a target speech synthesis model includes: Calculate the noise reduction loss based on the noise reduction spectrum data and the sample synthetic spectrum data to obtain noise reduction loss data; Calculate synthesis loss based on the first sample synthesized speech data and the second sample synthesized speech data to obtain speech synthesis loss data; Optimizing the parameters of the original diffusion model based on the noise reduction loss data to obtain a target diffusion model; Optimizing the parameters of the original vocoder based on the speech synthesis loss data to obtain a target vocoder; The target diffusion model and the target vocoder are subjected to model fusion to obtain the target speech synthesis model.

7. The method according to any one of claims 1 to 6, characterized in that: The step of obtaining reference speech data of the target object includes: Performing denoising processing on the original voice data acquired in advance to obtain denoised voice data; Performing human voice detection on the denoised speech data to obtain human voice segment information; The denoised voice data is subjected to voice extraction based on the human voice segment information to obtain the reference voice data.

8. A speech synthesis device, characterized in that: The device comprises: A sample data acquisition module, used to acquire sample voice data and sample text data; wherein the sample voice data has sample reference speaker characteristics; A speech synthesis sample generation module, used for performing speech synthesis according to the sample reference speaker characteristics and the sample text data to obtain a speech synthesis sample; wherein the speech synthesis sample includes first sample synthesis speech data and sample synthesis spectrum data; A sample spectrum synthesis module, used for performing spectrum synthesis on the sample speech data and the sample text data through a preset original spectrum synthesis model to obtain target sample spectrum data; A sample speech synthesis module is used to perform speech synthesis on the target sample spectrum data through a preset original speech synthesis model to obtain second sample synthesized speech data; A spectrum synthesis model optimization module, used for optimizing the original spectrum synthesis model according to the sample synthesis spectrum data and the target sample spectrum data to obtain a target spectrum synthesis model; A speech synthesis model optimization module, used for optimizing the original speech synthesis model according to the sample synthesis spectrum data, the first sample synthesis speech data and the second sample synthesis speech data to obtain a target speech synthesis model; A reference speech data acquisition module, used to acquire reference speech data of a target object; wherein the reference speech data has target reference speaker characteristics of the target object; A target spectrum synthesis module, used for performing spectrum synthesis on the reference speech data and the pre-acquired target text data through the target spectrum synthesis model to obtain target spectrum data; The target speech synthesis module is used to perform speech synthesis on the target spectrum data through the target speech synthesis model to obtain target synthesized speech data; wherein the target synthesized speech data has the target reference speaker characteristics.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images