Speech synthesis method, speech synthesis device, electronic device, and storage medium

Abstract

The application provides a speech synthesis method, a speech synthesis device, an electronic device and a storage medium, and belongs to the technical field of financial technologies. The method comprises the following steps: performing feature extraction on sample speech data to obtain reference speaker embedding features; inputting the reference speaker embedding features, the sample speech data and sample text data into a neural network model, performing feature extraction on the sample speech data to obtain preliminary speaker embedding features; performing feature fusion on the preliminary speaker embedding features and the reference speaker embedding features to obtain fused speaker embedding features; performing feature classification on the fused speaker embedding features to obtain target speaker embedding features; obtaining synthesized speech data based on the target speaker embedding features and the sample text data; training a speech synthesis model based on the synthesized speech data and the sample speech data; and synthesizing target speech data corresponding to target text data by using the speech synthesis model. The application can improve the accuracy of speech synthesis.

Description

Technical Field

[0001] This application relates to the field of financial technology, and in particular to a speech synthesis method, speech synthesis device, electronic device and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent voice interaction is being widely used in finance, logistics, customer service and other fields, improving the service level of enterprise customer service through functions such as intelligent marketing and content navigation.

[0003] Currently, chatbots are frequently used in financial service scenarios such as intelligent customer service and shopping guidance to provide corresponding service support to various users. The dialogue voice used by these chatbots is often generated through speech synthesis.

[0004] The speech synthesis of chatbots in related technologies often relies on the speaking style features of multiple reference speakers. However, when extracting the speaking style features of reference speakers from speech data, related technologies often suffer from problems such as low accuracy and incomplete feature extraction, which leads to low accuracy in the speech synthesis of chatbots. Summary of the Invention

[0005] The main objective of this application is to provide a speech synthesis method, speech synthesis device, electronic device, and storage medium, which aim to improve the accuracy of speech synthesis.

[0006] To achieve the above objectives, a first aspect of this application proposes a speech synthesis method, the method comprising:

[0007] Obtain sample speech data and corresponding sample text data;

[0008] Based on a preset feature extraction model, feature extraction is performed on the sample speech data to obtain reference speaker embedding features, wherein the reference speaker embedding features are used to characterize the speaking style of the sample object in the sample speech data;

[0009] The reference speaker embedding features, sample speech data, and sample text data are input into a preset neural network model, which includes a feature extraction network, a fully connected mapping network, and a multi-speaker classification network.

[0010] Based on the feature extraction network, feature extraction is performed on the sample speech data to obtain preliminary speaker embedding features;

[0011] Based on the fully connected mapping network, feature fusion is performed on the preliminary speaker embedding features and the reference speaker embedding features to obtain fused speaker embedding features;

[0012] Based on the multi-speaker classification network, the fused speaker embedding features are classified to obtain the target speaker embedding features;

[0013] Speech synthesis is performed based on the target speaker embedding features and the sample text data to obtain synthesized speech data;

[0014] The neural network model is updated with parameters based on the synthesized speech data and the sample speech data to obtain a speech synthesis model.

[0015] The target text data is acquired and input into the speech synthesis model to perform speech synthesis, thereby obtaining the target speech data.

[0016] In some embodiments, the feature extraction model includes a first network and a second network. The feature extraction based on the preset feature extraction model to obtain reference speaker embedding features from the sample speech data includes:

[0017] Based on the first network, the first feature is extracted from the sample speech data to obtain the first speaker embedding feature;

[0018] Based on the second network, a second feature is extracted from the sample speech data to obtain the second speaker embedding feature;

[0019] The reference speaker embedding features are obtained based on the first speaker embedding features and the second speaker embedding features.

[0020] In some embodiments, the step of extracting a first feature from the sample speech data based on the first network to obtain a first speaker embedding feature includes:

[0021] Based on the first network, the sample speech data is encoded to obtain speaker coding features;

[0022] The speaker encoding features are embedded with timbre to obtain the intermediate speaker embedding features;

[0023] The intermediate speaker embedding features are normalized to obtain the first speaker embedding features.

[0024] In some embodiments, the step of extracting second features from the sample speech data based on the second network to obtain second speaker embedding features includes:

[0025] Based on the second network, frame-level feature extraction is performed on the sample speech data to obtain sample speech frame features;

[0026] The features of the sample speech frames are encoded to obtain the features of the sample speech sentences;

[0027] The sample speech sentence features are subjected to dimensionality reduction processing to obtain low-dimensional speech sentence features;

[0028] The low-dimensional speech sentence features are normalized to obtain the second speaker embedding features.

[0029] In some embodiments, the feature fusion of the preliminary speaker embedding features and the reference speaker embedding features based on the fully connected mapping network to obtain fused speaker embedding features includes:

[0030] Based on the fully connected mapping network, the preliminary speaker embedding features are mapped to a vector space of a preset dimension to obtain the preliminary speaker embedding vector;

[0031] Based on the fully connected mapping network, the reference speaker embedding features are mapped to the vector space to obtain the reference speaker embedding vector;

[0032] In the vector space, the preliminary speaker embedding vector and the reference speaker embedding vector are fused to obtain the fused speaker embedding feature.

[0033] In some embodiments, the step of performing speech synthesis based on the target speaker embedding features and the sample text data to obtain synthesized speech data includes:

[0034] The sample text data is transformed to obtain a sample phoneme sequence;

[0035] Based on the speaker embedding features and the sample phoneme sequence, a spectrum is generated to obtain synthetic spectrum features;

[0036] The synthesized spectral features are subjected to feature transformation to obtain synthesized speech data.

[0037] In some embodiments, updating the parameters of the neural network model based on the synthesized speech data and the sample speech data to obtain a speech synthesis model includes:

[0038] Based on a preset first function, a first loss calculation is performed on the synthesized speech data and the sample speech data to obtain the spectral feature loss value;

[0039] Based on a preset second function, a second loss calculation is performed on the synthesized speech data and the sample speech data to obtain audio parameter loss values;

[0040] The neural network model is updated based on the spectral feature loss value and the audio parameter loss value to obtain a speech synthesis model.

[0041] To achieve the above objectives, a second aspect of this application provides a speech synthesis apparatus, the apparatus comprising:

[0042] The sample data acquisition module is used to acquire sample speech data and sample text data corresponding to the sample speech data.

[0043] The first feature extraction module is used to extract features from the sample speech data based on a preset feature extraction model to obtain reference speaker embedding features, wherein the reference speaker embedding features are used to characterize the speaking style of the sample object in the sample speech data.

[0044] The input module is used to input the reference speaker embedding features, sample speech data, and sample text data into a preset neural network model, wherein the neural network model includes a feature extraction network, a fully connected mapping network, and a multi-speaker classification network.

[0045] The second feature extraction module is used to extract features from the sample speech data based on the feature extraction network to obtain preliminary speaker embedding features;

[0046] The feature fusion module is used to perform feature fusion on the preliminary speaker embedding features and the reference speaker embedding features based on the fully connected mapping network to obtain fused speaker embedding features.

[0047] The feature classification module is used to classify the fused speaker embedding features based on the multi-speaker classification network to obtain the target speaker embedding features;

[0048] The first speech synthesis module is used to perform speech synthesis based on the target speaker embedding features and the sample text data to obtain synthesized speech data.

[0049] The parameter update module is used to update the parameters of the neural network model based on the synthesized speech data and the sample speech data to obtain a speech synthesis model.

[0050] The second speech synthesis module is used to acquire target text data and input the target text data into the speech synthesis model to perform speech synthesis and obtain target speech data.

[0051] To achieve the above objectives, a third aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0052] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0053] The speech synthesis method, speech synthesis device, electronic device, and storage medium proposed in this application acquire sample speech data and corresponding sample text data; extract features from the sample speech data based on a preset feature extraction model to obtain reference speaker embedding features, wherein the reference speaker embedding features are used to characterize the speaking style of the sample object in the sample speech data; input the reference speaker embedding features, sample speech data, and sample text data into a preset neural network model, the neural network model including a feature extraction network, a fully connected mapping network, and a multi-speaker classification network; extract features from the sample speech data based on the feature extraction network to obtain preliminary speaker embedding features; and then perform feature extraction on the sample speech data based on the fully connected mapping network. The process involves fusing initial speaker embedding features and reference speaker embedding features to obtain fused speaker embedding features. This method integrates feature information from speaker embedding features from different sources, improving the feature richness and quality of the fused speaker embedding features. The fused speaker embedding features are then classified using a multi-speaker classification network to obtain target speaker embedding features. Speech synthesis is performed based on the target speaker embedding features and sample text data to obtain synthesized speech data. The parameters of the neural network model are updated based on the synthesized speech data and sample speech data to obtain a speech synthesis model. This improves the model's ability to learn feature associations between the synthesized speech data and sample speech data, thus enhancing the model's speech synthesis performance. Finally, target text data is acquired and input into the speech synthesis model for speech synthesis to obtain target speech data. This improves the accuracy of speech synthesis, enabling the synthesized speech of the chatbot to better match the dialogue style preferences of the dialogue partner in intelligent dialogues involving insurance products, financial products, etc. By adopting dialogue methods and styles that are more interesting to the dialogue partner, the quality and effectiveness of dialogue are improved, enabling intelligent voice dialogue services and enhancing customer service quality and satisfaction. Attached Figure Description

[0054] Figure 1 This is a flowchart of the speech synthesis method provided in the embodiments of this application;

[0055] Figure 2 yes Figure 1 The flowchart of step S102 in the document;

[0056] Figure 3 yes Figure 2 The flowchart of step S201 in the text;

[0057] Figure 4 yes Figure 2 The flowchart of step S202 in the text;

[0058] Figure 5 yes Figure 1 The flowchart of step S105 in the process;

[0059] Figure 6 yes Figure 1 The flowchart of step S107 in the process;

[0060] Figure 7 yes Figure 1 The flowchart of step S108 in the process;

[0061] Figure 8 This is a schematic diagram of the speech synthesis device provided in the embodiments of this application;

[0062] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0066] First, let's analyze some of the terms used in this application:

[0067] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0068] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0069] Information Extraction (NER) is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.

[0070] With the rapid development of artificial intelligence technology, intelligent voice interaction is being widely used in finance, logistics, customer service and other fields, improving the service level of enterprise customer service through functions such as intelligent marketing and content navigation.

[0071] Currently, chatbots are frequently used in financial service scenarios such as intelligent customer service and shopping guidance to provide corresponding service support to various users. The dialogue voice used by these chatbots is often generated through speech synthesis.

[0072] Taking insurance service robots as an example, it is often necessary to merge the descriptive text of insurance products with the speaking style of a fixed subject to generate a voice description of the insurance product by that fixed subject. When the insurance service robot converses with some interested parties, it will automatically call upon this descriptive voice to introduce the insurance product to those parties.

[0073] The speech synthesis of chatbots in related technologies often relies on the speaking style features of multiple reference speakers. However, when extracting the speaking style features of reference speakers from speech data, related technologies often suffer from problems such as low accuracy and incomplete feature extraction, which leads to low accuracy in the speech synthesis of chatbots.

[0074] Based on this, embodiments of this application provide a speech synthesis method, a speech synthesis device, an electronic device, and a storage medium, aiming to improve the accuracy of speech synthesis.

[0075] The speech synthesis method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the speech synthesis method in this application is described.

[0076] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0077] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0078] The speech synthesis method provided in this application relates to the field of artificial intelligence technology. The speech synthesis method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speech synthesis method, but is not limited to the above forms.

[0079] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0080] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user voice data, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0081] Figure 1 This is an optional flowchart of the speech synthesis method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S109.

[0082] Step S101: Obtain sample speech data and sample text data corresponding to the sample speech data;

[0083] Step S102: Based on the preset feature extraction model, feature extraction is performed on the sample speech data to obtain reference speaker embedding features, wherein the reference speaker embedding features are used to characterize the speaking style of the sample object in the sample speech data.

[0084] Step S103: Input the reference speaker embedding features, sample speech data, and sample text data into a preset neural network model. The neural network model includes a feature extraction network, a fully connected mapping network, and a multi-speaker classification network.

[0085] Step S104: Extract features from the sample speech data based on the feature extraction network to obtain preliminary speaker embedding features;

[0086] Step S105: Based on the fully connected mapping network, feature fusion is performed on the preliminary speaker embedding features and the reference speaker embedding features to obtain the fused speaker embedding features;

[0087] Step S106: Based on the multi-speaker classification network, perform feature classification on the fused speaker embedding features to obtain the target speaker embedding features;

[0088] Step S107: Speech synthesis is performed based on the target speaker's embedded features and sample text data to obtain synthesized speech data;

[0089] Step S108: Update the parameters of the neural network model based on the synthesized speech data and sample speech data to obtain the speech synthesis model;

[0090] Step S109: Obtain the target text data and input the target text data into the speech synthesis model for speech synthesis to obtain the target speech data.

[0091] Steps S101 to S109 as shown in this embodiment involve acquiring sample speech data and corresponding sample text data; extracting features from the sample speech data based on a preset feature extraction model to obtain reference speaker embedding features, wherein the reference speaker embedding features are used to characterize the speaking style of the sample object in the sample speech data; inputting the reference speaker embedding features, sample speech data, and sample text data into a preset neural network model, which includes a feature extraction network, a fully connected mapping network, and a multi-speaker classification network; extracting features from the sample speech data based on the feature extraction network to obtain preliminary speaker embedding features; and then performing feature extraction on the preliminary speaker embedding features based on the fully connected mapping network. Feature fusion is performed on speaker embedding features and reference speaker embedding features to obtain fused speaker embedding features. This method integrates feature information from speaker embedding features from different sources, improving the feature richness and quality of the fused speaker embedding features. The fused speaker embedding features are then classified using a multi-speaker classification network to obtain target speaker embedding features. Speech synthesis is performed based on the target speaker embedding features and sample text data to obtain synthesized speech data. The parameters of the neural network model are updated based on the synthesized speech data and sample speech data to obtain a speech synthesis model. This improves the model's ability to learn feature association information from the synthesized speech data and sample speech data, thus enhancing the model's speech synthesis performance. Finally, target text data is obtained and input into the speech synthesis model for speech synthesis to obtain target speech data, which improves the accuracy of speech synthesis.

[0092] In step S101 of some embodiments, a web crawler can be written, and after setting up a data source, data can be crawled in a targeted manner to obtain sample audio data of the sample object. The data source can be various types of online platforms, social media, or specific audio databases, etc. The sample object can be an online user, speaker, singer, etc., and the sample audio data can be the sample object's music material, presentation, chat dialogue, etc. Simultaneously, sample text data corresponding to the sample audio data can be obtained from public datasets, or from existing text databases or online platforms, without limitation. The above method allows for convenient acquisition of sample audio and sample text data, improving data acquisition efficiency.

[0093] For example, in financial transaction scenarios, sample voice data is audio data containing commonly used dialogues in the financial field; in insurance sales scenarios, reference audio is audio data containing descriptions of a particular insurance product, such as its type, cost, and target audience.

[0094] Please see Figure 2In some embodiments, the feature extraction model includes a first network and a second network, and step S102 may include, but is not limited to, steps S201 to S203:

[0095] Step S201: Extract the first feature from the sample speech data based on the first network to obtain the first speaker embedding feature;

[0096] Step S202: Based on the second network, perform second feature extraction on the sample speech data to obtain the second speaker embedding features;

[0097] Step S203: Based on the first speaker embedding features and the second speaker embedding features, obtain the reference speaker embedding features.

[0098] In step S201 of some embodiments, the first network can be constructed based on the AdaIN-VC network. When performing first feature extraction on the sample speech data through the first network, the sample speech data is encoded based on the content encoder in the first network to obtain speaker encoded features, and timbre embedding is performed on the speaker encoded features to obtain intermediate speaker embedded features. Furthermore, in order to reduce the limitation of KL divergence on the intermediate speaker embedded features, the intermediate speaker embedded features can also be normalized to obtain the first speaker embedded features.

[0099] In step S202 of some embodiments, the second network can be constructed based on x-vector, d-vector, or deep-speaker networks. When extracting the second feature from the sample speech data using the second network, frame-level feature extraction is first performed on the sample speech data based on the second network to obtain sample speech frame features. Then, feature encoding is performed on the sample speech frame features to convert the frame-level features into sentence-level features to obtain sample speech sentence features. Finally, dimensionality reduction and normalization processing are performed on the sample speech sentence features to obtain the second speaker embedding features.

[0100] In step S203 of some embodiments, all first speaker embedding features and second speaker embedding features are integrated into a feature set, and all speaker embedding features in the feature set are used as reference speaker embedding features.

[0101] Through the above steps S201 to S203, different neural networks can be used to extract features from the sample speech data, resulting in speaker embedding features with different feature information. This makes the feature extraction process of the sample speech data more diversified, enriches the sources of speaker embedding features, effectively improves the feature richness and comprehensiveness of the reference speaker embedding features, and helps to improve the effect of speech synthesis.

[0102] Please see Figure 3 In some embodiments, step S201 may include, but is not limited to, steps S301 to S303:

[0103] Step S301: Encode the sample speech data based on the first network to obtain speaker coding features;

[0104] Step S302: Embed the timbre of the speaker coding features to obtain the middle speaker embedding features;

[0105] Step S303: Normalize the middle speaker embedding features to obtain the first speaker embedding features.

[0106] In step S301 of some embodiments, the sample speech data is encoded based on the first network to extract speaker information from the sample speech data and obtain speaker encoding features. The speaker information includes the speaking style characteristics of the sample object, which include the speaking speed, tone, etc. of the sample object.

[0107] In step S302 of some embodiments, a timbre parameter can be randomly called from a preset timbre database, and the speaker coding features can be embedded based on the called timbre parameter, so that the speaker coding features contain timbre information, thereby obtaining the intermediate speaker embedded features.

[0108] In step S303 of some embodiments, in order to reduce the limitation of KL divergence on the intermediate speaker embedding features, the intermediate speaker embedding features can be normalized to obtain the first speaker embedding features. This normalization process can be zero-mean normalization or maximum-minimum normalization, etc., and is not limited.

[0109] The steps S301 to S303 described above can conveniently encode the sample speech data, extract the speaker information from the sample speech data, and perform timbre embedding and normalization on the speaker information, thereby improving the comprehensiveness of the speaker feature information in the obtained first speaker embedding feature and also improving the feature quality of the obtained first speaker embedding feature.

[0110] Please see Figure 4 In some embodiments, step S202 may include, but is not limited to, steps S401 to S404:

[0111] Step S401: Based on the second network, perform frame-level feature extraction on the sample speech data to obtain sample speech frame features;

[0112] Step S402: Perform feature encoding on the sample speech frame features to obtain the sample speech sentence features;

[0113] Step S403: Dimensionality reduction is performed on the sample speech sentence features to obtain low-dimensional speech sentence features;

[0114] Step S404: Normalize the low-dimensional speech sentence features to obtain the second speaker embedding features.

[0115] In step S401 of some embodiments, frame-level feature extraction is performed on the sample speech data based on the ResCNN module or GRU module in the second network to extract speaker information from the sample speech data and obtain sample speech frame features. The speaker information includes the speaking style characteristics of the sample object, which includes the timbre, intonation, etc. of the sample object.

[0116] In step S402 of some embodiments, the sample speech frame features are encoded based on the time averaging layer of the second network, and the frame-level features are converted into sentence-level features to obtain the sample speech sentence features.

[0117] In step S403 of some embodiments, the affine transformation layer of the second network is used to perform dimensionality reduction processing on the sample speech sentence features, converting the high-dimensional sample speech sentence features into low-dimensional features to obtain low-dimensional speech sentence features.

[0118] In step S404 of some embodiments, the low-dimensional speech sentence features are normalized based on the length normalization layer of the second network, thereby shortening the feature length of the low-dimensional speech sentence features and obtaining the second speaker embedding features.

[0119] The steps S401 to S404 above can conveniently obtain sentence-level feature information of sample speech data. By reducing the feature dimension and feature length of sentence-level feature information through dimensionality reduction and length normalization, the final second speaker embedding features are more suitable for speech synthesis tasks and can improve the feature quality of the second speaker embedding features.

[0120] In step S103 of some embodiments, reference speaker embedding features, sample speech data, and sample text data can be input into a preset neural network model using a preset script program. The neural network model includes a feature extraction network, a fully connected mapping network, and a multi-speaker classification network. The feature extraction network is mainly used to extract speaker embedding features from the input speech data; the fully connected mapping network is mainly used to fuse speaker embedding features from different sources to obtain fused speaker embedding features that integrate the feature information of all speaker embedding features; the multi-speaker classification network is mainly used to distinguish the fused speaker embedding features of different speaking objects, that is, to classify the fused speaker embedding features to obtain the target speaker embedding features corresponding to each speaking object. This neural network model is mainly used to improve the feature quality of the target speaker embedding features corresponding to each speaking object, thereby using high-quality target speaker embedding features to synthesize better-quality synthesized speech data and improve the accuracy of speech synthesis.

[0121] In step S104 of some embodiments, when performing feature extraction on sample speech data based on the feature extraction network, the sample speech data is convolutionally processed to extract speaker feature information from the sample speech data and obtain preliminary speaker embedding features.

[0122] Please see Figure 5 In some embodiments, step S105 may include, but is not limited to, steps S501 to S503:

[0123] Step S501: Based on the fully connected mapping network, the preliminary speaker embedding features are mapped to a vector space of a preset dimension to obtain the preliminary speaker embedding vector.

[0124] Step S502: Based on the fully connected mapping network, the reference speaker embedding features are mapped to the vector space to obtain the reference speaker embedding vector;

[0125] Step S503: In the vector space, perform vector fusion on the preliminary speaker embedding vector and the reference speaker embedding vector to obtain the fused speaker embedding feature.

[0126] Since speaker embedding features from different sources may have different feature dimensions, it is necessary to unify the feature dimensions of all speaker embedding features to the same dimension in order to perform feature fusion.

[0127] In step S501 of some embodiments, a fully connected mapping network is used to map the preliminary speaker embedding features to a vector space of a preset dimension to obtain a preliminary speaker embedding vector. This vector space can be 512-dimensional or 256-dimensional, without limitation. For example, if the preliminary speaker embedding features are 256-dimensional features and the feature dimension of the preset dimension vector space is 256-dimensional, then the fully connected mapping network is used to map the preliminary speaker embedding features to the preset dimension vector space again, resulting in a preliminary speaker embedding vector with the same 256-dimensional dimension.

[0128] In step S502 of some embodiments, a fully connected mapping network is used to map the reference speaker embedding features to a vector space of a preset dimension to obtain reference speaker embedding vectors. Since the reference speaker embedding features include first speaker embedding features and second speaker embedding features, the first speaker embedding features and the second speaker embedding features need to be mapped separately. The first speaker embedding features are often 128-dimensional features, and the second speaker embedding features are often 256-dimensional features. Therefore, a fully connected mapping network is used to map the first speaker embedding features to the vector space, performing feature dimensionality upscaling on the first speaker embedding features; a fully connected mapping network is also used to map the second speaker embedding features to the vector space, performing feature dimensionality-invariant mapping on the second speaker embedding features, thus ensuring that the obtained reference speaker embedding vectors are all 256-dimensional.

[0129] In step S503 of some embodiments, vector fusion is performed on the preliminary speaker embedding vector and the reference speaker embedding vector in the vector space to obtain fused speaker embedding features. The vector fusion process can be vector addition or vector concatenation of the preliminary speaker embedding vector and the reference speaker embedding vector, without limitation.

[0130] Steps S501 to S503 described above can conveniently process speaker embedding features from different dimensions into features of the same dimension, enabling feature fusion of these speaker embedding features in the same dimension, thereby obtaining fused speaker embedding features containing richer feature information. Furthermore, using the fused speaker embedding features for speech synthesis can further improve the accuracy of speech synthesis.

[0131] In step S106 of some embodiments, when performing feature classification on the fused speaker embedding features based on a multi-speaker classification network to obtain the target speaker embedding features, a classification function in the multi-speaker classification network is used to score each fused speaker embedding feature. The target speaker embedding feature for each sample object is determined based on the scoring result. Specifically, the classification function can be a softmax function, etc. The probability of each fused speaker embedding feature belonging to each sample object is calculated using the classification function, and the probability is used as the scoring result. Based on the scoring result, the fused speaker embedding feature is determined as the target speaker embedding feature of the sample object with the highest probability in the scoring result. This method can improve the accuracy of feature classification and effectively distinguish the target speaker embedding features of different sample objects.

[0132] Please see Figure 6 In some embodiments, step S107 includes, but is not limited to, steps S601 to S603:

[0133] Step S601: Perform data conversion on the sample text data to obtain the sample phoneme sequence;

[0134] Step S602: Generate a spectrum based on speaker embedding features and sample phoneme sequences to obtain synthetic spectral features;

[0135] Step S603: Perform feature transformation on the synthesized spectral features to obtain synthesized speech data.

[0136] In step S601 of some embodiments, the sample text data is converted into phoneme form to obtain a sample phoneme sequence.

[0137] In step S602 of some embodiments, when generating a spectrum based on speaker embedding features and sample phoneme sequences to obtain synthetic spectral features, the speaker embedding features and sample phoneme sequences are first fused to obtain synthetic speech features, and then the synthetic speech features are decoded to obtain synthetic spectral features.

[0138] In step S603 of some embodiments, when performing feature transformation on the synthesized spectral features to obtain synthesized speech data, a preset vocoder can be used to perform feature transformation on the synthesized spectral features. This vocoder can be HiFi-GAN or MelGAN, etc., without limitation. The vocoder includes a deconvolution layer and a multi-receptive field fusion layer. Specifically, the synthesized spectral features are upsampled based on the deconvolution layer to achieve convolutional transpose of the synthesized spectral features, resulting in target spectral data with richer spectral feature content. The multi-receptive field fusion layer contains multiple residual blocks. When performing multi-scale feature fusion on the target spectral data based on the multi-receptive field fusion layer, each residual block can be used to reconstruct features of the target spectral data, obtaining speech waveform features at multiple scales. The speech waveform features at all scales are then fused to obtain synthesized speech data.

[0139] In a specific example, the synthesized speech data consists of a description of insurance or financial products, containing the speaking style and emotional tone of an animated character. This synthesized speech data can attract potential users with the unique speaking style and emotional tone of the animated character, making them more interested in the insurance or financial products recommended by the chatbot based on the synthesized speech data.

[0140] Steps S601 to S603 above enable the synthesized speech data to simultaneously include the text content features of the sample text data and the speaking style information of the speaker embedding features, thereby effectively improving the accuracy of speech synthesis.

[0141] Please see Figure 7 In some embodiments, step S108 may include, but is not limited to, steps S701 to S703:

[0142] Step S701: Perform a first loss calculation on the synthesized speech data and sample speech data based on a preset first function to obtain the spectral feature loss value;

[0143] Step S702: Perform a second loss calculation on the synthesized speech data and sample speech data based on a preset second function to obtain audio parameter loss values;

[0144] Step S703: Update the parameters of the neural network model based on the spectral feature loss value and the audio parameter loss value to obtain the speech synthesis model.

[0145] In step S701 of some embodiments, the preset first function can be a loss function based on the absolute value averaging of the differences. When performing the first loss calculation on the synthesized speech data and the sample speech data based on the preset first function, feature extraction is first performed on the synthesized speech data to obtain the synthesized Mel-Cepstral Features of the synthesized speech data, and feature extraction is performed on the sample speech data to obtain the sample Mel-Cepstral Features of the sample speech data. Then, the feature size of the sample Mel-Cepstral Features is calculated. Finally, the first loss is calculated using the first function, the sample Mel-Cepstral Features, the synthesized Mel-Cepstral Features, and the feature size to obtain the spectral feature loss value. This calculation process can be expressed as follows:

[0146]

[0147] In step S702 of some embodiments, the preset second function can be a loss function based on the average of the squared differences. When calculating the second loss based on the preset second function for the synthesized speech data and the sample speech data, firstly, feature extraction is performed on the synthesized speech data to obtain the synthesized pitch parameter, synthesized energy parameter, and synthesized duration of the synthesized speech data. Then, feature extraction is performed on the sample speech data to obtain the sample pitch parameter, sample energy parameter, and sample duration of the sample speech data. Next, the feature size of the sample speech data is calculated. Finally, the second loss is calculated using the second function, the synthesized speech data, and the sample speech data to obtain the audio parameter loss value. This calculation process can be represented as follows:

[0148]

[0149] In step S703 of some embodiments, when updating the parameters of the neural network model based on the spectral feature loss value and the audio parameter loss value to obtain the speech synthesis model, the audio parameter loss value and the spectral feature loss value are weighted and calculated to obtain the target model loss value. Further, based on the relationship between the target model loss value and a preset loss threshold, the model parameters are continuously adjusted until the target model loss value is less than the loss threshold, at which point training of the neural network model is stopped, and the speech synthesis model is obtained.

[0150] Steps S701 to S703 described above can be used to train the model based on the spectral features and speech parameters of the speech data. This improves the neural network model's ability to learn the correlation information between sample speech data and synthesized speech data, thereby improving the training effect of the model and enhancing the speech synthesis performance of the speech synthesis model.

[0151] In step S109 of some embodiments, target text data is acquired and input into a speech synthesis model for speech synthesis to obtain target speech data. The specific implementation process of this step is basically the same as that of steps S101 to S107 above, and will not be described again here for the sake of brevity.

[0152] It should be noted that the target text may contain financial terminology, financial business templates, product descriptions of insurance products and wealth management products, as well as commonly used dialogue phrases in the financial field.

[0153] The speech synthesis method of this application embodiment acquires sample speech data and corresponding sample text data; extracts features from the sample speech data based on a preset feature extraction model to obtain reference speaker embedding features, wherein the reference speaker embedding features are used to characterize the speaking style of the sample object in the sample speech data; inputs the reference speaker embedding features, sample speech data, and sample text data into a preset neural network model, the neural network model including a feature extraction network, a fully connected mapping network, and a multi-speaker classification network; extracts features from the sample speech data based on the feature extraction network to obtain preliminary speaker embedding features; and performs feature extraction on the preliminary speaker embedding features based on the fully connected mapping network. Feature fusion is performed on the speaker embedding features from different sources to obtain fused speaker embedding features. This process integrates feature information from speaker embedding features from different sources, improving the feature richness and quality of the fused speaker embedding features. The fused speaker embedding features are then classified using a multi-speaker classification network to obtain target speaker embedding features. Speech synthesis is performed based on the target speaker embedding features and sample text data to obtain synthesized speech data. The parameters of the neural network model are updated based on the synthesized speech data and sample speech data to obtain a speech synthesis model. This improves the model's ability to learn feature association information from the synthesized speech data and sample speech data, thus enhancing the model's speech synthesis performance. Finally, target text data is acquired and input into the speech synthesis model for speech synthesis to obtain target speech data. This improves the accuracy of speech synthesis, enabling the synthesized speech of the chatbot to better match the dialogue style preferences of the dialogue object during intelligent dialogues involving insurance products, financial products, etc. By adopting dialogue methods and styles that are more interesting to the dialogue object, the quality and effectiveness of dialogue are improved. This enables intelligent voice dialogue services, improving customer service quality and satisfaction, thereby increasing the success rate of financial transactions.

[0154] Please see Figure 8 This application also provides a speech synthesis apparatus that can implement the above-described speech synthesis method. The apparatus includes:

[0155] The sample data acquisition module 801 is used to acquire sample speech data and sample text data corresponding to the sample speech data;

[0156] The first feature extraction module 802 is used to extract features from the sample speech data based on a preset feature extraction model to obtain reference speaker embedding features, wherein the reference speaker embedding features are used to characterize the speaking style of the sample object in the sample speech data.

[0157] The input module 803 is used to input the reference speaker embedding features, sample speech data, and sample text data into a preset neural network model. The neural network model includes a feature extraction network, a fully connected mapping network, and a multi-speaker classification network.

[0158] The second feature extraction module 804 is used to extract features from sample speech data based on the feature extraction network to obtain preliminary speaker embedding features;

[0159] The feature fusion module 805 is used to fuse the preliminary speaker embedding features and the reference speaker embedding features based on a fully connected mapping network to obtain the fused speaker embedding features.

[0160] The feature classification module 806 is used to perform feature classification on the fused speaker embedding features based on a multi-speaker classification network to obtain the target speaker embedding features.

[0161] The first speech synthesis module 807 is used to perform speech synthesis based on the target speaker's embedded features and sample text data to obtain synthesized speech data.

[0162] The parameter update module 808 is used to update the parameters of the neural network model based on the synthesized speech data and sample speech data to obtain the speech synthesis model.

[0163] The second speech synthesis module 809 is used to acquire target text data and input the target text data into the speech synthesis model for speech synthesis to obtain target speech data.

[0164] The specific implementation of this speech synthesis device is basically the same as the specific implementation of the speech synthesis method described above, and will not be repeated here.

[0165] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned speech synthesis method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0166] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0167] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0168] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the speech synthesis method of the embodiments of this application.

[0169] The input / output interface 903 is used to implement information input and output;

[0170] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0171] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0172] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0173] This application also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the above-described speech synthesis method.

[0174] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0175] The speech synthesis method, speech synthesis device, electronic device, and computer-readable storage medium provided in this application embodiment acquire sample speech data and corresponding sample text data; extract features from the sample speech data based on a preset feature extraction model to obtain reference speaker embedding features, wherein the reference speaker embedding features are used to characterize the speaking style of the sample object in the sample speech data; input the reference speaker embedding features, sample speech data, and sample text data into a preset neural network model, the neural network model including a feature extraction network, a fully connected mapping network, and a multi-speaker classification network; extract features from the sample speech data based on the feature extraction network to obtain preliminary speaker embedding features; and then, based on the fully connected mapping network... The radio network fuses the initial speaker embedding features and the reference speaker embedding features to obtain the fused speaker embedding features. This method can integrate the feature information contained in speaker embedding features from different sources, improving the feature richness and quality of the fused speaker embedding features. Based on the multi-speaker classification network, the fused speaker embedding features are classified to obtain the target speaker embedding features. Based on the target speaker embedding features and sample text data, speech synthesis is performed to obtain synthesized speech data. Based on the synthesized speech data and sample speech data, the parameters of the neural network model are updated to obtain the speech synthesis model. This method can improve the model's ability to learn feature association information in the synthesized speech data and sample speech data, thereby improving the model's speech synthesis performance. Finally, the target text data is acquired and input into the speech synthesis model to obtain the target speech data. This improves the accuracy of speech synthesis, enabling the synthesized speech of the chatbot to better match the dialogue style preferences of the dialogue object during intelligent dialogues involving insurance products, financial products, etc. By adopting dialogue methods and styles that are more interesting to the dialogue object, the quality and effectiveness of the dialogue are improved. This enables intelligent voice dialogue services, improves customer service quality and customer satisfaction, and ultimately increases the success rate of financial transactions.

[0176] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0177] It will be understood by those skilled in the art that Figure 1-7 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0178] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0179] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0180] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0181] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0182] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0183] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0184] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0185] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0186] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech synthesis method, characterized in that, The method includes: Obtain sample speech data and corresponding sample text data; Based on a preset feature extraction model, feature extraction is performed on the sample speech data to obtain reference speaker embedding features, wherein the reference speaker embedding features are used to characterize the speaking style of the sample object in the sample speech data; The reference speaker embedding features, sample speech data, and sample text data are input into a preset neural network model, which includes a feature extraction network, a fully connected mapping network, and a multi-speaker classification network. Based on the feature extraction network, feature extraction is performed on the sample speech data to obtain preliminary speaker embedding features; Based on the fully connected mapping network, feature fusion is performed on the preliminary speaker embedding features and the reference speaker embedding features to obtain fused speaker embedding features; Based on the multi-speaker classification network, the fused speaker embedding features are classified to obtain the target speaker embedding features; Speech synthesis is performed based on the target speaker embedding features and the sample text data to obtain synthesized speech data; The neural network model is updated with parameters based on the synthesized speech data and the sample speech data to obtain a speech synthesis model. The target text data is acquired and input into the speech synthesis model to perform speech synthesis, thereby obtaining the target speech data.

2. The speech synthesis method according to claim 1, characterized in that, The feature extraction model includes a first network and a second network. The feature extraction based on the preset feature extraction model extracts features from the sample speech data to obtain reference speaker embedding features, including: Based on the first network, the first feature is extracted from the sample speech data to obtain the first speaker embedding feature; Based on the second network, a second feature is extracted from the sample speech data to obtain the second speaker embedding feature; The reference speaker embedding features are obtained based on the first speaker embedding features and the second speaker embedding features.

3. The speech synthesis method according to claim 2, characterized in that, The step of extracting a first feature from the sample speech data based on the first network to obtain a first speaker embedding feature includes: Based on the first network, the sample speech data is encoded to obtain speaker coding features; The speaker encoding features are embedded with timbre to obtain the intermediate speaker embedding features; The intermediate speaker embedding features are normalized to obtain the first speaker embedding features.

4. The speech synthesis method according to claim 2, characterized in that, The step of extracting second features from the sample speech data based on the second network to obtain second speaker embedding features includes: Based on the second network, frame-level feature extraction is performed on the sample speech data to obtain sample speech frame features; The features of the sample speech frames are encoded to obtain the features of the sample speech sentences; The sample speech sentence features are subjected to dimensionality reduction processing to obtain low-dimensional speech sentence features; The low-dimensional speech sentence features are normalized to obtain the second speaker embedding features.

5. The speech synthesis method according to claim 1, characterized in that, The step of fusing the preliminary speaker embedding features and the reference speaker embedding features based on the fully connected mapping network to obtain the fused speaker embedding features includes: Based on the fully connected mapping network, the preliminary speaker embedding features are mapped to a vector space of a preset dimension to obtain the preliminary speaker embedding vector; Based on the fully connected mapping network, the reference speaker embedding features are mapped to the vector space to obtain the reference speaker embedding vector; In the vector space, the preliminary speaker embedding vector and the reference speaker embedding vector are fused to obtain the fused speaker embedding feature.

6. The speech synthesis method according to claim 1, characterized in that, The process of synthesizing speech based on the target speaker's embedded features and the sample text data to obtain synthesized speech data includes: The sample text data is transformed to obtain a sample phoneme sequence; Based on the speaker embedding features and the sample phoneme sequence, a spectrum is generated to obtain synthetic spectrum features; The synthesized spectral features are subjected to feature transformation to obtain synthesized speech data.

7. The speech synthesis method according to any one of claims 1 to 6, characterized in that, The step of updating the parameters of the neural network model based on the synthesized speech data and the sample speech data to obtain a speech synthesis model includes: Based on a preset first function, a first loss calculation is performed on the synthesized speech data and the sample speech data to obtain the spectral feature loss value; Based on a preset second function, a second loss calculation is performed on the synthesized speech data and the sample speech data to obtain audio parameter loss values; The neural network model is updated based on the spectral feature loss value and the audio parameter loss value to obtain a speech synthesis model.

8. A speech synthesis device, characterized in that, The device includes: The sample data acquisition module is used to acquire sample speech data and sample text data corresponding to the sample speech data. The first feature extraction module is used to extract features from the sample speech data based on a preset feature extraction model to obtain reference speaker embedding features, wherein the reference speaker embedding features are used to characterize the speaking style of the sample object in the sample speech data. The input module is used to input the reference speaker embedding features, sample speech data, and sample text data into a preset neural network model, wherein the neural network model includes a feature extraction network, a fully connected mapping network, and a multi-speaker classification network. The second feature extraction module is used to extract features from the sample speech data based on the feature extraction network to obtain preliminary speaker embedding features; The feature fusion module is used to perform feature fusion on the preliminary speaker embedding features and the reference speaker embedding features based on the fully connected mapping network to obtain fused speaker embedding features. The feature classification module is used to classify the fused speaker embedding features based on the multi-speaker classification network to obtain the target speaker embedding features; The first speech synthesis module is used to perform speech synthesis based on the target speaker embedding features and the sample text data to obtain synthesized speech data. The parameter update module is used to update the parameters of the neural network model based on the synthesized speech data and the sample speech data to obtain a speech synthesis model. The second speech synthesis module is used to acquire target text data and input the target text data into the speech synthesis model to perform speech synthesis and obtain target speech data.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the speech synthesis method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech synthesis method according to any one of claims 1 to 7.

Images