Speech synthesis method, speech synthesis device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a speech synthesis method, a speech synthesis device, electronic equipment and a storage medium, and belongs to the technical field of financial science and technology. The method comprises the following steps: performing speech synthesis on a sample text and a sample speech with a sample rhythm feature based on an initial speech synthesis model to obtain an initial synthesized speech with a predicted rhythm feature; acquiring voice feedback data of the initial synthetic voice; performing rhythm optimization on the voice feedback data based on a rhythm optimization model to obtain rhythm adjustment data; performing parameter adjustment on initial speech feature distribution parameters of the initial speech synthesis model based on the rhythm adjustment data, and adjusting the initial speech synthesis model based on the adjusted speech feature distribution parameters to obtain a target speech synthesis model; and performing speech synthesis processing on the target text and the target speech with the target rhythm characteristics based on the target speech synthesis model to obtain a target synthesized speech. According to the embodiment of the invention, synthetic speech with more accurate rhythm expression can be generated.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to a speech synthesis method, a speech synthesis device, an electronic device and a storage medium. Background Art

[0002] Speech synthesis refers to the technology of converting a given text into a natural and understandable speech. At present, with the widespread use of intelligent speech technology in smart phone customer service, smart sales and other task scenarios in financial technology, synthesizing expressive speech is the focus of current research. In order to synthesize synthetic speech that is closer to the expression of real people's speech, related technologies usually learn the mapping from the input prompt text to the Mel spectrogram or discrete audio token (Token) to generate synthetic speech that meets the speech requirements corresponding to the prompt text. However, the combination diversity of the rhythm, time and spectral features of speech makes the speech expression of real people also diverse, so the generated synthetic speech should also be rich, and this learned mapping relationship is difficult to provide a synthetic speech that is more in line with the expression of real people's speech based on the same prompt text, making the rhythmic expression of the generated synthetic speech inaccurate. Therefore, how to generate synthetic speech with more accurate rhythmic expression has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a speech synthesis method, a speech synthesis device, an electronic device and a storage medium, which can generate synthesized speech with more accurate prosodic expression.

[0004] To achieve the above object, a first aspect of an embodiment of the present application provides a speech synthesis method, the method comprising:

[0005] Acquire sample data, the sample data including sample text and sample speech, the sample text having text content, and the sample speech having sample prosodic features;

[0006] Performing speech synthesis processing on the sample text and the sample speech based on a preset initial speech synthesis model to obtain an initial synthesized speech with predicted prosodic features, wherein the speech content of the initial synthesized speech is the text content;

[0007] Acquiring speech feedback data of the initial synthesized speech; wherein the speech feedback data is used to characterize the degree of difference between the sample prosodic feature and the predicted prosodic feature;

[0008] Performing prosody optimization detection on the speech feedback data based on a preset prosody optimization model to obtain prosody adjustment data;

[0009] Performing parameter adjustment on initial speech feature distribution parameters of the initial speech synthesis model based on the prosody adjustment data to obtain adjusted speech feature distribution parameters;

[0010] Adjust the initial speech synthesis model based on the adjusted speech feature distribution parameters to obtain a target speech synthesis model;

[0011] Based on the target speech synthesis model, speech synthesis processing is performed on the target text and the target speech with the target prosodic features to obtain the target synthesized speech with the target prosodic features.

[0012] In some embodiments, the obtaining of speech feedback data of the initial synthesized speech includes:

[0013] Obtaining a textual description of the feedback of the evaluation subject on the initial synthesized speech;

[0014] Performing feedback text detection on the feedback description text based on a preset feedback text detection model to obtain an initial feedback score;

[0015] Optimizing the initial feedback score based on the evaluation object to obtain an object feedback score;

[0016] The voice feedback data is obtained based on the feedback description text and the object feedback score.

[0017] In some embodiments, the step of optimizing the initial feedback score based on the evaluation object to obtain the object feedback score includes:

[0018] Acquiring the subject identity data and the subject voice preference degree of the evaluation subject;

[0019] Determining a voice optimization weight of the evaluation object based on the object identity data and the object voice preference degree;

[0020] The initial feedback score is optimized based on the speech optimization weight to obtain the object feedback score.

[0021] In some embodiments, adjusting the initial speech synthesis model based on the adjusted speech feature distribution parameters to obtain a target speech synthesis model includes:

[0022] Determining a sample feedback category of the sample data based on the speech feedback score;

[0023] Determining current sample distribution data for training the initial speech synthesis model based on the sample feedback category;

[0024] Determine sample distribution deviation data based on preset sample distribution data and the current sample distribution data;

[0025] The initial speech synthesis model is adjusted based on the adjusted speech feature distribution parameters and the sample distribution deviation data to obtain the target speech synthesis model.

[0026] In some embodiments, the sample data further includes a preset sample category, and the adjusting the initial speech synthesis model based on the adjusted speech feature distribution parameter and the sample distribution deviation data to obtain the target speech synthesis model includes:

[0027] Performing category loss calculation based on the preset sample category and the sample feedback category to obtain category loss data;

[0028] The initial speech synthesis model is adjusted based on the adjusted speech feature distribution parameters, the sample distribution deviation data and the category loss data to obtain the target speech synthesis model.

[0029] In some embodiments, the prosody adjustment data includes speech rate adjustment sub-data, volume adjustment sub-data, pitch adjustment sub-data and emotion adjustment sub-data, and the initial speech feature distribution parameters include initial speech rate distribution sub-parameters, initial volume distribution sub-parameters, initial pitch distribution sub-parameters and initial emotion distribution sub-parameters;

[0030] The step of adjusting the initial speech feature distribution parameters of the initial speech synthesis model based on the prosody adjustment data to obtain the adjusted speech feature distribution parameters includes:

[0031] Performing speech rate parameter adjustment on the initial speech rate distribution sub-parameter based on the speech rate adjustment sub-data to obtain an adjusted speech rate distribution sub-parameter;

[0032] Performing volume parameter adjustment on the initial volume distribution sub-parameter based on the volume adjustment sub-data to obtain adjusted volume distribution sub-parameter;

[0033] Performing pitch parameter adjustment on the initial pitch distribution sub-parameter based on the pitch adjustment sub-data to obtain adjusted pitch distribution sub-parameter;

[0034] Performing emotion parameter adjustment on the initial emotion distribution sub-parameter based on the emotion adjustment sub-data to obtain adjusted emotion distribution sub-parameter;

[0035] The adjusted speech feature distribution parameter is obtained based on the adjusted speech rate distribution sub-parameter, the adjusted volume distribution sub-parameter, the adjusted pitch distribution sub-parameter and the adjusted emotion distribution sub-parameter.

[0036] In some embodiments, the prosody optimization model includes an encoding unit, an aggregation unit, and a generation unit, and the prosody optimization detection is performed on the speech feedback data based on a preset prosody optimization model to obtain prosody adjustment data, including:

[0037] Performing encoding processing on the speech feedback data based on the encoding unit to obtain a speech feedback vector;

[0038] Performing vector aggregation processing on the speech feedback vector based on the aggregation unit to obtain an aggregation vector;

[0039] The generating unit performs rhythm optimization on the aggregated vector to obtain the rhythm adjustment data.

[0040] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech synthesis device, the device comprising:

[0041] A first acquisition module is used to acquire sample data, wherein the sample data includes sample text and sample speech, wherein the sample text has text content, and the sample speech has sample prosodic features;

[0042] A first synthesis module is used to perform speech synthesis processing on the sample text and the sample speech based on a preset initial speech synthesis model to obtain an initial synthesized speech with predicted prosodic features, wherein the speech content of the initial synthesized speech is the text content;

[0043] A second acquisition module is used to acquire speech feedback data of the initial synthesized speech; wherein the speech feedback data is used to characterize the degree of difference between the sample prosodic feature and the predicted prosodic feature;

[0044] A detection module, configured to perform a prosody optimization detection on the speech feedback data based on a preset prosody optimization model to obtain prosody adjustment data;

[0045] A parameter adjustment module, used for adjusting the initial speech feature distribution parameters of the initial speech synthesis model based on the prosody adjustment data to obtain adjusted speech feature distribution parameters;

[0046] A model adjustment module, used for adjusting the initial speech synthesis model based on the adjusted speech feature distribution parameters to obtain a target speech synthesis model;

[0047] The second synthesis module is used to perform speech synthesis processing on the target text and the target speech with the target prosodic features based on the target speech synthesis model to obtain the target synthesized speech with the target prosodic features.

[0048] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements any method proposed in the first aspect of the embodiment of the present application.

[0049] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application further proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements any method proposed in the first aspect of the embodiments of the present application.

[0050] The speech synthesis method, speech synthesis device, electronic device and storage medium proposed in the embodiments of the present application obtain sample data, wherein the sample data includes sample text and sample speech, the sample text has text content, and the sample speech has sample prosodic features; based on a preset initial speech synthesis model, speech synthesis processing is performed on the sample text and the sample speech to obtain an initial synthesized speech with predicted prosodic features, and the speech content of the initial synthesized speech is text content; further, speech feedback data of the initial synthesized speech is obtained; wherein the speech feedback data is used to characterize the degree of difference between the sample prosodic features and the predicted prosodic features; further, prosodic optimization detection is performed on the speech feedback data based on a preset prosodic optimization model to obtain prosodic adjustment data; further, initial speech feature distribution parameters of the initial speech synthesis model are parameter adjusted based on the prosodic adjustment data to obtain adjusted speech feature distribution parameters; further, the initial speech synthesis model is adjusted based on the adjusted speech feature distribution parameters to obtain a target speech synthesis model; finally, speech synthesis processing is performed on the target text and the target speech with target prosodic features based on the target speech synthesis model to obtain a target synthesized speech with target prosodic features. When training the target speech synthesis model, the embodiment of the present application simultaneously inputs the sample speech and sample text with the sample prosodic features into the initial speech synthesis model, so that the synthesized initial synthesized speech can learn the sample prosodic features. In addition, the present application will combine the speech feedback data and the prosodic optimization model to determine the prosodic adjustment direction of the initial speech feature distribution parameters, so that the difference between the predicted prosodic features of the synthesized speech and the sample prosodic features is smaller, so that when the target speech synthesis model provided by the embodiment of the present application performs speech synthesis processing on the target text and the target speech with the target prosodic features, a more accurate prosodic expression of the synthesized speech can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the speech synthesis method provided in an embodiment of the present application;

[0052] Figure 2It is a schematic diagram of a process of training a speech synthesis model provided in an embodiment of the present application;

[0053] Figure 3 yes Figure 1 A flowchart of the specific method of step S130;

[0054] Figure 4 yes Figure 3 A flowchart of the specific method of step S330;

[0055] Figure 5 yes Figure 1 A flowchart of the specific method of step S140;

[0056] Figure 6 yes Figure 1 A flowchart of the specific method of step S150;

[0057] Figure 7 yes Figure 1 A flowchart of the specific method of step S160;

[0058] Figure 8 yes Figure 7 A flowchart of the specific method of step S740;

[0059] Fig. 9 It is a module structure block diagram of the speech synthesis device provided in an embodiment of the present application;

[0060] Fig.10 It is a hardware structure diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0064] First, some nouns involved in this application are analyzed:

[0065] Artificial Intelligence (AI): is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0066] Natural Language Processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text comprehension. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information retrieval, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. Natural language processing involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0067] Text-To-Speech (TTS): is a technology that converts text into speech. TTS generally includes two steps: the first step is text processing, which is mainly to convert text into a phoneme sequence and mark the start and end time, frequency change and other information of each phoneme; the second step is speech synthesis, which is mainly to generate speech based on the phoneme sequence (as well as the marked start and end time, frequency change and other information).

[0068] Speech rhythm: also known as prosody, refers to the changes in pitch, duration, intensity, tone, etc. of speech, which is the expression of the rhythm and rhythm of speech. Rhythm can include intonation, pauses and rhythm, which can convey the speaker's emotions, attitudes and information. The role of speech rhythm can express emotions (that is, different emotions can be expressed through changes in pitch, duration, intensity and tone), distinguish meanings (that is, in some languages, the same speech unit can express different meanings through changes in rhythm), enhance the beauty of language (that is, in art forms such as poetry and songs, the use of rhythm can make the language more beautiful and pleasant to the ear), etc.

[0069] Mean Opinion Score (MOS): is a subjective evaluation method for measuring audio or video quality. MOS can collect the opinions of a group of people, rate the quality of audio or video, and then calculate the average of these scores to obtain an objective audio or video quality evaluation result.

[0070] Speech synthesis refers to the technology of converting a given text into a natural and understandable speech. At present, with the widespread use of intelligent speech technology in smart phone customer service, smart sales and other task scenarios in financial technology, synthesizing expressive speech is the focus of current research. In order to synthesize synthetic speech that is closer to the expression of real human speech, related technologies usually learn the mapping from the input prompt text to the Mel spectrogram or discrete audio token (Token) to generate synthetic speech that meets the speech requirements corresponding to the prompt text. However, this learned mapping is difficult to provide diversified generation based on the same text and voice prompt (Prompt), which hinders the application of large language model (LargeLanguageModel ls, LLMs) alignment methods based on paired preference data. This is because the current TTS input is either pure text, or a prompt audio is provided at the same time as the text input to guide the generation effect of the model (similar to PromptTTS). However, these two methods are based on the existing TTS model and cannot synthesize speech with diversity such as real human speech. For example, if a real person reads the same text 100 times, the sound effect will be different each time (speech speed, volume, pitch, emotion, etc.), but the current TTS basically synthesizes the same effect.

[0071] In addition, the diversity of the combination of speech rhythm, time and spectrum features makes the real person's speech expression also diverse, so the generated synthetic speech should also be rich. However, this learned mapping relationship is difficult to provide a synthetic speech that is more in line with the real person's speech expression based on the same prompt text, making the rhythm expression of the generated synthetic speech inaccurate. Therefore, how to generate synthetic speech with more accurate rhythm expression has become a technical problem that needs to be solved urgently.

[0072] Based on this, the speech synthesis method, speech synthesis device, electronic device and storage medium provided in the embodiments of the present application can generate synthesized speech with more accurate prosodic expression.

[0073] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0074] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0075] The speech synthesis method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The speech synthesis method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, or a smart watch, etc.; the server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (ContentDe l ivery Network, CDN), and big data and artificial intelligence platforms; the software can be an application that implements the speech synthesis method, etc., but is not limited to the above forms.

[0076] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0077] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the identity or characteristics of the object, such as object information, object voice data, object historical data, and object identity information, the permission or consent of the object will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain sensitive personal information of the object, the separate permission or separate consent of the object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the object, the necessary object-related data for the normal operation of the embodiment of the present application will be obtained.

[0078] Please refer to Figure 1 , Figure 1 is an optional flow chart of the speech synthesis method provided in the embodiment of the present application. In some embodiments of the present application, the speech synthesis method in the embodiment of the present application includes but is not limited to steps S110 to S170. Figure 1 These seven steps are introduced in detail.

[0079] Step S110, obtaining sample data, the sample data including sample text and sample speech;

[0080] Step S120, performing speech synthesis processing on the sample text and the sample speech based on a preset initial speech synthesis model to obtain an initial synthesized speech with predicted prosodic features;

[0081] Step S130, obtaining speech feedback data of the initial synthesized speech;

[0082] Step S140, performing prosody optimization detection on the speech feedback data based on a preset prosody optimization model to obtain prosody adjustment data;

[0083] Step S150, adjusting the initial speech feature distribution parameters of the initial speech synthesis model based on the prosody adjustment data to obtain adjusted speech feature distribution parameters;

[0084] Step S160, adjusting the initial speech synthesis model based on the adjusted speech feature distribution parameters to obtain a target speech synthesis model;

[0085] Step S170 , performing speech synthesis processing on the target text and the target speech with the target prosodic features based on the target speech synthesis model to obtain the target synthesized speech with the target prosodic features.

[0086] In steps S110 to S170 of some embodiments, the embodiments of the present application improve the training process of the model, that is, in the process of training the target speech synthesis model, the rhythm optimization model and the speech feedback data of the initial synthesized speech output by the model can be used to perform rhythm optimization, thereby adjusting the initial speech synthesis model, so that the generated synthesized speech can be more in line with the subject's perception and reception of speech, and improve the accuracy of the rhythmic expression of the generated synthesized speech.

[0087] In step S110 of some embodiments, sample data refers to data used to train the model. The present application may first obtain a training sample set, which includes multiple sample data, and the training sample set may be obtained from an existing synthetic speech data set as needed, or may be collected and annotated in real time according to a specific acquisition device. For each sample data, including sample text and sample speech, the sample text has text content, the sample speech refers to a prompt speech used to help the model generate prosodic features, and the sample speech has sample prosodic features. The sample prosodic features may characterize changes in pitch, length, intensity, tone, etc. of the corresponding sample text.

[0088] It should be noted that the sample speech is used to represent a reference speech when performing speech synthesis on the text content in the sample text, and can provide the characteristics of rhythm learning for the synthesized speech. Therefore, the speech content in the sample speech is not necessarily the same as the text content of the sample text. The sample text can be a text phoneme sequence obtained by a text-to-phoneme conversion model, and the text factor conversion model can adopt a Deep Voice3 model, a grapheme to phoneme conversion model (G2P), etc. In order to improve the efficiency of the model, the speech content of the sample speech can also be set to be the same as the sample text, so as to guide the training process of the model when the target output speech is used as the model for performing speech synthesis on the sample text. In addition, the sample speech is the speech recorded by the sample object according to the specific sample rhythm requirements, that is, the sample speech has the unique speaking style, rhythm and intonation expression of the sample object.

[0089] It should be noted that different voice rhythm expressions may correspond to different voice emotions, such as happiness, anger, sadness, excitement, and coquettishness, etc. The voice emotion may be the emotion expressed in a voice segment, the emotion expressed in a sentence, or the emotion expressed in a word, etc., and the embodiments of the present application do not specifically limit this.

[0090] It should be noted that the storage format of the sample voice in this application can be MP3 format, CDA format, WAV format, WMA format, RA format, MID I format, OGG format, APE format or AAC format, etc., and this application is not limited.

[0091] It should be noted that the speech synthesis method of the present application can be used to assist in car broadcasts and announcements, car navigation, electronic dictionaries, smart phones, voice assistants, e-book reading, etc. For example, in the e-book reading of financial technology, a target synthesized speech with target prosodic features and the speech content of the target text can be synthesized based on the input target text and the target speech with target prosodic features. Among them, the target synthesized speech can also be a male voice, a female voice, etc. with a variety of timbres, without specific limitation.

[0092] In step S120 of some embodiments, the initial synthesized speech is the speech synthesized by the initial speech synthesis model based on a pair of sample text and sample speech that are initially input, and the initial synthesized speech has predicted prosodic features, and the speech content of the initial synthesized speech is text content.

[0093] It should be noted that the process of speech synthesis by the initial speech synthesis model is equivalent to speech sampling, that is, sampling from the distribution learned by the model. Each speech feature (such as speech speed, volume, pitch, emotion, etc.) in the data learned by the model is not a fixed value, but a distribution. Therefore, when synthesizing, it is sampling from the distribution, and the sampling results will be different from each other, which is also a major feature of the current generative model.

[0094] It should be noted that the initial speech synthesis model of the present application can be constructed based on a large language model (Large Language Model 1, LLM, refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text), a machine learning model, etc., which is not limited here. Compared with the related art, a large amount of human feedback preference data is usually collected and a reward model is trained to achieve it, and the reward model is generally trained by minimizing the negative log-likelihood value of the manually annotated data, such as Proximal Policy Optimization (PPO), or, without using a reward model, by maximizing the gap between the preferred output and the non-preferred output, to calculate the difference in human feedback results for the same input, such as Direct Preference Optimization (DPO), etc. However, human evaluation of speech quality is subjective and personalized, resulting in uncertainty in human feedback of samples, and the rhythm control of speech synthesis is challenging, and the generation method adopted by the related art lacks an efficient solution for the object to quickly and accurately modify the output, thereby reducing the accuracy of the rhythm expression of the synthesized speech. The initial speech synthesis model of the present application is equivalent to an uncertainty perception learning model, which can generate different voices (i.e., voice sampling) through text and prompt voice, and can perform perceptual optimization on human feedback, so that the voice effect generated by the model is as close to human perception as possible, while taking into account the subjective bias (i.e., uncertainty) of different evaluators. Specifically, the voice feedback data of the present application can provide an effective supervision signal, so that the distribution of the generated voice is closer to the real human voice distribution, eliminating the dependence on a large amount of preference data, and adapting to the uncertainty of human subjective evaluation.

[0095] Please refer to Figure 2 , Figure 2 1 is a schematic diagram of a process of training a speech synthesis model provided in an embodiment of the present application. Specifically, during the model training process, sample text and sample speech can be input into the initial speech synthesis model for speech synthesis, and speech sampling will be performed during the speech synthesis process, so that multiple initial synthesized speech with predicted prosodic features can be obtained. The predicted prosodic feature refers to a feature obtained by the initial speech synthesis model based on the learned prosodic distribution sampling.

[0096] In step S130 of some embodiments, for each initial synthesized speech, speech feedback data corresponding to the initial synthesized speech may be obtained. The speech feedback data refers to data formed after the evaluation object evaluates the initial synthesized speech, and is used to characterize the degree of difference between the sample prosodic features and the predicted prosodic features.

[0097] It should be noted that the evaluation object refers to an object that is different from the sample object and can express the feeling of the synthesized speech more accurately and fairly.

[0098] Please refer to Figure 3 , Figure 3 is a flowchart of a specific method of step S130 provided in an embodiment of the present application. In some embodiments of the present application, step S130 may specifically include but is not limited to steps S310 to S340. Figure 3 These four steps are introduced in detail.

[0099] Step S310, obtaining the evaluation object's feedback description text on the initial synthesized speech;

[0100] Step S320, performing feedback text detection on the feedback description text based on a preset feedback text detection model to obtain an initial feedback score;

[0101] Step S330, optimizing the initial feedback score based on the evaluation object to obtain the object feedback score;

[0102] Step S340: obtaining voice feedback data based on the feedback description text and the object feedback score.

[0103] In step S310 of some embodiments, the feedback description text refers to the evaluation object's opinion on the rhythm quality of the initial synthesized speech, and the feedback description texts corresponding to different evaluation objects may be different. Therefore, multiple feedback description texts of multiple evaluation objects on the initial synthesized speech can be obtained to learn the uncertainty of human feedback. For example, a feedback description text is "the rhythm of the synthesized speech is too slow and the tone is too low."

[0104] In step S320 of some embodiments, feedback text detection is performed on each feedback description text based on a preset feedback text detection model to obtain an initial feedback score. The initial feedback score refers to the scoring form of the feedback description text, which can be MOS, and the MOS score can usually be between 1 and 5, where 5 represents the highest quality audio or video, and 1 represents the lowest quality. This scoring range helps users understand the quality of audio or video.

[0105] It should be noted that the feedback text detection model can be constructed based on the perceptual evaluation of speech quality (PESQ) algorithm and can be used to measure speech quality.

[0106] In step S330 of some embodiments, in order to more accurately obtain the uncertainty feedback of the evaluation object, the present application can further optimize the initial feedback score based on the personalized characteristics of the evaluation object to obtain the object feedback score. The object feedback score is used to characterize the feedback score obtained after taking into account the subjective uncertainty of different evaluation objects. At this time, when a generated speech with low uncertainty is given, the model can be updated more actively, and when a generated speech with high uncertainty is given, the update of the model becomes conservative.

[0107] Please refer to Figure 4 , Figure 4 is a flowchart of a specific method of step S330 provided in an embodiment of the present application. In some embodiments of the present application, step S330 may specifically include but is not limited to steps S410 to S430. Figure 4 These three steps are introduced in detail.

[0108] Step S410, obtaining the object identity data and the object voice preference degree of the evaluation object;

[0109] Step S420, determining the voice optimization weight of the evaluation object based on the object identity data and the object voice preference degree;

[0110] Step S430, optimizing the initial feedback score based on the speech optimization weight to obtain the object feedback score.

[0111] In step S410 of some embodiments, the subject identity data is used to characterize the basic information of the evaluation subject, such as age, region, usage habits, etc. The subject voice preference refers to the specific preferences of the evaluation subject for voice, including voice style, speaking speed, intonation, etc., which can be obtained through questionnaires, user settings or historical data collection.

[0112] In step S420 of some embodiments, the present application may further determine a first uncertainty value based on the object identity data, determine a second uncertainty value based on the object voice preference, and perform weighted calculation on the first uncertainty value and the second uncertainty value based on a preset first weight corresponding to the object identity data and a second weight corresponding to the object voice preference to obtain a voice optimization weight. In this way, by considering the two personalized features of the object identity data and the object voice preference, the importance of different features can be reflected when optimizing the score.

[0113] In step S430 of some embodiments, the initial feedback score is further optimized based on the speech optimization weight, that is, the object feedback score is obtained by weighted calculation.

[0114] It should be noted that, when determining the speech optimization weight of the evaluation object, the present application may also adopt linear regression, weighted average, etc., so as to adjust the initial feedback score according to the weight so as to make it more consistent with the actual feedback of the evaluation object.

[0115] In step S340 of some embodiments, after determining the feedback description text and the object feedback score, the present application can obtain voice feedback data. Among them, for multiple feedback description texts, the present application can also extract key information, cluster, and generate text for multiple feedback description texts, so as to obtain the key feedback text that can best reflect the uncertainty of the object, and obtain voice feedback data based on the key feedback text and the object feedback score, so as to help the model generate a more accurate synthesized speech with rhythmic expression.

[0116] It should be noted that the prosody adjustment data of the present application may also include only one of the feedback description text and the object feedback score, which is not limited here.

[0117] In step S140 of some embodiments, after obtaining the feedback description text and the object feedback score, the speech feedback data can be subjected to prosody optimization detection based on a preset prosody optimization model to obtain prosody adjustment data. The prosody optimization model refers to providing the initial speech synthesis model with the direction that needs to be optimized by learning the uncertainty information of the evaluation object in the speech feedback data, such as how to adjust the speech rate, volume, pitch, etc., so as to improve the prosody expression accuracy of the synthesized speech output by the model. The prosody adjustment data refers to the data for adjusting the speech feature distribution parameters of the initial speech synthesis model so that the initial speech synthesis model can generate speech that better meets the uncertainty requirements of the evaluation object. For example, the input feedback description text is "I feel that the synthesized speech rhythm is too slow and the tone is too low", and the object feedback score is "2", indicating that the quality is low. At this time, after the detection of the prosody optimization model, the prosody adjustment data obtained can be "speed up the speech rate, increase the volume, adjust the tone, etc."

[0118] It should be noted that the rhythm optimization model can be a model built based on a structure such as a probabilistic generation model, and specifically can be a variational auto encoder (VAE), a generative adversarial network (GAN), a diffusion model, an autoregressive model, and other model structures, which are not specifically limited here. In other words, the rhythm optimization model can be equivalent to a "multi-instance" encoder, which can regard the rhythm features of part of the input as an unordered feature set, and aggregate them into a vector of fixed length through a self-attention mechanism, and use part of the input to control rhythm generation. In this way, the rhythm control under different missing modes can be robust.

[0119] Please refer to Figure 5 , Figure 5 is a flowchart of a specific method of step S140 provided in an embodiment of the present application. In some embodiments of the present application, the rhythm optimization model includes an encoding unit, an aggregation unit, and a generation unit, and step S140 may specifically include but is not limited to steps S510 to S530. Figure 5 These three steps are introduced in detail.

[0120] Step S510, encoding the speech feedback data based on the encoding unit to obtain a speech feedback vector;

[0121] Step S520, performing vector aggregation processing on the speech feedback vector based on the aggregation unit to obtain an aggregated vector;

[0122] Step S530: performing rhythm optimization generation on the aggregated vector based on the generation unit to obtain rhythm adjustment data.

[0123] In steps S510 to S530 of some embodiments, the present application may first encode the voice feedback data based on the encoding unit to obtain a voice feedback vector, that is, convert the original voice feedback data into a digital representation for further processing and analysis. The main function of the encoding unit is to analyze and convert the voice data, and feature extraction, deep learning models (such as convolutional neural networks or recursive neural networks) and the like may be used to convert the original voice signal into a vector representation. Moreover, after encoding, the characteristic information of the voice (such as pitch, speed, emotion, etc.) can be effectively captured. The aggregation unit is used to merge multiple voice feedback vectors into a more representative aggregation vector in order to better summarize and understand the overall trend or characteristics of the feedback, and is implemented according to the standards for aggregation (such as time windows, user features, etc.) required, combined with methods such as weighted average, clustering algorithms, principal component analysis, etc. Further, the aggregation vector may be rhythmically optimized and generated based on the generation unit to obtain rhythm adjustment data to improve the naturalness and comprehensibility of voice interaction.

[0124] In step S150 of some embodiments, the prosody adjustment data includes speech rate adjustment sub-data, volume adjustment sub-data, pitch adjustment sub-data and emotion adjustment sub-data, and the initial speech feature distribution parameters include initial speech rate distribution sub-parameters, initial volume distribution sub-parameters, initial pitch distribution sub-parameters and initial emotion distribution sub-parameters. Among them, the initial speech feature distribution parameters refer to the distribution parameters that the initial speech synthesis model relies on when generating speech samples, which can help the model generate a speech distribution that better meets the needs.

[0125] Please refer to Figure 6 , Figure 6is a flowchart of a specific method of step S150 provided in an embodiment of the present application. In some embodiments of the present application, step S150 may specifically include but is not limited to steps S610 to S650. Figure 6 These five steps are introduced in detail.

[0126] Step S610, adjusting the initial speech rate distribution sub-parameter based on the speech rate adjustment sub-data to obtain an adjusted speech rate distribution sub-parameter;

[0127] Step S620, adjusting the initial volume distribution sub-parameter based on the volume adjustment sub-data to obtain adjusted volume distribution sub-parameter;

[0128] Step S630, adjusting the pitch parameters of the initial pitch distribution sub-parameters based on the pitch adjustment sub-data to obtain adjusted pitch distribution sub-parameters;

[0129] Step S640, adjusting the initial emotion distribution sub-parameter based on the emotion adjustment sub-data to obtain adjusted emotion distribution sub-parameter;

[0130] Step S650, obtaining adjusted speech feature distribution parameters based on the adjusted speech rate distribution sub-parameter, the adjusted volume distribution sub-parameter, the adjusted pitch distribution sub-parameter, and the adjusted emotion distribution sub-parameter.

[0131] In steps S610 to S650 of some embodiments, the present application can adjust the corresponding distribution sub-parameters based on the adjustment sub-parameters of different feature types output from the prosody optimization model, so that the speech effect generated by the model is as close as possible to human perception, while taking into account the subjective bias (that is, uncertainty) of different evaluators. When given a generated speech with low uncertainty, the model can be updated more actively, and when there is high uncertainty, the update becomes conservative.

[0132] It should be noted that the rhythm adjustment data of the present application may also include sound length adjustment sub-data, sound intensity adjustment sub-data, etc., which can be adjusted according to actual needs. Among them, pitch refers to the height of the sound, which is the most basic element in rhythm. Sound length refers to the length of time of pronunciation, which is an important element in rhythm. Sound intensity refers to the strength of the sound, which is also an important part of rhythm. Tone refers to the rise and fall of the sound, which is particularly important for tonal languages ​​such as Chinese.

[0133] It should be noted that, by adjusting the emotion parameters of the initial emotion distribution sub-parameters based on the emotion adjustment sub-data, the present application can make the synthesized speech emotional and more in line with human perception and acceptance.

[0134] In step S160 of some embodiments, after obtaining the adjusted speech feature distribution parameters, the initial speech synthesis model can be further adjusted based on the adjusted speech feature distribution parameters until a preset training end condition is reached to obtain a target speech synthesis model.

[0135] It should be noted that the preset training end condition can be when the synthesis accuracy of the initial speech synthesis model is greater than or equal to the preset accuracy threshold, or reaches the preset number of iterations, that is, the currently adjusted model can generate complete speech that meets user feedback. In addition, the synthesis accuracy is obtained by calculating the similarity between the sample prosodic features of the sample speech and the predicted prosodic features of the initial synthesized speech, and the function for similarity calculation can be selected according to actual needs, such as cosine similarity calculation, time axis comparison method, etc., which is not specifically limited here.

[0136] It should be noted that when calculating the similarity between the sample rhythmic features and the predicted rhythmic features, the similarity calculation of the pitch, length, speed, tone and other features may be involved, and the mean square error loss function, cross entropy loss function, contrast loss function, etc. may be specifically used, which is not specifically limited here.

[0137] Please refer to Figure 7 , Figure 7 is a flowchart of a specific method of step S160 provided in an embodiment of the present application. In some embodiments of the present application, step S160 may specifically include but is not limited to steps S710 to S740. Figure 7 These four steps are introduced in detail.

[0138] Step S710, determining a sample feedback category of the sample data based on the speech feedback score;

[0139] Step S720, determining current sample distribution data for training the initial speech synthesis model based on the sample feedback category;

[0140] Step S730, determining sample distribution deviation data based on preset sample distribution data and current sample distribution data;

[0141] Step S740, adjusting the initial speech synthesis model based on the adjusted speech feature distribution parameters and sample distribution deviation data to obtain a target speech synthesis model.

[0142] In step S710 of some embodiments, when training the target speech synthesis model, the present application not only relies on the prosody adjustment parameters output by the prosody optimization model, but also considers the distribution of the learned samples. Figure 2Specifically, the present application can determine the sample feedback category of the sample data based on the speech feedback score, and the sample feedback category at this time includes positive samples, neutral samples and negative samples. Among them, the positive sample indicates that the similarity between the predicted prosodic features of the synthesized initial synthesized speech and the sample prosodic features of the sample speech is high. The neutral sample indicates that the similarity between the predicted prosodic features of the synthesized initial synthesized speech and the sample prosodic features of the sample speech is average. The negative sample indicates that the similarity between the predicted prosodic features of the synthesized initial synthesized speech and the sample prosodic features of the sample speech is low. For example, the speech feedback score is any value from 1 to 5. At this time, if the speech feedback score is any value in [1,2), the current sampled initial synthesized speech is determined to be a negative sample. If the speech feedback score is any value in [2,4], the current sampled initial synthesized speech is determined to be a neutral sample. If the speech feedback score is any value in (4,5], the current sampled initial synthesized speech is determined to be a negative sample.

[0143] In steps S720 to S740 of some embodiments, after obtaining the sample feedback category of the initial synthesized speech generated after multiple speech sampling, the present application can determine the current sample distribution data for training the initial speech synthesis model based on the multiple sample feedback categories. The current sample distribution data refers to the distribution of positive samples, neutral samples and negative samples in the current initial synthesized speech. The preset sample distribution data refers to the sample distribution that the generated initial synthesized speech needs to meet. Further, based on the preset sample distribution data and the current sample distribution data, the sample distribution deviation data is determined, and the sample distribution deviation data is used to characterize the distribution difference between the preset sample distribution data and the current sample distribution data. Further, the initial speech synthesis model can be adjusted based on the adjusted speech feature distribution parameters and the sample distribution deviation data to obtain the target speech synthesis model, so that the model meets the preset sample distribution data when generating speech, and the rhythm adjustment data of the rhythm features (such as speech speed, volume, pitch, emotion, etc.) can be combined with human feedback to make adjustments, so that the speech effect generated by the model is as close to human perception as possible, while taking into account the subjective bias (that is, uncertainty) of different evaluators.

[0144] Please refer to Figure 8 , Figure 8 is a flowchart of a specific method of step S740 provided in an embodiment of the present application. In some embodiments of the present application, step S740 may specifically include but is not limited to steps S810 to S820. Figure 8 These two steps are introduced in detail.

[0145] Step S810, performing category loss calculation based on the preset sample category and the sample feedback category to obtain category loss data;

[0146] Step S820, adjusting the initial speech synthesis model based on the adjusted speech feature distribution parameters, sample distribution deviation data and category loss data to obtain a target speech synthesis model.

[0147] In step S810 and step S820 of some embodiments, the sample data of the present application also includes a preset sample category, which refers to a sample category to which the initial synthesized speech belongs that is pre-labeled. Specifically, the present application can perform category loss calculation on the preset sample category and the sample feedback category to obtain category loss data, and further adjust the initial speech synthesis model based on the adjusted speech feature distribution parameters, sample distribution deviation data, and category loss data to obtain a target speech synthesis model.

[0148] In the above embodiment, when adjusting the initial speech synthesis model, the present application will simultaneously consider the diversity feedback of the evaluation object, the sample distribution deviation and the speech category loss to generate a complete speech that conforms to the object feedback, thereby improving the rhythmic expression accuracy of the synthesized speech.

[0149] In step S170 of some embodiments, in actual applications, such as smart phones, a text-to-speech speech synthesis device can be installed on the smart phone, and the speech synthesis model trained by the present application is deployed in the speech synthesis device. When an operation of converting a target text into a synthesized speech having a target prosodic feature of a target speech is detected, the smart phone can generate a speech synthesis service request and send the speech synthesis service request to the speech synthesis device. By responding to the speech synthesis service request, the smart phone uses the speech synthesis device to extract the target text and the target speech having the target prosodic feature from the speech synthesis service request, so as to synthesize the target text into a target synthesized speech containing the target prosodic feature through the target speech synthesis model.

[0150] It should be noted that in a specific application, such as applying the present application to a terminal, when a user needs to perform speech synthesis at the terminal, the target text and the target speech with the target prosodic features that need to be synthesized can be selected on the terminal page, and a pop-up box can be displayed on the terminal page. The user touches the synthesized speech button in the pop-up box, and the target text and the target speech with the target prosodic features are sent to the speech synthesis device for speech synthesis processing through a speech synthesis service request. Afterwards, the voice is broadcasted through the speaker of the terminal, so that the user can hear the target synthesized speech with the same prosody and emotional intonation as the target speech.

[0151] A speech synthesis method provided in the embodiment of the present application combines speech feedback data and a prosody optimization model to determine the prosody adjustment direction of the initial speech feature distribution parameters, so that the difference between the predicted prosody features of the synthesized speech and the sample prosody features is smaller, so that when the target speech synthesis model provided in the embodiment of the present application performs speech synthesis processing on the target text and the target speech with the target prosody features, a synthesized speech with more accurate prosody expression can be generated. Compared with the related art, which only generates a matching and user-preferred speech according to the user's instructions (such as text descriptions, for example: generate a slow and gentle female voice for me), the present application will be implemented by combining text and "prompt speech" when performing speech synthesis, that is, a short reference / prompt speech can be provided to allow the model to synthesize a given text according to this prosody style, and the synthesized prosody effect can be adjusted at any time by description (i.e., speech feedback data). Therefore, the present application can generate a more accurate prosody expression of synthesized speech.

[0152] Please refer to Fig. 9 , Fig. 9 : is a schematic diagram of the module structure of the speech synthesis device provided in the embodiment of the present application. In some embodiments of the present application, the speech synthesis device may specifically include:

[0153] A first acquisition module 910 is used to acquire sample data, where the sample data includes sample text and sample speech, where the sample text has text content and the sample speech has sample prosodic features;

[0154] A first synthesis module 920 is used to perform speech synthesis processing on the sample text and the sample speech based on a preset initial speech synthesis model to obtain an initial synthesized speech with predicted prosodic features, wherein the speech content of the initial synthesized speech is text content;

[0155] The second acquisition module 930 is used to acquire speech feedback data of the initial synthesized speech; wherein the speech feedback data is used to characterize the degree of difference between the prosodic features of the sample and the predicted prosodic features;

[0156] A detection module 940 is used to perform prosody optimization detection on the speech feedback data based on a preset prosody optimization model to obtain prosody adjustment data;

[0157] A parameter adjustment module 950 is used to adjust the initial speech feature distribution parameters of the initial speech synthesis model based on the prosody adjustment data to obtain adjusted speech feature distribution parameters;

[0158] A model adjustment module 960, configured to adjust the initial speech synthesis model based on the adjusted speech feature distribution parameters to obtain a target speech synthesis model;

[0159] The second synthesis module 970 is used to perform speech synthesis processing on the target text and the target speech with the target prosodic features based on the target speech synthesis model to obtain the target synthesized speech with the target prosodic features.

[0160] It should be noted that the speech synthesis device of the embodiment of the present application is used to execute the above-mentioned speech synthesis method, and the speech synthesis device of the embodiment of the present application corresponds to the aforementioned speech synthesis method. For the specific training process, please refer to the aforementioned speech synthesis method, which will not be described in detail here.

[0161] An embodiment of the present application further provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the speech synthesis method of the embodiment of the present application is implemented.

[0162] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0163] Combine the following Fig.10 The electronic device according to the embodiment of the present application is introduced in detail.

[0164] Please refer to Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, including:

[0165] The processor 1010 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0166] The memory 1020 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1020 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1020, and the processor 1010 calls and executes the speech synthesis method of the embodiment of this application;

[0167] Input / output interface 1030, used to implement information input and output;

[0168] The communication interface 1040 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0169] Bus 1050 , which transmits information between various components of the device (e.g., processor 1010 , memory 1020 , input / output interface 1030 , and communication interface 1040 );

[0170] The processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 are connected to each other in communication within the device via a bus 1050 .

[0171] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the speech synthesis method of the embodiment of the present application is implemented.

[0172] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0173] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0174] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0175] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0176] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0177] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0178] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0179] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0180] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0181] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0182] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (RandomAccess Memory, RAM), disk or optical disk and other media that can store programs.

[0183] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Acquire sample data, the sample data including sample text and sample speech, the sample text having text content, and the sample speech having sample prosodic features; Performing speech synthesis processing on the sample text and the sample speech based on a preset initial speech synthesis model to obtain an initial synthesized speech with predicted prosodic features, wherein the speech content of the initial synthesized speech is the text content; Acquiring speech feedback data of the initial synthesized speech; wherein the speech feedback data is used to characterize the degree of difference between the sample prosodic feature and the predicted prosodic feature; Performing prosody optimization detection on the speech feedback data based on a preset prosody optimization model to obtain prosody adjustment data; Performing parameter adjustment on initial speech feature distribution parameters of the initial speech synthesis model based on the prosody adjustment data to obtain adjusted speech feature distribution parameters; Adjust the initial speech synthesis model based on the adjusted speech feature distribution parameters to obtain a target speech synthesis model; Based on the target speech synthesis model, speech synthesis processing is performed on the target text and the target speech with the target prosodic features to obtain the target synthesized speech with the target prosodic features.

2. The method according to claim 1, characterized in that: The obtaining of speech feedback data of the initial synthesized speech comprises: Obtaining a textual description of the feedback of the evaluation subject on the initial synthesized speech; Performing feedback text detection on the feedback description text based on a preset feedback text detection model to obtain an initial feedback score; Optimizing the initial feedback score based on the evaluation object to obtain an object feedback score; The voice feedback data is obtained based on the feedback description text and the object feedback score.

3. The method according to claim 2, characterized in that The step of optimizing the initial feedback score based on the evaluation object to obtain the object feedback score includes: Acquiring the subject identity data and the subject voice preference degree of the evaluation subject; Determining a voice optimization weight of the evaluation object based on the object identity data and the object voice preference degree; The initial feedback score is optimized based on the speech optimization weight to obtain the object feedback score.

4. The method according to claim 2, characterized in that: The step of adjusting the initial speech synthesis model based on the adjusted speech feature distribution parameters to obtain a target speech synthesis model includes: Determining a sample feedback category of the sample data based on the speech feedback score; Determining current sample distribution data for training the initial speech synthesis model based on the sample feedback category; Determine sample distribution deviation data based on preset sample distribution data and the current sample distribution data; The initial speech synthesis model is adjusted based on the adjusted speech feature distribution parameters and the sample distribution deviation data to obtain the target speech synthesis model.

5. The method according to claim 4, characterized in that The sample data also includes a preset sample category, and the initial speech synthesis model is adjusted based on the adjusted speech feature distribution parameter and the sample distribution deviation data to obtain the target speech synthesis model, including: Performing category loss calculation based on the preset sample category and the sample feedback category to obtain category loss data; The initial speech synthesis model is adjusted based on the adjusted speech feature distribution parameters, the sample distribution deviation data and the category loss data to obtain the target speech synthesis model.

6. The method according to any one of claims 1 to 5, characterized in that: The prosody adjustment data includes speech rate adjustment sub-data, volume adjustment sub-data, pitch adjustment sub-data and emotion adjustment sub-data, and the initial speech feature distribution parameters include initial speech rate distribution sub-parameters, initial volume distribution sub-parameters, initial pitch distribution sub-parameters and initial emotion distribution sub-parameters; The step of adjusting the initial speech feature distribution parameters of the initial speech synthesis model based on the prosody adjustment data to obtain the adjusted speech feature distribution parameters includes: Performing speech rate parameter adjustment on the initial speech rate distribution sub-parameter based on the speech rate adjustment sub-data to obtain an adjusted speech rate distribution sub-parameter; Performing volume parameter adjustment on the initial volume distribution sub-parameter based on the volume adjustment sub-data to obtain adjusted volume distribution sub-parameter; Performing pitch parameter adjustment on the initial pitch distribution sub-parameter based on the pitch adjustment sub-data to obtain adjusted pitch distribution sub-parameter; Performing emotion parameter adjustment on the initial emotion distribution sub-parameter based on the emotion adjustment sub-data to obtain adjusted emotion distribution sub-parameter; The adjusted speech feature distribution parameter is obtained based on the adjusted speech rate distribution sub-parameter, the adjusted volume distribution sub-parameter, the adjusted pitch distribution sub-parameter and the adjusted emotion distribution sub-parameter.

7. The method according to any one of claims 1 to 5, characterized in that: The prosody optimization model includes an encoding unit, an aggregation unit, and a generation unit. The prosody optimization detection is performed on the speech feedback data based on the preset prosody optimization model to obtain prosody adjustment data, including: Performing encoding processing on the speech feedback data based on the encoding unit to obtain a speech feedback vector; Performing vector aggregation processing on the speech feedback vector based on the aggregation unit to obtain an aggregation vector; The generating unit performs rhythm optimization on the aggregated vector to obtain the rhythm adjustment data.

8. A speech synthesis device, characterized in that: The device comprises: A first acquisition module is used to acquire sample data, wherein the sample data includes sample text and sample speech, wherein the sample text has text content, and the sample speech has sample prosodic features; A first synthesis module is used to perform speech synthesis processing on the sample text and the sample speech based on a preset initial speech synthesis model to obtain an initial synthesized speech with predicted prosodic features, wherein the speech content of the initial synthesized speech is the text content; A second acquisition module is used to acquire speech feedback data of the initial synthesized speech; wherein the speech feedback data is used to characterize the degree of difference between the sample prosodic feature and the predicted prosodic feature; A detection module, configured to perform a prosody optimization detection on the speech feedback data based on a preset prosody optimization model to obtain prosody adjustment data; A parameter adjustment module, used for adjusting the initial speech feature distribution parameters of the initial speech synthesis model based on the prosody adjustment data to obtain adjusted speech feature distribution parameters; A model adjustment module, used for adjusting the initial speech synthesis model based on the adjusted speech feature distribution parameters to obtain a target speech synthesis model; The second synthesis module is used to perform speech synthesis processing on the target text and the target speech with the target prosodic features based on the target speech synthesis model to obtain the target synthesized speech with the target prosodic features.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images