Speech synthesis method, speech synthesis model training method and related devices

Abstract

The present application discloses a speech synthesis method, a speech synthesis model training method, and related devices. The method includes: obtaining a target text and a target speech to be synthesized; extracting the speaker's voice features in the target speech; generating a target synthesized speech based on the speaker's voice features and the target text, wherein the target synthesized speech is a speech that refers to the sound characteristics of the target speech and whose pronunciation content is consistent with the target text; wherein the voice features are extracted using a Q-former; and / or the target synthesized speech is generated using a target speech synthesis model, wherein the target speech synthesis model is obtained by reinforcement learning an initial speech synthesis model based on the user's preference for a first sample synthesized speech, and the first sample synthesized speech is generated using the initial speech synthesis model. In this way, the present application can improve the quality of the target synthesized speech.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a speech synthesis method, a speech synthesis model training method, and related devices. Background Art

[0002] Voice cloning has become a hot topic in the field of speech synthesis in recent years. It aims to build a speech synthesis system that can reproduce the target speaker's voice by recording a short audio clip (usually 5-15 seconds) using common recording devices such as mobile phones, computers, and voice recorders.

[0003] While voice cloning technology based on large models is becoming increasingly mainstream, synthesized speech still suffers from issues such as incorrect pronunciation, unusual rhythms, and inconsistent speech speeds. Addressing these issues in synthesized speech is crucial. Summary of the Invention

[0004] The main technical problem solved by this application is to provide a speech synthesis method, a speech synthesis model training method and related devices, which can improve the quality of the target synthesized speech.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: to provide a speech synthesis method, the method comprising: obtaining a target text and a target speech to be synthesized; extracting the voice features of the speaker in the target speech; generating a target synthesized speech based on the voice features of the speaker and the target text, wherein the target synthesized speech is a speech that refers to the sound characteristics of the target speech and whose pronunciation content is consistent with the target text; wherein the voice features are extracted using a Q-former; and / or the target synthesized speech is generated using a target speech synthesis model, wherein the target speech synthesis model is obtained by reinforcement learning of an initial speech synthesis model according to the user's preference for the first sample synthesized speech, and the first sample synthesized speech is generated using the initial speech synthesis model.

[0006] To solve the above technical problems, another technical solution adopted in this application is: providing a training method for a target speech synthesis model, including: obtaining the model to be optimized for this reinforcement learning; wherein, the model to be reinforced for the first time is the initial speech synthesis model, and the model to be reinforced for the non-first time is the model obtained by the previous reinforcement learning; obtaining each third sample synthesized speech generated by the model to be optimized; using the user's preference for each third sample synthesized speech, adjusting the network parameters of the model to be optimized, and repeating the above steps to obtain the target speech synthesis model; wherein, the adjustment direction of the model to be optimized includes: increasing the probability of generating the third sample synthesized speech with a high degree of user preference.

[0007] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a speech synthesis device, comprising a first acquisition module, an extraction module, and a speech generation module. The first acquisition module is used to acquire a target text and a target speech to be synthesized; the extraction module is used to extract the speaker's voice features from the target speech; the speech generation module is used to generate a target synthesized speech based on the speaker's voice features and the target text, wherein the target synthesized speech is a speech that refers to the sound characteristics of the target speech and whose pronunciation content is consistent with the target text; wherein the voice features are extracted using a Q-former; and / or the target synthesized speech is generated using a target speech synthesis model, which is obtained by reinforcement learning of an initial speech synthesis model based on the user's preference for a first sample synthesized speech, and the first sample synthesized speech is generated using the initial speech synthesis model.

[0008] To solve the above technical problems, another technical solution adopted by this application is to provide a target speech synthesis model training device, including: a second acquisition module, a third acquisition module, and a parameter adjustment module. The second acquisition module is used to obtain the model to be optimized for the current reinforcement learning; wherein, the model to be reinforced for the first time is the initial speech synthesis model, and the model to be reinforced for the non-first time is the model obtained by the previous reinforcement learning; the third acquisition module is used to obtain each third sample synthesized speech generated by the current model to be optimized; the parameter adjustment module is used to adjust the network parameters of the current model to be optimized based on the user's preference for each third sample synthesized speech, and repeat the above steps to obtain the target speech synthesis model; wherein, the adjustment direction of the model to be optimized includes: increasing the generation probability of the third sample synthesized speech with a high degree of user preference.

[0009] To solve the above technical problems, another technical solution adopted in this application is: to provide an electronic device, including a memory and a processor coupled to each other, the memory storing program instructions; the processor is used to execute the program instructions stored in the memory to implement the above method.

[0010] In order to solve the above technical problems, another technical solution adopted in this application is: providing a computer-readable storage medium for storing program instructions, which can be executed to implement the above method.

[0011] In the above scheme, the target synthesized speech is generated based on the speaker's voice features and the target text. The voice features are extracted using Q-former; and / or the target synthesized speech is generated using a target speech synthesis model, which is obtained by performing reinforcement learning on an initial speech synthesis model according to the user's preference for the first sample synthesized speech. The method of extracting the speaker's voice features using Q-former can extract voice features important for generating the target synthesized speech. Furthermore, using the voice features extracted by Q-former for speech synthesis can make the synthesized speech more consistent with the speaker's voice characteristics, making the synthesized speech more natural and accurate. Furthermore, since the target speech synthesis model can be obtained by performing reinforcement learning on the initial speech synthesis model according to the user's preference for the synthesized speech, the target speech synthesis model obtained through reinforcement learning has a higher probability of generating a target synthesized speech that meets the user's preferences, thereby facilitating the generation of a target synthesized speech that meets the user's preferences. Compared with the method of directly using a speech synthesis model to generate a target synthesized speech based on the input target text and target speech, the present application can improve the quality of the target synthesized speech by using either a Q-former to extract the speaker's voice features or a target speech synthesis model obtained by reinforcement learning using the user's preference for speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a flow chart of an embodiment of a speech synthesis method provided by the present application;

[0013] Figure 2 This is a schematic diagram of the structure of an embodiment of a Q-former network provided by the present application;

[0014] Figure 3 This is a flowchart of an embodiment of obtaining a target speech synthesis model through reinforcement learning provided by the present application;

[0015] Figure 4 This is a flowchart of another embodiment of obtaining a target speech synthesis model through reinforcement learning provided by the present application;

[0016] Figure 5 This is a flowchart of an embodiment of a target speech synthesis model training method provided by the present application;

[0017] Figure 6 This is a schematic diagram of the framework of an embodiment of a speech synthesis device provided by the present application;

[0018] Figure 7 This is a schematic diagram of the framework of an embodiment of a target speech synthesis model training device provided by the present application;

[0019] Figure 8This is a schematic diagram of the framework of an embodiment of an electronic device provided by the present application;

[0020] Figure 9 It is a schematic diagram of the framework of the computer-readable storage medium provided by this application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and effects of this application clearer and more specific, this application is further described in detail below with reference to the accompanying drawings and examples.

[0022] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0023] Currently, voice cloning technology based on large models is gradually becoming mainstream, but the following problems still exist: (1) Since large models have no explicit time constraints and only align voice tokens and text through the attention mechanism within the large model, the synthesized speech often has problems such as pronunciation errors, strange rhythm, and fast and slow speech speed; (2) Large models for speech synthesis are based on the temporal correlation of modeled speech, and there is diversity in sampling. Sometimes the generated synthesized speech does not meet the user's perceptual preferences for timbre, rhythm, etc.

[0024] To solve the above problems, this application proposes a speech synthesis method, a speech synthesis model training method and related devices. Specifically:

[0025] See also Figure 1 , Figure 1 It is a flow chart of an embodiment of the speech synthesis method provided by this application. It should be noted that if there are substantially the same results, this embodiment does not Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes:

[0026] S11: Obtain target text and target speech to be synthesized.

[0027] S12: Extract the speaker's voice features in the target speech.

[0028] S13: Generate a target synthesized speech based on the speaker's voice characteristics and the target text. The target synthesized speech is a speech that refers to the sound characteristics of the target speech and has pronunciation content consistent with the target text.

[0029] This embodiment is used to convert the target text into speech with the speaker's voice characteristics after obtaining the speaker's target speech and the target text to be synthesized, namely the target synthesized speech mentioned herein, with reference to the speaker's voice characteristics.

[0030] The target speech is a segment of audio (e.g., 5-15 seconds) of a speaker recorded using any electronic device with a recording function; the target text is the text to be converted into the corresponding speech; and the synthesized target speech is a speech that refers to the sound characteristics of the target speech and whose pronunciation content is consistent with the target text.

[0031] For example, the target text is "Today the weather is fine, the sky is clear and blue", the target speech is an audio clip spoken by a speaker, and the target synthesized speech is the sound characteristics of the reference speaker's audio, and the generated pronunciation content is the speech of "Today the weather is fine, the sky is clear and blue".

[0032] In this embodiment, the reference sound characteristics may be at least some of all sound characteristics. For example, the reference sound characteristics may be all sound characteristics, or they may be sound characteristics of a subset of all sound characteristics. For example, the reference sound characteristics may be at least one of timbre, intonation, speech rate, rhythm, and emotion, or a combination of at least two.

[0033] In an optional embodiment, the sound features are extracted using an audio feature extraction network, wherein the audio feature extraction network is composed of a plurality of attention modules, each attention module including a self-attention submodule and a cross-attention submodule.

[0034] The input data of the self-attention submodule are several learnable query vectors. Different learnable query vectors are used to extract sound sub-features of different dimensions of the target speech. The self-attention submodule is used to perform self-attention processing on the input several learnable query vectors to obtain the fusion results of several learnable query vectors.

[0035] The input data of the cross-attention submodule includes the fusion result output by the self-attention submodule and the audio features of the target speech. The cross-attention submodule uses the audio features of the target speech as the key vector and value vector, and the fusion result as the query vector. It performs cross-attention processing on the query vector, key vector, and value vector to extract the sound features.

[0036] The audio feature extraction network may be, but is not limited to, a Q-former network, or any other network composed of several attention modules.

[0037] See also Figure 2 , Figure 2 This is a schematic diagram of the structure of an embodiment of the Q-former network provided by this application. To facilitate understanding of the process of extracting sound features using the aforementioned audio feature extraction network, the following briefly describes the sound feature extraction process using a Q-former as an example.

[0038] First, the Q-former network consists of N attention modules, each of which includes a self-attention submodule and a cross-attention submodule. The input data for the self-attention submodule is a set of discrete learnable query vectors. This set of learnable query vectors includes several different learnable query vectors, each used to extract acoustic sub-features of different dimensions of the target speech. For example, learnable query vector Q1 is used to extract timbre features of the target speech, learnable query vector Q2 is used to extract prosodic features of the target speech, and learnable query vector Q3 is used to extract emotional characteristics of the target speech.

[0039] The specific features extracted by each learnable query vector are learned. In other words, the learnable query vector is learnable; it is not a fixed value. During the learning process, it can automatically learn features important for synthesized speech generation (such as timbre, intonation, or emotion) through backpropagation optimization. Compared to static feature extraction using a pre-trained network, this embodiment utilizes a Q-former to extract sound features, which dynamically learns features important for speech synthesis. Therefore, using the sound features extracted by the Q-former for subsequent speech synthesis facilitates the generation of speech that is more consistent with the speaker's voice characteristics and has a natural rhythm.

[0040] Furthermore, the self-attention submodule in the Q-former is used to perform self-attention interaction on several learnable query vectors to obtain the interaction result of the learnable query vector, and then use the interaction result as the query vector Q, and use the audio features of the target speech as the value vector V and the key vector K respectively, and perform cross-attention processing with the audio features of the target speech through the cross-attention submodule to query the sound features related to each learnable vector from the target speech, and then fuse the queried sound features. In the process of performing the above-mentioned attention processing, the attention mechanism can focus on local key features, which helps to retain more details. Therefore, the above-mentioned method of feature extraction using Q-former can extract richer and more detailed sound features about the speaker from the target speech. In one embodiment, the audio features of the target speech are obtained by performing speech discretization processing on the target speech, obtaining the corresponding speech tokens, and then encoding the speech tokens. Of course, in other embodiments, the audio features of the target speech can also be obtained by other methods.

[0041] It should be noted that since Q-former can extract more abundant sound features of the speaker and important sound features for synthesized speech generation from the target speech, the sound features extracted using Q-former can make the generated target synthesized speech features more consistent with the speaker's sound features, which is conducive to generating target synthesized speech that is highly similar to the speaker's sound features.

[0042] At the same time, it should be noted that the method of using Q-former to extract the voice features of the target speaker provided in this embodiment can automatically learn the most important features for synthesized speech generation. Therefore, it is conducive to generating target synthesized speech that is highly similar to the speaker's voice characteristics. It can effectively solve the problems of pronunciation errors, strange rhythms, and fast and slow speech speeds in synthesized speech caused by the lack of explicit duration constraints in large models, thereby making the generated target synthesized speech more natural and more consistent with the speaker's voice characteristics.

[0043] The specific number of learnable query vectors and the data dimensions of each learnable query vector can be determined based on the desired effect of the final synthesized speech. While meeting the desired effect, the number of learnable query vectors and / or the data dimensions of each learnable query vector can be appropriately reduced to avoid introducing excessive redundancy or even erroneous information. This can significantly improve the naturalness, prosody and timbre similarity, or stability of the synthesized speech generated by the target speech synthesis model.

[0044] In another optional embodiment, the target synthesized speech is generated using a target speech synthesis model. The target speech synthesis model is obtained by performing reinforcement learning on an initial speech synthesis model based on the user's preference for the first sample synthesized speech, and the first sample synthesized speech is generated using the initial speech synthesis model.

[0045] Among them, the initial speech synthesis model must undergo several rounds of reinforcement learning before it can learn the target speech synthesis model. Each round of reinforcement learning involves the adjustment of the model network parameters, and the adjustment of each network parameter is essentially based on the model obtained by the previous reinforcement learning.

[0046] For example, the initial speech synthesis model needs to go through n rounds of reinforcement learning to obtain the target speech synthesis model, where the model to be reinforced for the first time is Q0 (i.e. the initial speech synthesis model), Q1 is obtained after the first reinforcement learning, Q1 is obtained after the second reinforcement learning, Q2 is obtained after the third reinforcement learning, and Q2 is obtained after the third reinforcement learning. Repeat the above steps until Q n-1 After the nth reinforcement learning, we get Q n (Target Speech Synthesis Model).

[0047] The first sample synthesized speech is generated using the initial speech synthesis model. Specifically, the first sample synthesized speech is generated using the model to be reinforced in each round of learning. In this embodiment, the first sample synthesized speech includes several second sample synthesized speech generated by the model to be reinforced in each round of learning.

[0048] In this embodiment, the process of performing reinforcement learning on the initial synthesis model according to the user's preference for the first sample synthesized speech to obtain the target speech synthesis model is essentially the process of adjusting the parameters of the learning model to be reinforced in each round according to the user's preference for the second sample synthesized speech generated by the learning model to be reinforced in each round.

[0049] Continuing with the above example, for the first learning model to be reinforced Q0, the user's preference for the second sample synthesized speech generated by Q0 is used to adjust the network parameters of Q0 to obtain Q1; for the second learning model to be reinforced Q1, the user's preference for the second sample synthesized speech generated by Q1 is used to adjust the network parameters of Q1 to obtain Q2. Repeat the above operation until the target speech synthesis model is obtained.

[0050] It should be noted that, according to the user's preference for the first sample synthesized speech, the initial synthesis model is reinforced with learning, and the target speech synthesis model obtained can make the target synthesized speech generated by the target speech synthesis model more consistent with the user's preference for the target synthesized speech. The specific method of reinforcing the initial speech synthesis model to obtain the target speech synthesis model according to the user's preference for the first sample synthesized speech can be referred to below. Figure 3 and Figure 4 Description of the illustrated embodiment.

[0051] In one embodiment, the user's preference for the first sample synthesized speech includes preference sub-levels for at least one of the following: prosodic naturalness, timbre similarity, and intelligibility. That is, during the reinforcement learning process, network parameters may be adjusted based on at least one of the prosodic naturalness, timbre similarity, and intelligibility of the first sample synthesized speech, so that the target synthesized speech ultimately obtained through reinforcement learning can better meet the user's preferences in at least one of prosodic naturalness, timbre similarity, and intelligibility.

[0052] Preferably, the preference level of the first sample synthesized speech can be set to include preference sub-levels in terms of rhythmic naturalness, timbre similarity and intelligibility, so that the target synthesized speech generated by the target synthesized speech obtained by the final reinforcement learning meets the user preferences in terms of rhythmic naturalness, timbre similarity and intelligibility.

[0053] The user's preference for the first sample synthesized speech is obtained through manual evaluation and / or automated evaluation tools. It should be noted that whether the results are obtained through manual evaluation or automated evaluation tools, they are all based on the user's preferences and can represent the user's preference for the corresponding synthesized speech.

[0054] In an optional embodiment, the user's preference for the first sample synthesized speech is obtained by marking the first sample synthesized speech based on the user's preference for the first sample synthesized speech after the first sample synthesized speech is generated using the initial speech synthesis model. For example, the user scores or ranks the first sample synthesized speech based on prosodic naturalness, timbre similarity, and intelligibility.

[0055] In another optional embodiment, for some objectively measurable data, an automated evaluation tool may be used to assist in scoring, so as to reduce the annotation burden on the user.

[0056] In one embodiment, the preference level of the first sample synthesized speech includes preference sub-levels of at least one of the following regarding the first sample synthesized speech: prosodic naturalness, timbre similarity, and intelligibility; the automated evaluation tool includes at least one of the following: a voiceprint verification system, a speech recognition system, and a trained reward model.

[0057] Among them, the timbre similarity is used to quantify the degree of timbre similarity between the first sample synthesized speech and the target speech. In this embodiment, the voiceprint verification system can be used to determine the timbre similarity between the first sample synthesized speech and the target speech, and then the user's preference for the timbre of the first sample synthesized speech is determined based on the timbre similarity. It is understandable that when the user scores the timbre quality of the first sample synthesized speech, the reference is the speaker's target speech. Among them, the more similar the timbre of the first sample synthesized speech is to the target speech, the more likely the user is to give a higher score. Therefore, in this embodiment, the timbre similarity and the timbre preference are set to be positively correlated.

[0058] In one embodiment, a speech recognition system can be used to determine the intelligibility of the first sample synthesized speech. Speech intelligibility represents the degree to which a listener correctly recognizes the speech content. For example, if a listener correctly recognizes 75 out of 100 words, the intelligibility is 75%. In an alternative embodiment, intelligibility can be determined by calculating the WER metric using a speech recognition system (e.g., an ASR system).

[0059] It is understandable that when the user scores the quality of the first sample synthesized speech in terms of intelligibility, the higher the intelligibility of the first sample synthesized speech, the more likely the user is to give a higher score. Therefore, in this embodiment, the intelligibility and the preference sub-degree of intelligibility are set to be positively correlated.

[0060] It should be noted that rhythmic naturalness is a subjective feeling and it is difficult to measure it objectively. Therefore, manual evaluation can be used to obtain users' preference for rhythmic naturalness.

[0061] In some embodiments, a reward model can also be pre-trained using the user's preference levels for the synthesized speech output by the initial speech synthesis model in various aspects, so that the trained reward model can output a reward value used to represent the user's preference level for the first sample synthesized speech.

[0062] For example, the sample reference speech input to the initial speech synthesis model and the sample synthesized speech output by the initial speech synthesis model are used as input data for the reward model. The reward model outputs a corresponding reward value based on the input data, and then the reward model is trained using a cross-entropy loss function so that the trained reward model can simulate the user's preference for the first sample synthesized speech. The cross-entropy loss function is expressed as follows:

[0063]

[0064] Where x is the input sample reference speech, y0 and y1 are two different sample synthesized speech generated by the model, and i is the user's preference label (i=0 means the user thinks y0 is better than y1, and vice versa). The reward model is the input x and the output yi Rating (reward value)

[0065] σ represents the Sigmoid function, which maps the reward difference to the probability interval (0,1). ) represents the preference pair annotated by the user, and the annotator will mark which output is better, for example, y0 is better than y1, is the reward difference, that is, the reward difference between preferred output and non-preferred output.

[0066] The essence of this loss function is to force the reward model's score to be consistent with the user's preference through cross-entropy loss, thereby providing a reliable reward signal for subsequent reinforcement learning (such as PPO).

[0067] In summary, the initial speech synthesis model is a large model that can generate different sample synthesized speech samples based on the same sample reference speech. During the reward model training process, preference data pairs are first constructed; each preference data pair includes a highly preferred sample synthesized speech (the aforementioned preferred output) and a less preferred sample synthesized speech (the aforementioned non-preferred output). The constructed preference data pairs are then used to perform the aforementioned loss calculation, and the reward model parameters are adjusted based on the loss, so that the reward model can output a reward value that represents the user's preference for the first sample synthesized speech.

[0068] In the above scheme, the target synthesized speech is generated based on the speaker's voice features and the target text. The voice features are extracted using Q-former; and / or the target synthesized speech is generated using a target speech synthesis model, which is obtained by performing reinforcement learning on an initial speech synthesis model according to the user's preference for the first sample synthesized speech. The method of extracting the speaker's voice features using Q-former can extract voice features important for generating the target synthesized speech. Furthermore, using the voice features extracted by Q-former for speech synthesis can make the synthesized speech more consistent with the speaker's voice characteristics, making the synthesized speech more natural and accurate. Furthermore, since the target speech synthesis model can be obtained by performing reinforcement learning on the initial speech synthesis model according to the user's preference for the synthesized speech, the target speech synthesis model obtained through reinforcement learning has a higher probability of generating a target synthesized speech that meets the user's preferences, thereby facilitating the generation of a target synthesized speech that meets the user's preferences. Compared with the method of directly using a speech synthesis model to generate a target synthesized speech based on the input target text and target speech, the present application can improve the quality of the target synthesized speech by using either a Q-former to extract the speaker's voice features or a target speech synthesis model obtained by reinforcement learning using the user's preference for speech.

[0069] In one embodiment, a reinforcement learning algorithm similar to DPO may be used to train the initial speech synthesis model.

[0070] Specifically, see Figure 3 , Figure 3 This is a flow chart of an embodiment of obtaining a target speech synthesis model through reinforcement learning provided by this application. In this embodiment, based on the user's preference for the first sample synthesized speech, reinforcement learning is performed on the initial speech synthesis model to obtain the target speech synthesis model, including:

[0071] S31: The model to be reinforced learned this time is used as the model to be optimized, wherein the model to be reinforced learned for the first time is the initial speech synthesis model, and the model to be reinforced learned for the non-first time is the model obtained by the previous reinforcement learning; the first sample synthesized speech includes several second sample synthesized speech generated by the model to be optimized for each reinforcement learning.

[0072] As we can see from the previous article, the initial speech synthesis model needs to go through several rounds of reinforcement learning before it can learn the target speech synthesis model. Each round of reinforcement learning involves the adjustment of the model network parameters. In essence, the adjustment of each network parameter is based on the model obtained by the previous reinforcement learning. For example, the initial speech synthesis model needs to go through n rounds of reinforcement learning to obtain the target speech synthesis model. The first model to be reinforced is Q0 (i.e., the initial speech synthesis model). After the first reinforcement learning, Q1 is obtained. Q1 is obtained after the second reinforcement learning. Q2 is obtained after the third reinforcement learning. The above steps are repeated until Q n-1 After the nth reinforcement learning, we get Q n (Target Speech Synthesis Model).

[0073] In this embodiment, the process of performing reinforcement learning on the initial synthesis model according to the user's preference for the first sample synthesized speech to obtain the target speech synthesis model is essentially the process of adjusting the parameters of the learning model to be reinforced in each round according to the user's preference for the second sample synthesized speech generated by the learning model to be reinforced in each round.

[0074] For the model to be reinforced this time (this round), first, the model to be reinforced is used as the model to be optimized this time, and then several second sample synthesized speech generated by the model to be optimized based on the sample reference speech are obtained, and the user's preference for each second sample synthesized speech in this round is obtained. Among them, the method of obtaining the preference degree can be referred to the previous description and will not be repeated here.

[0075] S32: Based on the user's preference for each second sample synthesized speech, construct at least one positive and negative speech pair; the preference for the positive second sample synthesized speech in the positive and negative speech pair is higher than the preference for the negative second sample synthesized speech.

[0076] The second synthesized speech samples described in step S32 are generated by the model to be optimized. Step S32 is intended to construct at least one positive-negative speech pair, where each pair includes a positive second synthesized speech sample (a positive speech example) and a negative second synthesized speech sample (a negative speech example), and the positive speech sample in each pair is preferred over the negative speech sample.

[0077] S33: Obtain a first generation probability of the model to be optimized for synthesizing the positive second sample speech, and a second generation probability of the model to be optimized for synthesizing the negative second sample speech.

[0078] S34: Based on the first generation probability and the corresponding second generation probability, adjust the network parameters of the model to be optimized, and repeat the above steps to obtain the target speech synthesis model; wherein, the adjustment direction of the network parameters of the model to be optimized is: increasing the first generation probability of the positive second sample synthesized speech, and reducing the second generation probability of the negative second sample synthesized speech.

[0079] It is understandable that the purpose of reinforcement learning training is to expect the trained model to output synthesized speech that meets user preferences with a higher probability. Therefore, during the reinforcement learning training process, after obtaining the first generation probability of the optimized model for the positive second sample synthesized speech and the second generation probability of the optimized model for the negative second sample synthesized speech, the network parameters of the optimized model are adjusted in a direction that increases the first generation probability of the positive second sample synthesized speech and decreases the second generation probability of the negative second sample synthesized speech. This allows the model, after parameter adjustment, to increase the first generation probability of the positive second sample synthesized speech (positive speech) and decrease the second generation probability of the negative second sample synthesized speech (negative speech).

[0080] In one embodiment, the difference between the first generation probability of the positive second sample synthesized speech and the second generation probability of the negative second sample synthesized speech can be used as an objective function, and the network parameters of the model to be optimized can be adjusted in the direction of maximizing the objective function.

[0081] In another embodiment, the network parameters of the model to be optimized can be adjusted in the following manner. Specifically, the following steps are included:

[0082] Step 1: Obtain a third generation probability of the initial speech synthesis model generating a positive second sample synthesized speech, and obtain a fourth generation probability of the initial speech synthesis model generating a negative second sample synthesized speech.

[0083] Step 2: Obtain a first ratio between the first generation probability and the third generation probability of the positive second sample synthesized speech, and obtain a second ratio between the second generation probability and the fourth generation probability of the negative second sample synthesized speech.

[0084] Step 3: Adjust the network parameters of the model to be optimized based on the difference between the first ratio of the positive second sample synthesized speech and the second ratio of the corresponding negative second sample synthesized speech.

[0085] This embodiment takes into account that the initial speech synthesis model is trained using a large amount of training data and has good basic stability. Reinforcement learning is tuned based on the initial speech synthesis model and uses less training data. Therefore, during the reinforcement learning process, in order to enable the reinforcement learning model to maintain the basic stability of the initial speech synthesis model, it is necessary to prevent the model to be optimized from deviating excessively from the basic capabilities of the initial speech synthesis model.

[0086] Therefore, in this embodiment, when adjusting the network parameters of the model to be optimized, it is necessary to consider the probability difference between the model to be optimized and the initial speech synthesis model in generating the second sample synthesized speech, so as to prevent the model to be optimized from deviating excessively from the original capabilities of the initial speech synthesis model.

[0087] For easier understanding Figure 3 For the steps of the embodiment shown, please refer to the following formula. Figure 3 The process of the illustrated embodiment is briefly described:

[0088]

[0089] Where, is the model to be optimized, is the initial speech synthesis model, represents the input sample reference speech, represents the generated positive second sample synthesized speech, represents the generated negative second sample synthesized speech, represents the first generation probability of the second sample synthesized speech, represents the third generation probability of the initial speech synthesis model generating the second sample synthesized speech, represents a first ratio between the first generation probability and the third generation probability of the synthesized speech of the positive second sample, represents the second generation probability of the negative second sample synthesized speech, represents the fourth generation probability of the initial speech synthesis model generating the negative second sample synthesized speech, represents a second ratio between the second generation probability and the fourth generation probability of the negative second sample synthesized speech, σ represents an activation function, and β is a coefficient.

[0090] From the formula, it can be seen that step three adjusts the network parameters of the model to be optimized based on the difference between the first ratio of the positive second sample synthesized speech and the corresponding second ratio of the negative second sample synthesized speech. In fact, the network parameters are adjusted in the direction of increasing the difference between the first ratio and the second ratio, so as to train the model to be optimized to increase the probability of generating positive example speech and reduce the probability of generating negative example speech.

[0091] In another embodiment, a reinforcement learning algorithm similar to PPO may be used to train the initial speech synthesis model.

[0092] Specifically, see Figure 4 , Figure 4 This is a flow chart of another embodiment of obtaining a target speech synthesis model through reinforcement learning provided by the present application. In this embodiment, based on the user's preference for the first sample synthesized speech, reinforcement learning is performed on the initial speech synthesis model to obtain the target speech synthesis model, including:

[0093] S41: The model to be reinforced learned this time is used as the model to be optimized, wherein the model to be reinforced learned for the first time is the initial speech synthesis model, and the model to be reinforced learned for the non-first time is the model obtained by the previous reinforcement learning; the first sample synthesized speech includes several second sample synthesized speech generated by the model to be optimized for each reinforcement learning, and the preference degree of the first sample synthesized speech includes the preference sub-degrees of each second sample synthesized speech.

[0094] For details, please refer to the relevant description of step S31, which will not be repeated here.

[0095] S42: Determine the expected reward corresponding to the model to be optimized based on the preference sub-degree of each second sample synthesized speech.

[0096] The second sample synthesized speech described in step S42 is the second sample synthesized speech generated by the current model to be optimized. Step S42 is to comprehensively consider the preference sub-degrees of the second sample synthesized speech and determine the expected reward corresponding to the current model to be optimized.

[0097] Prior to executing step S42, a reward model is first obtained for obtaining a reward value for the first synthesized speech sample. The reward value represents the user's preference for the first synthesized speech sample. The reward value for the first synthesized speech sample includes sub-reward values ​​for each second synthesized speech sample. The training method for the reward model is described in the relevant sections above and is not further elaborated here.

[0098] Optionally, the expected reward corresponding to the model to be optimized can be obtained by weighted summing the sub-reward values ​​of each second sample synthesized speech using the generation probability of each second sample synthesized speech; of course, in other implementations, the existing method for obtaining the expected reward can also be directly adopted.

[0099] S43: Determine the objective function based on the expected reward.

[0100] S44: Adjust the network parameters of the model to be optimized in the direction of maximizing the objective function, and repeat the above steps to obtain the target speech synthesis model; wherein the value of the objective function is positively correlated with the expected reward.

[0101] In one embodiment, the expected reward can be directly used as the objective function, and then the network parameters of the model to be optimized are adjusted in the direction of maximizing the objective function, and the above steps are repeated until the target speech synthesis model is obtained.

[0102] In another embodiment, the product of the divergence constraint term and the target coefficient can be obtained first, and then the difference between the expected reward and the product can be used as the objective function. Then, the network parameters of the model to be optimized are adjusted in the direction of maximizing the objective function, and the above steps are repeated until the target speech synthesis model is obtained.

[0103] In one embodiment, considering that the reward model is trained by the synthetic speech generated by the initial speech synthesis model, and the initial speech synthesis model is trained using a large amount of training data, the basic stability of the initial speech synthesis model is good, and reinforcement learning is tuned on the basis of the initial speech synthesis model, and the reinforcement learning uses less training data, therefore, in the reinforcement learning process, in order to enable the reinforcement learning model to maintain the basic stability of the initial speech synthesis model, it is necessary to prevent the model to be optimized from deviating too much from the basic capabilities of the initial speech synthesis model. Therefore, in order to prevent the model to be optimized from deviating too far from the initial speech synthesis model, a constraint item can be set to constrain the degree of deviation between the model to be optimized and the initial speech synthesis model.

[0104] For example, please refer to the following formula:

[0105]

[0106] Where, Represents the sub-reward value of the second sample synthesized speech output by the reward model, reflecting the user preference, Represents the divergence constraint term, which is used to characterize the distribution difference between the synthesized speech generated by the optimized model and the initial speech synthesis model. represents the model to be optimized, represents the initial speech synthesis model, β is the target coefficient that controls the strength of the divergence constraint, E represents the expected reward, and θ represents the network parameters of the model to be optimized.

[0107] In one specific embodiment, the target text and target speech to be synthesized are first obtained. A Q-former network is then used to extract the speaker's vocal features from the target speech. A target speech synthesis model is then used to decode the target synthesized speech based on the speaker's vocal features and the target text. During the decoding process to obtain the target synthesized speech, an autoregressive approach is used to generate individual speech tokens, which are then synthesized to produce the target synthesized speech.

[0108] See also Figure 5 , Figure 5This is a flow chart of an embodiment of the target speech synthesis model training method provided by this application. The target speech synthesis model training method of this embodiment includes:

[0109] S51: Obtain the model to be optimized for the current reinforcement learning; wherein, the model to be reinforced learned for the first time is the initial speech synthesis model, and the model to be reinforced learned for the non-first time is the model obtained by the previous reinforcement learning.

[0110] S52: Obtain each third sample synthesized speech generated by the model to be optimized.

[0111] S53: Using the user's preference for each third sample synthesized speech, adjust the network parameters of the model to be optimized, and repeat the above steps to obtain the target speech synthesis model; wherein, the adjustment direction of the model to be optimized includes: increasing the generation probability of the third sample synthesized speech with a high user preference.

[0112] The target speech synthesis model training method in this embodiment, and the method of adjusting the network parameters of the model to be optimized by using the user's preference for each third sample synthesized speech, can be referenced. Figure 3 and Figure 4 The illustrated embodiment utilizes the user's preference for the first sample synthesized speech to perform reinforcement learning on the initial speech synthesis model to obtain the target speech synthesis model.

[0113] It should be noted that, considering that the small model is limited by its modeling capabilities, the generated speech has a large gap with natural speech in terms of naturalness and similarity. Therefore, in this embodiment, the above-mentioned target speech synthesis model is a large model.

[0114] See also Figure 6 , Figure 6 : This is a schematic diagram of the framework of an embodiment of a speech synthesis device provided by the present application. In this embodiment, the speech synthesis device 60 includes a first acquisition module 61, an extraction module 62, and a speech generation module 63. Among them, the first acquisition module 61 is used to obtain the target text and target speech to be synthesized, the extraction module 62 is used to extract the voice features of the speaker in the target speech, and the speech generation module 63 is used to generate a target synthesized speech based on the voice features of the speaker and the target text. The target synthesized speech is a speech that refers to the sound characteristics of the target speech and whose pronunciation content is consistent with the target text; wherein the voice features are extracted using Q-former; and / or, the target synthesized speech is generated using a target speech synthesis model, which is obtained by reinforcing the initial speech synthesis model according to the user's preference for the first sample synthesized speech, and the first sample synthesized speech is generated using the initial speech synthesis model.

[0115] In some embodiments, the preference level of the first sample synthesized speech includes preference sub-levels regarding at least one of the following items of the first sample synthesized speech: prosodic naturalness, timbre similarity, and intelligibility.

[0116] In some embodiments, each preference sub-level is obtained through manual evaluation and / or automated evaluation tools; the automated evaluation tools include at least one of the following: a voiceprint verification system, a speech recognition system and a trained reward model; wherein the voiceprint verification system is used to determine the timbre similarity between the first sample synthesized speech and the target speech, and the timbre similarity is positively correlated with the corresponding preference sub-level, and the speech recognition system is used to determine the intelligibility of the first sample synthesized speech, and the intelligibility is positively correlated with the corresponding preference sub-level; the reward model can determine the user's various preference sub-levels for the first sample synthesized speech.

[0117] In some embodiments, according to the user's preference for the first sample synthesized speech, reinforcement learning is performed on the initial speech synthesis model to obtain a target speech synthesis model, including: taking the model to be reinforced learned this time as the model to be optimized, wherein the model to be reinforced learned for the first time is the initial speech synthesis model, and the model to be reinforced learned for the non-first time is the model obtained by the previous reinforcement learning; the first sample synthesized speech includes several second sample synthesized speech generated by the model to be optimized for each reinforcement learning; based on the user's preference for each second sample synthesized speech, at least one positive and negative speech pair is constructed; the preference for the positive second sample synthesized speech in the positive and negative speech pairs is higher than the preference for the negative second sample synthesized speech; obtaining a first generation probability of the model to be optimized for the positive second sample synthesized speech, and a second generation probability of the model to be optimized for the negative second sample synthesized speech; based on the first generation probability and the corresponding second generation probability, adjusting the network parameters of the model to be optimized, and repeating the above steps to obtain the target speech synthesis model; wherein the adjustment direction of the network parameters of the model to be optimized is: increasing the first generation probability of the positive second sample synthesized speech, and reducing the second generation probability of the negative second sample synthesized speech.

[0118] In some embodiments, based on the first generation probability and the corresponding second generation probability, the network parameters of the model to be optimized are adjusted, including: obtaining the third generation probability of the initial speech synthesis model generating the positive second sample synthesized speech, and obtaining the fourth generation probability of the initial speech synthesis model generating the negative second sample synthesized speech; obtaining the first ratio between the first generation probability and the third generation probability of the positive second sample synthesized speech, and obtaining the second ratio between the second generation probability and the fourth generation probability of the negative second sample synthesized speech; based on the difference between the first ratio of the positive second sample synthesized speech and the corresponding second ratio of the negative second sample synthesized speech, the network parameters of the model to be optimized are adjusted.

[0119] In some embodiments, according to the user's preference for the first sample synthesized speech, reinforcement learning is performed on the initial speech synthesis model to obtain a target speech synthesis model, including: taking the model to be reinforced learned this time as the model to be optimized, wherein the model to be reinforced learned for the first time is the initial speech synthesis model, and the model to be reinforced learned for the non-first time is the model obtained by the previous reinforcement learning; the first sample synthesized speech includes several second sample synthesized speech generated by the model to be optimized for each reinforcement learning, and the preference level of the first sample synthesized speech includes the preference sub-level of each second sample synthesized speech; based on the preference sub-level of each second sample synthesized speech, the expected reward corresponding to the model to be optimized is determined; based on the expected reward, the objective function is determined; in the direction of maximizing the objective function, the network parameters of the model to be optimized are adjusted, and the above steps are repeated to obtain the target speech synthesis model; wherein the value of the objective function is positively correlated with the expected reward.

[0120] In some embodiments, before the speech generation module 63 generates the target synthesized speech based on the speaker's voice features and the target text, it also includes: obtaining a reward model for obtaining a reward value for the first sample synthesized speech, the reward value is used to characterize the user's preference for the first sample synthesized speech, and the reward value of the first sample synthesized speech includes a sub-reward value for each second sample synthesized speech; the expected reward is determined based on the sub-reward value of each second sample synthesized speech; and / or, based on the expected reward, determining the objective function, including: obtaining the product of the divergence constraint term and the target coefficient; the divergence constraint term is used to characterize the difference in probability distribution of each second sample synthesized speech generated by the model to be optimized and the initial speech synthesis model; and using the difference between the expected reward and the product as the objective function.

[0121] In some embodiments, the target speech synthesis model is a large model; and / or, the sound features extracted by the extraction module 62 are extracted using an audio feature extraction network, and the audio feature extraction network is composed of several attention modules, each attention module including a self-attention sub-module and a cross-attention sub-module; wherein the self-attention sub-module is used to perform self-attention processing on several input learnable query vectors to obtain a fusion result of several learnable query vectors, and different learnable query vectors are used to extract sound sub-features of different dimensions of the target speech; the cross-attention sub-module is used to use the audio features of the target speech as key vectors and value vectors respectively, and use the fusion result as the query vector, and perform cross-attention processing based on the query vector, key vector and value vector to obtain sound features.

[0122] See also Figure 7 , Figure 7It is a schematic diagram of the framework of an embodiment of the target speech synthesis model training device provided in this application. In this embodiment, the target speech synthesis model training device 70 includes a second acquisition module 71, a third acquisition module 72 and a parameter adjustment module 73. The second acquisition module 71 is used to obtain the model to be optimized for this reinforcement learning; wherein, the model to be reinforced for the first time is the initial speech synthesis model, and the model to be reinforced for the non-first time is the model obtained by the last reinforcement learning; the third acquisition module 72 is used to obtain each third sample synthesized speech generated by the model to be optimized; the parameter adjustment module 73 is used to adjust the network parameters of the model to be optimized this time using the user's preference for each third sample synthesized speech, and repeat the above steps to obtain the target speech synthesis model; wherein, the adjustment direction of the model to be optimized includes: increasing the generation probability of the third sample synthesized speech with a high degree of user preference.

[0123] See also Figure 8 , Figure 8 FIG. 8 is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application. In this embodiment, the electronic device 80 includes a memory 81 and a processor 82 coupled to each other.

[0124] The memory 81 stores program instructions, and the processor 82 is configured to execute the program instructions stored in the memory 81 to implement the steps of any of the above-described method implementations. In a specific implementation scenario, the electronic device 80 may include, but is not limited to, a microcomputer and a server. In addition, the electronic device 80 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0125] Specifically, the processor 82 is used to control itself and the memory 81 to implement the steps of any of the above-mentioned embodiments. The processor 82 may also be referred to as a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip having signal processing capabilities. The processor 82 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor may be a microprocessor or any conventional processor. In addition, the processor 82 may be implemented by an integrated circuit chip.

[0126] See also Figure 9 , Figure 9Schematic diagram of the framework of the computer-readable storage medium provided by the present application. The computer-readable storage medium 90 of the embodiment of the present application stores program instructions 91. When the program instructions 91 are executed, the method provided by any embodiment of the above-mentioned method and any non-conflicting combination is implemented. The program instructions 91 can be stored in the computer-readable storage medium 90 in the form of a program file in the form of a software product, so that a computer device (which can be a personal computer, server, or network device, etc.) can execute all or part of the steps of the method of each embodiment of the present application. The aforementioned computer-readable storage medium 90 includes: various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, server, mobile phone, and tablet.

[0127] In the above scheme, the target synthesized speech is generated based on the speaker's voice features and the target text. The voice features are extracted using Q-former; and / or the target synthesized speech is generated using a target speech synthesis model, which is obtained by performing reinforcement learning on an initial speech synthesis model according to the user's preference for the first sample synthesized speech. The method of extracting the speaker's voice features using Q-former can extract voice features important for generating the target synthesized speech. Furthermore, using the voice features extracted by Q-former for speech synthesis can make the synthesized speech more consistent with the speaker's voice characteristics, making the synthesized speech more natural and accurate. Furthermore, since the target speech synthesis model can be obtained by performing reinforcement learning on the initial speech synthesis model according to the user's preference for the synthesized speech, the target speech synthesis model obtained through reinforcement learning has a higher probability of generating a target synthesized speech that meets the user's preferences, thereby facilitating the generation of a target synthesized speech that meets the user's preferences. Compared with the method of directly using a speech synthesis model to generate a target synthesized speech based on the input target text and target speech, the present application can improve the quality of the target synthesized speech by using either a Q-former to extract the speaker's voice features or a target speech synthesis model obtained by reinforcement learning using the user's preference for speech.

[0128] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. Its specific implementation can refer to the description of the above method embodiment. For the sake of brevity, it will not be repeated here.

[0129] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0131] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0132] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0133] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the various implementation methods of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0134] The above description is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Obtain the target text and target speech to be synthesized; A Q-former is used to extract the voice features of the speaker in the target speech; the voice features include the voice characteristics of the speaker in the target speech; the Q-former is composed of a self-attention submodule and a cross-attention submodule; wherein the self-attention submodule is used to perform self-attention processing on a plurality of input learnable query vectors to obtain a fusion result of a plurality of learnable query vectors, wherein the different learnable query vectors are learned and used to extract the voice subfeatures of different dimensions of the target speech; the cross-attention submodule is used to use the audio features of the target speech as a key vector and a value vector respectively, use the fusion result as a query vector, perform cross-attention processing based on the query vector, the key vector and the value vector to obtain the voice features; the audio features of the target speech are obtained by performing speech discretization processing on the target speech to obtain the corresponding speech tokens, and then encoding the speech tokens; A target speech synthesis model is used to generate a target synthesized speech based on the speaker's voice characteristics and the target text. The target synthesized speech is a speech that refers to the sound characteristics of the target speech and has pronunciation content consistent with the target text; the target speech synthesis model is a large model.

2. The method according to claim 1, characterized in that The target speech synthesis model is obtained by performing reinforcement learning on the initial speech synthesis model according to the user's preference for the first sample synthesized speech, and the first sample synthesized speech is generated using the initial speech synthesis model; The preference level of the first sample synthesized speech includes preference sub-levels of at least one of the following items regarding the first sample synthesized speech: prosodic naturalness, timbre similarity, and intelligibility.

3. The method according to claim 2, characterized in that Each of the preference sub-levels is obtained through manual evaluation and / or automated evaluation tools; The automated assessment tool includes at least one of the following: a voiceprint verification system, a speech recognition system, and a trained reward model; Among them, the voiceprint verification system is used to determine the timbre similarity between the first sample synthesized speech and the target speech, and the timbre similarity is positively correlated with the corresponding preference sub-degree; the speech recognition system is used to determine the intelligibility of the first sample synthesized speech, and the intelligibility is positively correlated with the corresponding preference sub-degree; the reward model can determine the user's various preference sub-degrees for the first sample synthesized speech.

4. The method according to claim 1, wherein The target speech synthesis model is obtained by performing reinforcement learning on the initial speech synthesis model according to the user's preference for the first sample synthesized speech, and the first sample synthesized speech is generated using the initial speech synthesis model; The step of performing reinforcement learning on the initial speech synthesis model to obtain the target speech synthesis model according to the user's preference for the first sample synthesized speech includes: The model to be reinforced learned this time is used as the model to be optimized, wherein the model to be reinforced learned for the first time is the initial speech synthesis model, and the model to be reinforced learned for the non-first time is the model obtained by the previous reinforcement learning; the first sample synthesized speech includes a plurality of second sample synthesized speech generated by the model to be optimized in each reinforcement learning; Based on the user's preference for each second sample synthesized speech, constructing at least one positive and negative speech pair; wherein the preference for the positive second sample synthesized speech in the positive and negative speech pair is higher than the preference for the negative second sample synthesized speech; Obtaining a first generation probability of the model to be optimized for the positive second sample synthesized speech, and a second generation probability of the model to be optimized for the negative second sample synthesized speech; Based on the first generation probability and the corresponding second generation probability, adjust the network parameters of the model to be optimized, and repeat the above steps to obtain the target speech synthesis model; wherein, the adjustment direction of the network parameters of the model to be optimized is: increasing the first generation probability of the positive second sample synthesized speech, and reducing the second generation probability of the negative second sample synthesized speech.

5. The method according to claim 4, characterized in that The adjusting the network parameters of the model to be optimized based on the first generation probability and the corresponding second generation probability includes: Obtaining a third generation probability of the initial speech synthesis model generating the positive second sample synthesized speech, and obtaining a fourth generation probability of the initial speech synthesis model generating the negative second sample synthesized speech; Obtaining a first ratio between the first generation probability and the third generation probability of the positive second sample synthesized speech, and obtaining a second ratio between the second generation probability and the fourth generation probability of the negative second sample synthesized speech; Based on the difference between the first ratio of the positive second sample synthesized speech and the corresponding second ratio of the negative second sample synthesized speech, the network parameters of the model to be optimized are adjusted.

6. The method according to claim 1, characterized in that The target speech synthesis model is obtained by performing reinforcement learning on the initial speech synthesis model according to the user's preference for the first sample synthesized speech, and the first sample synthesized speech is generated using the initial speech synthesis model; The step of performing reinforcement learning on the initial speech synthesis model to obtain the target speech synthesis model according to the user's preference for the first sample synthesized speech includes: The model to be reinforced learned this time is used as the model to be optimized, wherein the model to be reinforced learned for the first time is the initial speech synthesis model, and the model to be reinforced learned for the non-first time is the model obtained by the previous reinforcement learning; the first sample synthesized speech includes a plurality of second sample synthesized speech generated by the model to be optimized in each reinforcement learning, and the preference level of the first sample synthesized speech includes the preference sub-level of each second sample synthesized speech; determining an expected reward corresponding to the model to be optimized based on the preference sub-degree of each second sample synthesized speech; Determining an objective function based on the expected reward; Adjust the network parameters of the model to be optimized in the direction of maximizing the objective function, and repeat the above steps to obtain the target speech synthesis model; wherein the value of the objective function is positively correlated with the expected reward.

7. The method according to claim 6, characterized in that Before generating the target synthesized speech based on the speaker's voice features and the target text, the method further includes: Obtaining a reward model for obtaining a reward value for the first sample synthesized speech, wherein the reward value is used to represent a user's preference for the first sample synthesized speech, and the reward value for the first sample synthesized speech includes sub-reward values ​​for each of the second sample synthesized speech; The expected reward is determined based on the sub-reward value of each second sample synthesized speech; And / or, determining an objective function based on the expected reward includes: Obtaining the product of a divergence constraint term and a target coefficient; the divergence constraint term is used to characterize the difference in probability distribution of each second sample synthesized speech generated by the current to-be-optimized model and the initial speech synthesis model; The difference between the expected reward and the product is used as the objective function.

8. A target speech synthesis model training method, characterized in that: The method comprises: Obtain the model to be optimized for this reinforcement learning; the model to be optimized for the first reinforcement learning is the initial speech synthesis model, and the model to be optimized for the second reinforcement learning is the model obtained from the previous reinforcement learning. Obtaining each third sample synthesized speech generated by the model to be optimized; Using the user's preference for each of the third sample synthesized speech, adjust the network parameters of the model to be optimized, and repeat the above steps to obtain the target speech synthesis model; wherein the adjustment direction of the model to be optimized includes: increasing the probability of generating the third sample synthesized speech with a high user preference; The target speech synthesis model is used to generate a target synthesized speech based on the speaker's voice features in the target speech extracted using the Q-former and the target text to be synthesized; The sound features include the sound features of the speaker in the target speech; the Q-former is composed of a self-attention submodule and a cross-attention submodule; wherein, the self-attention submodule is used to perform self-attention processing on several input learnable query vectors to obtain a fusion result of several learnable query vectors, and the different learnable query vectors are learned and used to extract sound sub-features of different dimensions of the target speech. The cross-attention submodule is used to use the audio features of the target speech as key vectors and value vectors respectively, and use the fusion result as the query vector, and perform cross-attention processing based on the query vector, key vector and value vector to obtain the sound features; the audio features of the target speech are obtained by performing speech discretization processing on the target speech, obtaining the corresponding speech token, and then encoding the speech token; the target synthesized speech is a speech that refers to the sound features of the target speech and whose pronunciation content is consistent with the target text; wherein, the target speech synthesis model is a large model.

9. A speech synthesis device, characterized in that: The device comprises: A first acquisition module is used to acquire the target text and target speech to be synthesized; An extraction module is configured to extract the voice features of the speaker in the target speech using a Q-former; the voice features include the voice characteristics of the speaker in the target speech; the Q-former is composed of a self-attention submodule and a cross-attention submodule; wherein the self-attention submodule is configured to perform self-attention processing on a plurality of input learnable query vectors to obtain a fusion result of a plurality of learnable query vectors, wherein the different learnable query vectors are learned and used to extract voice subfeatures of different dimensions of the target speech; the cross-attention submodule is configured to use the audio features of the target speech as a key vector and a value vector respectively, use the fusion result as a query vector, perform cross-attention processing based on the query vector, the key vector, and the value vector to obtain the voice features; the audio features of the target speech are obtained by performing speech discretization processing on the target speech to obtain corresponding speech tokens, and then encoding the speech tokens; The speech generation module is used to generate a target synthesized speech based on the speaker's voice characteristics and the target text using a target speech synthesis model. The target synthesized speech is a speech that refers to the sound characteristics of the target speech and whose pronunciation content is consistent with the target text; the target speech synthesis model is a large model.

10. A target speech synthesis model training device, characterized in that: The device comprises: The second acquisition module is used to obtain the model to be optimized for the current reinforcement learning; wherein the model to be reinforced for the first time is the initial speech synthesis model, and the model to be reinforced for the non-first time is the model obtained by the previous reinforcement learning; A third acquisition module is used to obtain each third sample synthesized speech generated by the model to be optimized; a parameter adjustment module, configured to adjust the network parameters of the model to be optimized based on the user's preference for each of the third sample synthesized speech, and repeat the above steps to obtain the target speech synthesis model; wherein the adjustment direction of the model to be optimized includes: increasing the probability of generating the third sample synthesized speech with a high user preference; The target speech synthesis model is used to generate a target synthesized speech based on the speaker's voice features in the target speech extracted using the Q-former and the target text to be synthesized; The sound features include the sound features of the speaker in the target speech; the Q-former is composed of a self-attention submodule and a cross-attention submodule; wherein, the self-attention submodule is used to perform self-attention processing on several input learnable query vectors to obtain a fusion result of several learnable query vectors, and the different learnable query vectors are learned and used to extract sound sub-features of different dimensions of the target speech. The cross-attention submodule is used to use the audio features of the target speech as key vectors and value vectors respectively, and use the fusion result as the query vector, and perform cross-attention processing based on the query vector, key vector and value vector to obtain the sound features; the audio features of the target speech are obtained by performing speech discretization processing on the target speech, obtaining the corresponding speech token, and then encoding the speech token; the target synthesized speech is a speech that refers to the sound features of the target speech and whose pronunciation content is consistent with the target text; wherein, the target speech synthesis model is a large model.

11. An electronic device, characterized in that: comprising a memory and a processor coupled to each other, The memory stores program instructions; The processor is configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program instructions that can be run by a processor, and the program instructions can be executed by the processor to implement the method according to any one of claims 1 to 8.

Images