Artificial intelligence-based speech synthesis method and device, computer equipment and medium

Abstract

The present application is suitable for the technical field of speech synthesis, and particularly relates to a speech synthesis method and device based on artificial intelligence, a computer device and a medium. The present application inputs a face image into trained age feature encoders, mouth shape feature encoders and gender feature encoders respectively, obtains age features, mouth shape features and gender features, and determines the fusion weights corresponding to the age features, mouth shape features and gender features respectively to obtain fused timbre features. The present application inputs a target text into a trained text feature encoder to obtain target text features, and inputs the fused timbre features and the target text features into a trained speech synthesis model to obtain a target speech. By extracting the age features, mouth shape features and gender features in the face image that affect the timbre of the target speaker, and determining the corresponding fusion weights, the timbre of the target speaker is comprehensively and accurately represented, the timbre embedding effect of the target speaker in the target speech is improved, and the synthesis quality of the speech is improved.

Description

TECHNICAL FIELD

[0001] The present application is suitable for the technical field of speech synthesis, and particularly relates to a speech synthesis method and device based on artificial intelligence, a computer device, and a medium. BACKGROUND

[0002] The goal of speech synthesis technology is to convert text information into a speech signal, and the target speech signal needs to embed the timbre information of a specific speaker. Existing speech synthesis technology uses a parametric method, which needs to extract the corresponding statistical model from the statistical model library for each predicted phoneme, and uses a manually modulated parameter model for speech synthesis. This method cannot effectively embed the timbre characteristics of a specific speaker in the target speech signal, resulting in poor timbre effect of the synthesized speech.

[0003] Therefore, in the field of speech synthesis technology, how to improve the timbre embedding effect of the speaker in the synthesized speech has become a problem to be solved. SUMMARY

[0004] Therefore, the embodiments of the present application provide a speech synthesis method and device based on artificial intelligence, a computer device, and a medium to solve the problem that the difference between the timbre of the synthesized speech and the target user's timbre is large in the prior art under zero speech samples or light speech samples.

[0005] In a first aspect, the embodiments of the present application provide a speech synthesis method based on artificial intelligence, which comprises:

[0006] obtaining a face image of a target speaker, inputting the face image into a trained age feature encoder to obtain an age feature of the target speaker, inputting the face image into a trained mouth shape feature encoder to obtain a mouth shape feature of the target speaker, and inputting the face image into a trained gender feature encoder to obtain a gender feature of the target speaker;

[0007] determining a first fusion weight value of the age feature, a second fusion weight value of the mouth shape feature, and a third fusion weight value of the gender feature;

[0008] weighting and summing the age feature, the mouth shape feature, and the gender feature according to the first fusion weight value, the second fusion weight value, and the third fusion weight value to determine a weighted sum result as a fusion timbre feature;

[0009] obtaining a target text of a speech to be synthesized, inputting the target text into a trained text feature encoder to obtain a target text feature;

[0010] inputting the fusion timbre feature and the target text feature into a trained speech synthesis model to obtain a target speech.

[0011] In a second aspect, an embodiment of the present application provides a speech synthesis device based on artificial intelligence, the speech synthesis device comprising:

[0012] a feature extraction module configured to obtain a face image of a target speaker, input the face image into a trained age feature encoder to obtain an age feature of the target speaker, input the face image into a trained mouth feature encoder to obtain a mouth feature of the target speaker, and input the face image into a trained gender feature encoder to obtain a gender feature of the target speaker;

[0013] a fusion weight determination module configured to determine a first fusion weight of the age feature, a second fusion weight of the mouth feature, and a third fusion weight of the gender feature;

[0014] a timbre feature fusion module configured to perform weighted summation on the age feature, the mouth feature, and the gender feature according to the first fusion weight, the second fusion weight, and the third fusion weight, and determine a weighted summation result as a fusion timbre feature;

[0015] a text feature determination module configured to obtain a target text to be synthesized, input the target text into a trained text feature encoder to obtain a target text feature;

[0016] a target speech generation module configured to input the fusion timbre feature and the target text feature into a trained speech synthesis model to obtain a target speech.

[0017] In a third aspect, an embodiment of the present application provides a computer device, the computer device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the speech synthesis method according to the first aspect when executing the computer program.

[0018] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, and the computer program is executable on a processor to implement the speech synthesis method according to the first aspect.

[0019] Compared with the prior art, the embodiment of the present application has the beneficial effects that: the face image of the target speaker is obtained, the face image is input into the trained age feature encoder to obtain the age feature of the target speaker, the face image is input into the trained mouth shape feature encoder to obtain the mouth shape feature of the target speaker, the face image is input into the trained gender feature encoder to obtain the gender feature of the target speaker, the first fusion weight value of the age feature, the second fusion weight value of the mouth shape feature, and the third fusion weight value of the gender feature are determined, then the age feature, the mouth shape feature, and the gender feature are weighted and summed according to the first fusion weight value, the second fusion weight value, and the third fusion weight value, the weighted sum result is determined as the fusion timbre feature, then the target text of the to-be-synthesized speech is obtained, the target text feature is obtained by inputting the target text into the trained text feature encoder, and the fusion timbre feature and the target text feature are input into the trained speech synthesis model to obtain the target speech. By extracting the age feature, the mouth shape feature, and the gender feature that affect the timbre of the target speaker, and dynamically determining the fusion weight values of the age feature and the mouth shape feature, the timbre of the target speaker is comprehensively and accurately represented, so that the timbre embedding effect of the target speaker in the synthesized target speech is improved, and the synthesis quality of the speech is improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 is an application environment schematic diagram of a speech synthesis method provided by the first embodiment of the present application;

[0022] Figure 2 is a flowchart of a speech synthesis method provided by the first embodiment of the present application;

[0023] Figure 3 is a structural schematic diagram of a speech synthesis device provided by the second embodiment of the present application;

[0024] Figure 4 is a structural schematic diagram of a computer device provided by the third embodiment of the present application. DETAILED DESCRIPTION

[0025] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0026] It will be understood that the terms "comprises" and / or "comprising," when used in this specification, include the presence of one or more features, integers, steps, operations, elements, and / or components described in the specification, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0027] It will be understood that the term "and / or," when used in the specification and in the following claims, is intended to mean one or more of the associated listed items can be present, and includes the possibilities of one or more of the associated listed items being present, and all possible combinations of one or more of the associated listed items.

[0028] As used in the description of the application and the following claims, the term "if" can be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon [the described condition or event] being detected" or "in response to [the described condition or event] being detected," depending on the context.

[0029] In addition, the description in the specification of the application and the appended claims, the terms "first," "second," "third," etc. are used merely as labels, and are not intended to impose numerical or sequential order unless it is clearly indicated by context. These terms are also used in the description in a relative sense and for the purpose of distinguishing between various features.

[0030] Reference in the specification to "one embodiment" or "an embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in some embodiments" in various places in the specification are not necessarily all referring to the same embodiment, although it can. Furthermore, the terms "comprises," "comprising," "includes," "including," "has," "having" and the like are intended to be open-ended terms that specifically permit the presence of one or more other features, integers, steps, operations, elements, and / or components in the example of the application. The terms "comprises," "comprising," "includes," "including," "has," "having" and the like are not meant to be open-ended terms that specifically permit the presence of one or more other features, integers, steps, operations, elements, and / or components in the example of the application.

[0031] The embodiment of the present application can acquire and process related data based on artificial intelligence technology. The artificial intelligence (AI) is the theory, method, technology and application system for using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain optimal results.

[0032] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric identification technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0033] It should be understood that the size of the serial number of each step in the following embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0034] In order to illustrate the technical scheme of the present application, the following will be described by specific embodiments.

[0035] The voice synthesis method provided by the embodiment of the present application can be applied in the application environment such as Figure 1 , wherein the client and the server communicate. The client includes but is not limited to a palm computer, a desktop computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cloud computer device, a personal digital assistant (PDA) and the like. The server can be realized by an independent server or a server cluster composed of multiple servers.

[0036] Referring to Figure 2 , it is a flowchart of a voice synthesis method provided by the embodiment of the present application, and the voice synthesis method can be applied to the client in Figure 1 , and the voice synthesis method can include the following steps:

[0037] In step S201, the face image of the target speaker is acquired, the face image is input into the trained age feature encoder to obtain the age feature of the target speaker, the face image is input into the trained mouth feature encoder to obtain the mouth feature of the target speaker, and the face image is input into the trained gender feature encoder to obtain the gender feature of the target speaker.

[0038] The face image of the target speaker can be selected from a speaking video of the target speaker. In order to ensure that the age feature, the mouth shape feature and the gender feature of the target speaker can be effectively extracted from the selected face image, a face image frame with the most open mouth shape and the most unobstructed front face in the speaking video is selected as the face image to be subjected to feature extraction.

[0039] After the face image of the target speaker is obtained, the face image is input into the trained age feature encoder to extract the age feature of the target speaker, the age feature being a feature vector representing the age of the speaker; the face image is input into the trained mouth shape feature encoder to extract the mouth shape feature of the target speaker, the mouth shape feature being a feature vector representing the mouth opening degree of the speaker when speaking normally; and the face image is input into the trained gender feature encoder to extract the gender feature of the target speaker, the gender feature being a feature vector representing the gender category of the speaker.

[0040] By inputting the face image into the trained age feature encoder, the mouth shape feature encoder and the gender feature encoder respectively, the age, the mouth opening degree and the gender of the target speaker are extracted, so as to comprehensively and accurately represent the voice characteristics of the target speaker according to the obtained age feature, the mouth shape feature and the gender feature, thereby improving the voice effect of the target speaker in the target voice generated according to the age feature, the mouth shape feature and the gender feature.

[0041] Optionally, the method for determining the mouth opening degree of each face image frame in the speaking video comprises:

[0042] M face image frames in the speaking video are intercepted;

[0043] Each face image frame is input into the trained mouth shape detection model to obtain a mouth shape bounding box of each face image frame;

[0044] The area of each mouth shape bounding box is calculated, and the face image frame corresponding to the mouth shape bounding box with the largest area is determined as the face image of the target speaker.

[0045] The trained mouth shape detection model comprises a mouth shape detection encoder and a mouth shape detection decoder. The mouth shape detection encoder is used for feature extraction of the face image frame, and the input is a normalized face image frame and the output is a Featuremap (feature map). The mouth shape detection decoder is used for upsampling the Featuremap and generating a mouth shape bounding box.

[0046] Then, the face image of the target speaker can be determined through the trained mouth shape detection model, so that the mouth shape features of the target speaker can be effectively extracted from the selected face image, and the timbre embedding effect of the target speaker in the synthesized target speech is improved.

[0047] In step S202, the first fusion weight of the age feature, the second fusion weight of the mouth shape feature, and the third fusion weight of the gender feature are determined.

[0048] In actual situations, the age feature, the mouth shape feature, and the gender feature have different influences on the timbre of the target speaker, and therefore, the age feature, the mouth shape feature, and the gender feature are fused, and the timbre of the target speaker is represented according to the fused features.

[0049] In actual situations, the age feature, the mouth shape feature, and the gender feature have different influences on the timbre of the target speaker, and therefore, the age feature, the mouth shape feature, and the gender feature are fused, and the timbre of the target speaker is represented according to the fused features.

[0050] Optionally, the first fusion weight of the age feature, the second fusion weight of the mouth shape feature, and the third fusion weight of the gender feature are determined, including:

[0051] The age feature is input into the trained age classification model to obtain N age interval probability values of the target speaker belonging to N preset age intervals;

[0052] The first fusion weight of the age feature is determined according to the N age interval probability values.

[0053] The preset age intervals include N of childhood, childhood, adolescence, youth, middle age, old age, old age, old age, and old age, and N is an integer greater than 1.

[0054] The age feature is a feature vector representing the age of the speaker, and therefore, when the first fusion weight of the age feature is determined, the age feature is first input into the trained age classification model to obtain N age interval probability values of the target speaker belonging to N preset age intervals.

[0055] The preset age intervals include N of childhood, childhood, adolescence, youth, middle age, old age, old age, old age, and old age, and N is an integer greater than 1.

[0056] According to the N age interval probability values, whether the age characteristics of the target speaker are distinct and whether the accuracy of the age classification model is high can be determined, and the reliability of the N age interval probability values can be determined to dynamically adjust the weight of the age characteristics in feature fusion, thereby improving the rationality and accuracy of feature fusion.

[0057] Before the age characteristics are input into the trained age classification model, the age classification model needs to be trained to improve the classification accuracy of the age classification model.

[0058] Optionally, the age classification model includes a fully connected layer, the sample age characteristics of the sample speaker are used as training samples, and the actual age interval of the sample speaker is used as a training label. The training process of the age classification model includes:

[0059] The sample age characteristics are input into the fully connected layer for feature mapping to obtain N age interval probability values of the sample speaker belonging to N preset age intervals;

[0060] A loss function is calculated according to the N age interval probability values and the actual age interval, and the parameters of the fully connected layer are corrected in reverse according to the gradient descent method until the loss function converges, thereby obtaining the trained age classification model.

[0061] The actual age interval of the sample speaker can be determined according to the actual age of the sample person, and the actual age interval is one of the N preset age intervals. The age classification model is used to determine N age interval probability values of the age characteristics corresponding to the speaker belonging to N preset age intervals according to the input age characteristics, and to calculate a loss function according to the N age interval probability values and the actual age interval. The loss function is used to represent the goodness of the output result, thereby correcting the parameters of the fully connected layer in reverse according to the gradient descent method until the loss function converges, to improve the classification accuracy of the age classification model, and to obtain the trained age classification model.

[0062] Optionally, the first fusion weight of the age characteristics is determined according to the N age interval probability values, including:

[0063] The maximum probability value in the N age interval probability values is determined as a target probability value;

[0064] The sum of the N-1 age interval probability values other than the target probability value in the N age interval probability values is calculated, and the sum is determined as a non-target probability value;

[0065] The ratio of the target probability value to the non-target probability value is calculated, the ratio is normalized to obtain a normalized ratio, and the normalized ratio is determined as the first fusion weight.

[0066] Wherein, for the obtained N age interval probability values, when the reliability of the N age interval probability values is high, then the reliability of the voice of the target speaker represented according to the age feature is also high, so that the weight of the age feature in the feature fusion is high, and vice versa, when the reliability of the N age interval probability values is low, then the reliability of the voice of the target speaker represented according to the age feature is also low, so that the weight of the age feature in the feature fusion is low, so as to improve the rationality and accuracy of the feature fusion. Therefore, the weight of the age feature in the feature fusion can be dynamically adjusted according to the obtained N age interval probability values, so as to improve the rationality and accuracy of the feature fusion.

[0067] Therefore, first, the maximum probability value in the N age interval probability values is determined as the target probability value, and the sum of the probability of the N-1 age interval probability values except the target probability value is calculated, and the sum of the probability is determined as the non-target probability value, then the greater the target probability value, the greater the probability that the age of the target speaker is in the age interval corresponding to the target probability value, that is, the higher the reliability of the obtained N age interval probability values, and further, the higher the reliability of the voice of the target speaker represented according to the age feature, therefore, the ratio of the target probability value to the non-target probability value is calculated, and the ratio is normalized to obtain a normalized ratio, and the normalized ratio is determined as the first fusion weight, so that the influence degree of the age of the target speaker on the voice of the target speaker is represented by the product of the age feature and the corresponding first fusion weight, so as to improve the rationality and accuracy of the feature fusion.

[0068] Optionally, the first fusion weight of the age feature, the second fusion weight of the mouth shape feature, and the third fusion weight of the gender feature are determined, comprising:

[0069] The face image is subjected to grayscale processing to obtain a face grayscale image;

[0070] The grayscale variance of the face image is calculated to determine the clarity degree of the face image;

[0071] The clarity degree is subjected to normalization processing, and the normalized clarity degree is determined as the second fusion weight of the mouth shape feature.

[0072] Wherein, the face image used to extract the mouth shape feature is a face image frame cut from the face video, the proportion of the mouth shape region to the whole face image is small, and the clarity degree of the face image has a great influence on the extraction accuracy of the mouth shape feature, thereby affecting the accuracy of the subsequent feature fusion.

[0073] Therefore, the weight of the mouth shape feature in the feature fusion can be dynamically adjusted by determining the clarity degree of the face image, so as to improve the rationality and accuracy of the feature fusion.

[0074] When the facial image is highly clear, there are more high-frequency components in the facial image. Therefore, taking the average gray value of all pixels in the facial image as a reference, the difference of the gray value of each pixel is calculated and the sum of the squares is obtained. Then, the total number of pixels is used for standardization to characterize the average degree of gray value change in the facial image. The greater the average degree of gray value change, the clearer the facial image, and the greater the weight of mouth features in feature fusion.

[0075] Specifically, the facial image is first converted to grayscale to obtain a grayscale image of the face and the grayscale value of each pixel in the grayscale image of the face. Then, the average grayscale value of all pixels is calculated. The difference between the grayscale values ​​of each pixel is calculated and the sum of squares is obtained. Then, the total number of pixels is used to standardize the image to obtain the grayscale variance of the facial image, which is the clarity of the facial image.

[0076] The greater the clarity of the facial image, the greater the weight of the mouth shape feature in feature fusion, and the greater the second fusion weight corresponding to the mouth shape feature. Therefore, after normalizing the clarity, the normalized clarity is determined as the second fusion weight of the mouth shape feature.

[0077] For example, age features are input into a trained age classification model to obtain N probability values ​​for the target speaker belonging to N preset age ranges, and these N probability values ​​are denoted as P1, P2, ..., P... N .

[0078] By comparing the probability values ​​P1, P2, ..., P of N age intervals... N The maximum probability value among the values ​​is determined to be the target probability value, denoted as P. m Then calculate the sum of probabilities of the N-1 age intervals excluding the target probability value from the N age interval probability values, determine the sum of probabilities as the non-target probability value, and denote the non-target probability value as P. f Then, the ratio of the target probability value to the non-target probability value is calculated as P:

[0079]

[0080] In the formula, P m Let P be the target probability value. f This represents the non-target probability value.

[0081] Then, the ratio P is normalized to obtain the normalized ratio P0. The normalized ratio P0 is then determined to be the first fusion weight α1.

[0082] The facial image is converted to grayscale to obtain a grayscale image of the face. The grayscale value of each pixel in the grayscale image of the face is determined. The number of rows of pixels in the grayscale image of the face is denoted as H, the number of columns of pixels in the grayscale image of the face is denoted as L, and the grayscale value of the pixel in the i-th (i = 1, 2, ..., H) row and j-th (j = 1, 2, ..., L) column of the grayscale image of the face is denoted as x. ij Then the average grayscale value of all pixels can be calculated. for:

[0083]

[0084] In the formula, H is the number of pixel rows in the grayscale image of the face, L is the number of pixel columns in the grayscale image of the face, and x ij Let be the gray value of the pixel in the i-th row and j-th column of the face grayscale image.

[0085] Average grayscale value of all pixels Based on this, the grayscale variance of the facial image is obtained by taking the sum of the squares of the differences in the grayscale values ​​of each pixel and then standardizing it using the total number of pixels. The grayscale variance s is then calculated. 2 for:

[0086]

[0087] In the formula, H is the number of pixel rows in the grayscale image of the face, L is the number of pixel columns in the grayscale image of the face, and x ij Let be the grayscale value of the pixel in the i-th row and j-th column of the facial grayscale image. This represents the average grayscale value.

[0088] Then, the sharpness is normalized to obtain the normalized sharpness s0. 2 The normalized sharpness s0 2 This is the second fusion weight α2 for the mouth shape feature.

[0089] Step S203: Based on the first fusion weight, the second fusion weight, and the third fusion weight, the age feature, mouth shape feature, and gender feature are weighted and summed to determine the weighted summation result as the fused timbre feature.

[0090] Among them, since gender is divided into male and female, and the distinction between male and female is relatively obvious, the reliability and accuracy of gender features are high; since the difference between male and female timbre is also large, gender features also have a large weight in feature fusion.

[0091] In the feature fusion process, the specific value of the third fusion weight of the gender feature can be preset according to actual conditions, and since the first fusion weight is obtained by normalizing the ratio of the target probability value and the non-target probability value, and the second fusion weight is obtained by normalizing the gray variance of the face gray image, the value range of the first fusion weight and the second fusion weight is (0, 1), therefore, the third fusion weight can be preset in the range (0, 1] to regulate the value range of the three fusion weights and improve the rationality and accuracy in the feature fusion process.

[0092] Then, the first fusion weight of the age feature, the second fusion weight of the mouth shape feature, and the third fusion weight of the gender feature are combined to perform weighted summation on the age feature, the mouth shape feature, and the gender feature, and the weighted summation result is determined as the fused timbre feature, which is used to comprehensively represent the timbre of the target speaker and effectively improve the timbre embedding effect of the target speaker in the synthesized target speech.

[0093] Optionally, the first fusion weight, the second fusion weight, and the third fusion weight are used to perform weighted summation on the age feature, the mouth shape feature, and the gender feature, and the weighted summation result is determined as the fused timbre feature, which includes:

[0094] The first fusion weight, the second fusion weight, and the third fusion weight are normalized to obtain the normalized first fusion weight, the normalized second fusion weight, and the normalized third fusion weight.

[0095] The normalized first fusion weight, the normalized second fusion weight, and the normalized third fusion weight are used to perform weighted summation on the age feature, the mouth shape feature, and the gender feature, and the weighted summation result is determined as the fused timbre feature.

[0096] The first fusion weight, the second fusion weight, and the third fusion weight are normalized to obtain the normalized first fusion weight, the normalized second fusion weight, and the normalized third fusion weight, so that the three fusion weights are on the same scale, and the normalized first fusion weight, the normalized second fusion weight, and the normalized third fusion weight are used to perform weighted summation on the age feature, the mouth shape feature, and the gender feature, and the weighted summation result is determined as the fused timbre feature.

[0097] For example, the age feature is denoted as T1, the mouth shape feature is denoted as T2, and the gender feature is denoted as T3, and the third fusion weight is preset as α3 = 1 according to actual conditions.

[0098] Then, the first fusion weight α1, the second fusion weight α2, and the third fusion weight α3 are normalized, and the normalized first fusion weight α1', the normalized second fusion weight α2', and the normalized third fusion weight α3' are:

[0099]

[0100]

[0101]

[0102] In the formula, a1 is the first fusion weight, a2 is the second fusion weight, and a3 is the third fusion weight.

[0103] Then, according to the normalized first fusion weight a1', the normalized second fusion weight a2', and the normalized third fusion weight a3', the age feature T1, the mouth shape feature T2, and the gender feature T3 are weighted and summed, so that the fused timbre feature T is obtained:

[0104] T = a1'T1 + a2'T2 + a3'T3

[0105] In the formula, a1' is the normalized first fusion weight, T1 is the age feature, a2' is the normalized second fusion weight, T2 is the mouth shape feature, a3' is the normalized third fusion weight, and T3 is the gender feature.

[0106] Step S204: obtaining a target text of the speech to be synthesized, inputting the target text into the trained text feature encoder to obtain a target text feature.

[0107] The target text is the text content corresponding to the speech to be synthesized. The synthesized speech needs to include both timbre information and text content information. Therefore, the target text is input into the trained text feature encoder to obtain the target text feature, which is used to represent the text content information in the speech to be synthesized.

[0108] Step S205: inputting the fused timbre feature and the target text feature into the trained speech synthesis model to obtain a target speech.

[0109] The fused timbre feature is used to represent the timbre information of the target speaker, and the target text feature is used to represent the text content information in the speech to be synthesized. Therefore, the fused timbre feature and the target text feature are simultaneously input into the trained speech synthesis model, so that the target speech with good timbre embedding effect of the target speaker can be obtained.

[0110] Before the fused timbre feature and the target text feature are input into the trained speech synthesis model, the speech synthesis model needs to be trained to improve the synthesis quality of the speech.

[0111] Optionally, the speech synthesis model includes an audio-visual neural network and a vocoder:

[0112] The audio-visual neural network is used for inputting the fused timbre feature and the target text feature, and outputting a speech spectrum;

[0113] The vocoder is used for inputting the speech spectrum and outputting a target speech.

[0114] The audio-visual neural network is used for inputting the fused timbre feature and the target text feature, and outputting a speech spectrum; the speech spectrum is input into the vocoder, the speech spectrum is converted into a time domain signal by the vocoder, and then the target speech is output according to the time domain signal.

[0115] In the embodiment of the present application, the face image of the target speaker is obtained, the face image is input into the trained age feature encoder to obtain the age feature of the target speaker, the face image is input into the trained mouth shape feature encoder to obtain the mouth shape feature of the target speaker, the face image is input into the trained gender feature encoder to obtain the gender feature of the target speaker, the first fusion weight of the age feature, the second fusion weight of the mouth shape feature, and the third fusion weight of the gender feature are determined, then the age feature, the mouth shape feature, and the gender feature are weighted and summed according to the first fusion weight, the second fusion weight, and the third fusion weight, the weighted and summed result is determined as the fused timbre feature, the target text of the to-be-synthesized speech is obtained, the target text is input into the trained text feature encoder to obtain the target text feature, and the fused timbre feature and the target text feature are input into the trained speech synthesis model to obtain the target speech. The age feature, the mouth shape feature, and the gender feature that affect the timbre of the target speaker are extracted, and the fusion weights corresponding to the age feature and the mouth shape feature are dynamically determined, so that the timbre of the target speaker is comprehensively and accurately represented, thereby improving the timbre embedding effect of the target speaker in the synthesized target speech and improving the synthesis quality of the speech.

[0116] The speech synthesis method of the embodiment of the present application is as follows, Figure 3 The structure block diagram of the speech synthesis device provided by the second embodiment of the present application is shown, only the parts related to the embodiments of the present application are shown for the convenience of description.

[0117] Referring to Figure 3 The speech synthesis device comprises:

[0118] The feature extraction module 31 is configured to obtain the face image of the target speaker, input the face image into the trained age feature encoder to obtain the age feature of the target speaker, input the face image into the trained mouth shape feature encoder to obtain the mouth shape feature of the target speaker, and input the face image into the trained gender feature encoder to obtain the gender feature of the target speaker.

[0119] The fusion weight determination module 32 is configured to determine a first fusion weight of the age feature, a second fusion weight of the mouth shape feature, and a third fusion weight of the gender feature.

[0120] The timbre feature fusion module 33 is configured to perform weighted summation on the age feature, the mouth shape feature, and the gender feature according to the first fusion weight, the second fusion weight, and the third fusion weight, and determine a weighted summation result as a fusion timbre feature.

[0121] The text feature determination module 34 is configured to obtain a target text of the to-be-synthesized speech, and input the target text into the trained text feature encoder to obtain a target text feature.

[0122] The target speech generation module 35 is configured to input the fusion timbre feature and the target text feature into the trained speech synthesis model to obtain a target speech.

[0123] Optionally, the fusion weight determination module 32 includes:

[0124] The probability value determination sub-module is configured to input the age feature into the trained age classification model to obtain N age interval probability values of the target speaker belonging to N preset age intervals.

[0125] The first fusion weight determination sub-module is configured to determine the first fusion weight of the age feature according to the N age interval probability values.

[0126] Optionally, the probability value determination sub-module includes:

[0127] The model training unit is configured to determine that the age classification model includes a fully connected layer, and to use a sample age feature of a sample speaker as a training sample and to use an actual age interval of the sample speaker as a training label.

[0128] The feature mapping unit is configured to input the sample age feature into the fully connected layer for feature mapping to obtain N age interval probability values of the sample speaker belonging to N preset age intervals.

[0129] The parameter correction unit is configured to calculate a loss function according to the N age interval probability values and the actual age interval, and to correct parameters of the fully connected layer in a reverse direction according to a gradient descent method until the loss function converges, to obtain the trained age classification model.

[0130] Optionally, the first fusion weight determination sub-module includes:

[0131] The target probability value determination unit is configured to determine a maximum probability value in the N age interval probability values as a target probability value.

[0132] The non-target probability value determination unit is configured to calculate a sum of N-1 age interval probability values except the target probability value in the N age interval probability values, and determine the sum as the non-target probability value.

[0133] The first fusion weight determination unit is configured to calculate a ratio of the target probability value to the non-target probability value, perform normalization processing on the ratio, and determine the normalized ratio as the first fusion weight.

[0134] Optionally, the fusion weight determination module 32 includes:

[0135] The gray processing submodule is configured to perform gray processing on the face image to obtain a face gray image.

[0136] The clarity determination submodule is configured to calculate a gray variance of the face image, and determine the gray variance as the clarity of the face image.

[0137] The second fusion weight determination submodule is configured to perform normalization processing on the clarity, and determine the normalized clarity as the second fusion weight of the mouth shape feature.

[0138] Optionally, the timbre feature fusion module 33 includes:

[0139] The weight normalization submodule is configured to perform normalization processing on the first fusion weight, the second fusion weight, and the third fusion weight to obtain the normalized first fusion weight, the normalized second fusion weight, and the normalized third fusion weight.

[0140] The fused timbre feature determination submodule is configured to perform weighted summation on the age feature, the mouth shape feature, and the gender feature according to the normalized first fusion weight, the normalized second fusion weight, and the normalized third fusion weight, and determine a result of the weighted summation as the fused timbre feature.

[0141] Optionally, the target speech generation module 35 includes:

[0142] The speech synthesis model determination submodule is configured to determine that the speech synthesis model includes an audio-visual neural network and a vocoder.

[0143] The speech spectrum determination submodule is configured to determine that the audio-visual neural network is used to input the fused timbre feature and the target text feature, and output a speech spectrum.

[0144] The target speech generation submodule is configured to determine that the vocoder is used to input the speech spectrum, and output the target speech.

[0145] It should be noted that the information interaction, execution process and the like between the above modules are based on the same concept as the method embodiments of the present application, and the specific functions and the technical effects brought by the same can be referred to the method embodiments part, and will not be repeated here.

[0146] Figure 4 A structural schematic diagram of a computer device is provided for the third embodiment of the present application. As shown in the figure, the computer device of this embodiment comprises at least one processor (only one processor is shown in the figure), a memory, and a computer program stored in the memory and executable on the at least one processor, and the processor implements the steps in any of the voice synthesis method embodiments described above when executing the computer program. Figure 4 Figure 4 The computer device can comprise, but is not limited to, a processor, a memory. Those skilled in the art can understand that the computer device is only an example and does not constitute a limitation to the computer device, and the computer device can comprise more or less components than the figure, or combine certain components, or different components, for example, it can also comprise a network interface, a display screen, an input device and the like.

[0147] The computer device can comprise, but is not limited to, a processor, a memory. Those skilled in the art can understand that the computer device is only an example and does not constitute a limitation to the computer device, and the computer device can comprise more or less components than the figure, or combine certain components, or different components, for example, it can also comprise a network interface, a display screen, an input device and the like. Figure 4 The processor can be a CPU, and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0148]

[0149] ​​The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be a memory of the computer device, and the internal memory provides an environment for running of the operating system and the computer-readable instructions in the readable storage medium. The readable storage medium can be a hard disk of the computer device, and in other embodiments, can also be an external storage device of the computer device, for example, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory can include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a BootLoader, data, and other programs, such as program codes of computer programs, etc. The memory can also be used to temporarily store data that has been output or will be output.

[0150] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the unit and module in the above device can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here. If the integrated unit is realized in the form of software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the present application realizes all or part of the processes in the above-mentioned embodiment methods, which can be completed by a computer program to instruct related hardware. The computer program can be stored in a computer readable storage medium, and when the processor executes the computer program, the steps of the above-mentioned method embodiment can be realized. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form, etc. The computer readable medium can at least include any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0151] The present application realizes all or part of the processes in the above-mentioned embodiment methods, which can also be completed by a computer program product. When the computer program product runs on the computer device, it makes the computer device execute the steps that can realize the above-mentioned method embodiments.

[0152] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0153] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0154] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / computer device and method can be implemented in other manners. For example, the described apparatus / computer device embodiments are merely schematic. For example, the division of the modules or units can be different, and each can include a plurality of sub-units. Some or all of the modules or units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0155] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e. may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0156] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An artificial intelligence-based speech synthesis method, characterized by, The voice synthesis method comprises: obtaining a face image of a target speaker, inputting the face image into a trained age feature encoder to obtain an age feature of the target speaker, inputting the face image into a trained mouth shape feature encoder to obtain a mouth shape feature of the target speaker, and inputting the face image into a trained gender feature encoder to obtain a gender feature of the target speaker; the age feature is a feature vector representing the age of the speaker; the mouth shape feature is a feature vector representing the mouth opening degree of the speaker during normal speaking; the gender feature is a feature vector representing the gender category of the speaker; the trained age feature encoder, mouth shape feature encoder and gender feature encoder are respectively used for feature extraction of the age, mouth opening degree and gender of the target speaker, so as to represent the voice characteristics of the target speaker according to the obtained age feature, mouth shape feature and gender feature; determining a first fusion weight of the age feature, a second fusion weight of the mouth shape feature, and a third fusion weight of the gender feature; specifically comprising: classifying the target speaker according to the age feature to obtain an age interval probability value; determining the first fusion weight according to the age interval probability value; and determining the second fusion weight according to the clarity of the face image of the target speaker; weighting and summing the age feature, the mouth shape feature and the gender feature according to the first fusion weight, the second fusion weight and the third fusion weight to determine a weighted sum result as a fusion voice characteristic; obtaining a target text of a to-be-synthesized voice, inputting the target text into a trained text feature encoder to obtain a target text feature; inputting the fusion voice characteristic and the target text feature into a trained voice synthesis model to obtain a target voice.

2. The speech synthesis method of claim 1, wherein, determining a first fusion weight of the age feature, a second fusion weight of the mouth shape feature, and a third fusion weight of the gender feature, comprises: inputting the age feature into a trained age classification model to obtain N age interval probability values of the target speaker belonging to N preset age intervals, the preset age intervals including N of childhood, infancy, adolescence, youth, prime, middle age, old age, longevity and longevity, and N is an integer greater than 1; determining the first fusion weight of the age feature according to the N age interval probability values.

3. The speech synthesis method according to claim 2, characterized by, The age classification model comprises a fully connected layer, and a sample age feature of a sample speaker is used as a training sample, and an actual age interval of the sample speaker is used as a training label; The training process of the age classification model comprises: inputting the sample age feature into the fully connected layer for feature mapping to obtain N age interval probability values of the sample speaker belonging to N preset age intervals; calculating a loss function according to the N age interval probability values and the actual age interval, and reversely correcting parameters of the fully connected layer according to a gradient descent method until the loss function converges, to obtain a trained age classification model.

4. The speech synthesis method of claim 2, wherein, The first fusion weight value of the age feature is determined according to N age interval probability values, comprising: determining the maximum probability value in the N age interval probability values as a target probability value; calculating the sum of the probability values of the N-1 age interval probability values except the target probability value, and determining the sum of the probability values as a non-target probability value; calculating the ratio of the target probability value to the non-target probability value, normalizing the ratio to obtain a normalized ratio, and determining the normalized ratio as the first fusion weight value.

5. The speech synthesis method of claim 1, wherein, The first fusion weight value of the age feature, the second fusion weight value of the mouth shape feature, and the third fusion weight value of the gender feature are determined, comprising: gray processing the face image to obtain a face gray image; calculating the gray variance of the face gray image to determine the clarity of the face image; normalizing the clarity to determine the normalized clarity as the second fusion weight value of the mouth shape feature.

6. The speech synthesis method of claim 1, wherein, The age feature, the mouth shape feature, and the gender feature are weighted and summed according to the first fusion weight value, the second fusion weight value, and the third fusion weight value to determine a weighted sum result as a fusion timbre feature, comprising: normalizing the first fusion weight value, the second fusion weight value, and the third fusion weight value to obtain a normalized first fusion weight value, a normalized second fusion weight value, and a normalized third fusion weight value; weighting and summing the age feature, the mouth shape feature, and the gender feature according to the normalized first fusion weight value, the normalized second fusion weight value, and the normalized third fusion weight value to determine a weighted sum result as a fusion timbre feature.

7. The speech synthesis method of claim 1, wherein, The speech synthesis model comprises an audio-visual neural network and a vocoder: The audio-visual neural network is used to input the fusion timbre feature and the target text feature, and output a speech spectrum; The vocoder is used to input the speech spectrum and output a target speech.

8. An artificial intelligence-based speech synthesis device, characterized by, The speech synthesis device comprises: a feature extraction module for obtaining a face image of a target speaker, inputting the face image into a trained age feature encoder to obtain an age feature of the target speaker, inputting the face image into a trained mouth shape feature encoder to obtain a mouth shape feature of the target speaker, and inputting the face image into a trained gender feature encoder to obtain a gender feature of the target speaker; the age feature is a feature vector representing the age of the speaker; the mouth shape feature is a feature vector representing the mouth opening degree of the speaker when speaking normally; the gender feature is a feature vector representing the gender category of the speaker; the trained age feature encoder, mouth shape feature encoder, and gender feature encoder are used for feature extraction of the age, mouth opening degree, and gender of the target speaker, respectively, to represent the timbre feature of the target speaker according to the obtained age feature, mouth shape feature, and gender feature. The fusion weight determination module is configured to determine a first fusion weight of the age feature, a second fusion weight of the mouth shape feature, and a third fusion weight of the gender feature. Specifically, the age feature is used to classify the target speaker by age to obtain age interval probability values, and the first fusion weight is determined according to the age interval probability values. The second fusion weight is determined according to the clarity of the face image of the target speaker. The timbre feature fusion module is configured to perform weighted summation on the age feature, the mouth shape feature, and the gender feature according to the first fusion weight, the second fusion weight, and the third fusion weight, and determine a weighted summation result as a fusion timbre feature. The text feature determination module is configured to obtain a target text to be synthesized, and input the target text into a trained text feature encoder to obtain a target text feature. The target speech generation module is configured to input the fusion timbre feature and the target text feature into a trained speech synthesis model to obtain a target speech.

9. A computer device, comprising: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the speech synthesis method of any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the speech synthesis method of any one of claims 1 to 7.

Images