Artificial intelligence-based speech synthesis method and device, computer equipment and medium

Abstract

The present application is suitable for the medical technology field, and particularly relates to a speech synthesis method and device, computer equipment and medium. The present application extracts the phoneme features of the target text through the phoneme encoder, extracts the style features of the reference audio using the style encoder, predicts the predicted text of the style features using the text predictor, and when the similarity between the predicted text and the real text is less than the similarity threshold, the corresponding phoneme attribute is given to each phoneme in the phoneme features by combining the style features and the preset phoneme attribute adaptation parameters, the target audio corresponding to the target text is decoded based on the audio generator, the target audio with the same style as the reference audio and the target text as the speech content is synthesized through the style features, the accuracy of the speech synthesis system is improved, and in the medical technology field, the medical staff can accurately handle a large amount of redundant and tedious repetitive work efficiently, greatly improving the work efficiency and work quality of the medical staff.

Description

Technical Field

[0001] This invention relates to the field of medical technology, and in particular to a speech synthesis method, device, computer equipment and medium based on artificial intelligence. Background Technology

[0002] Speech synthesis systems are technologies that convert text into speech, giving machines the ability to speak fluently like humans. They play a crucial role in various aspects of life, including voice dialogue systems, intelligent voice assistants, telephone information retrieval systems, in-car navigation, audiobooks, real-time information broadcasting systems, information access and communication for visually or verbally impaired individuals, and disease information dissemination. For example, in the medical field, due to the complexity of medical procedures and the large amount of repetitive basic work required of medical staff, intelligent speech technology has been increasingly widely applied in recent years. With the continuous development of internet big data and speech synthesis technology, speech synthesis technology, as an assistant to doctors, can help them handle a large amount of tedious and repetitive work due to its speed, accuracy, and low error rate, greatly reducing the burden on medical staff.

[0003] Existing speech synthesis systems can accurately read articles or answer user inquiries. However, because the synthesized text and the reference audio are consistent during training, the synthesized text often differs greatly from the reference audio in actual use. This results in synthesized speech that sounds mechanical, lacks emotion, and fails to achieve the nuanced intonation of a human voice, greatly reducing the user experience.

[0004] Therefore, in the field of medical technology, improving the accuracy of speech synthesis results has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a speech synthesis method, apparatus, computer equipment and medium based on artificial intelligence to solve the problem of low accuracy of existing speech synthesis results.

[0006] In a first aspect, embodiments of the present invention provide an artificial intelligence-based speech synthesis method, the speech synthesis method comprising:

[0007] Obtain target text and reference audio; use a trained phoneme encoder to extract phoneme features from the target text to obtain phoneme features; use a trained style encoder to extract style features from the reference audio to obtain style features.

[0008] The style features are used to predict the text using a trained text predictor to obtain the predicted text.

[0009] Obtain the real text corresponding to the reference audio, calculate the similarity between the predicted text and the real text, and detect whether the similarity is less than a preset similarity threshold;

[0010] If the similarity is detected to be less than the similarity threshold, then according to the style features and in combination with the preset phoneme attribute adaptation parameters, each phoneme in the phoneme features is assigned a corresponding phoneme attribute.

[0011] The trained audio generator is used to decode the phoneme attributes corresponding to all phonemes to obtain the target audio corresponding to the target text.

[0012] Secondly, embodiments of the present invention provide an artificial intelligence-based speech synthesis device, the speech synthesis device comprising:

[0013] The feature extraction module is used to acquire target text and reference audio, extract phoneme features from the target text using a trained phoneme encoder, and extract style features from the reference audio using a trained style encoder.

[0014] The text prediction module is used to predict the style features using a trained text predictor to obtain the predicted text.

[0015] The similarity calculation module is used to obtain the real text corresponding to the reference audio, calculate the similarity between the predicted text and the real text, and detect whether the similarity is less than a preset similarity threshold.

[0016] The attribute determination module is used to assign a corresponding phoneme attribute to each phoneme in the phoneme feature based on the style feature and in combination with the preset phoneme attribute adaptation parameters if the similarity is detected to be less than the similarity threshold.

[0017] The audio synthesis module is used to decode the phoneme attributes corresponding to all phonemes using a trained audio generator to obtain the target audio corresponding to the target text.

[0018] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the speech synthesis method as described in the first aspect.

[0019] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the speech synthesis method as described in the first aspect.

[0020] The beneficial effects of this invention compared to existing technologies are as follows: A target text and a reference audio are acquired; a trained phoneme encoder is used to extract phoneme features from the target text; a trained style encoder is used to extract style features from the reference audio; a trained text predictor is used to predict the style features to obtain the predicted text; the real text corresponding to the reference audio is acquired; the similarity between the predicted text and the real text is calculated; the similarity is checked to see if it is less than a preset similarity threshold; if the similarity is less than the threshold, then based on the style features and combined with preset phoneme attribute adaptation parameters, the phoneme features are adjusted... Each phoneme in the model is assigned a corresponding phoneme attribute. A trained audio generator is used to decode the phoneme attributes of all phonemes to obtain the target audio of the target text. A trained style encoder extracts only the style features of the reference audio without extracting the text information of the reference audio. Based on the style features and the target text, target audio with the same style as the reference audio and the target text as the speech content is synthesized. This improves the accuracy of speech synthesis in non-parallel data. In the field of medical technology, it efficiently assists medical staff in accurately handling a large amount of tedious and repetitive work, greatly improving the work efficiency and quality of medical staff. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of an application environment for an artificial intelligence-based speech synthesis method provided in Embodiment 1 of the present invention;

[0023] Figure 2 This is a flowchart illustrating an artificial intelligence-based speech synthesis method provided in Embodiment 1 of the present invention.

[0024] Figure 3 This is a schematic diagram of the structure of an artificial intelligence-based speech synthesis device provided in Embodiment 2 of the present invention;

[0025] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0027] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0028] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0030] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0032] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0033] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0034] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0035] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0036] The first embodiment of this invention provides an artificial intelligence-based speech synthesis method, which can be applied to, for example... Figure 1 In this application environment, the client communicates with the server. Clients include, but are not limited to, handheld computers, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, and personal digital assistants (PDAs). The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0037] See Figure 2 This is a flowchart illustrating an artificial intelligence-based speech synthesis method provided in Embodiment 1 of the present invention. The speech synthesis method described above can be applied to... Figure 1 In a client application, the speech synthesis method may include the following steps:

[0038] Step S201: Obtain the target text and reference audio. Use the trained phoneme encoder to extract phoneme features from the target text to obtain phoneme features. Use the trained style encoder to extract style features from the reference audio to obtain style features.

[0039] In speech synthesis tasks, reference audio provides style information for the synthesized audio, including the speaker's timbre, emotion, and prosody. Target text provides content information, and the style information of the reference audio and the content information of the target text are then fused to generate audio that matches the style of the reference audio but uses the target text as its speech content. Speech synthesis plays a crucial role in various scenarios, including voice dialogue systems, intelligent voice assistants, telephone information retrieval systems, in-vehicle navigation, audiobooks, real-time information broadcasting systems, and information acquisition and communication for visually or verbally impaired individuals.

[0040] For example, in the field of medical technology, the target text can be popular science text related to diseases, and the reference audio can be the audio of medical workers speaking. The target text can provide the content information of the synthesized audio, and the reference audio can provide the style information of the synthesized audio. The style information of the reference audio and the content information of the target text are then integrated to generate target synthesized speech that is consistent with the style of the reference audio and uses the target text as the speech content. This effectively assists medical workers in carrying out disease popular science work, reduces the burden on medical workers, and improves their work efficiency and quality.

[0041] Existing speech synthesis systems typically train their models based on sample reference audio and sample training text. The sample training text and the sample reference audio correspond to the same real-world sample text, meaning they are parallel data. However, in practice, speech synthesis systems often use a fixed reference audio and different target texts to synthesize target audio with consistent style but inconsistent content, thereby improving efficiency and reducing cost. Therefore, in actual use, the target text and the reference audio correspond to different real-world texts, meaning they are non-parallel data. This results in lower accuracy of the speech synthesized by existing systems, making it difficult to meet user needs.

[0042] Therefore, this embodiment obtains reference audio and target text that are inconsistent with the real text corresponding to the reference audio based on actual application scenarios, and extracts style information of the reference audio and text content information of the target text to synthesize target audio that is consistent with the style of the reference audio and uses the target text as speech content.

[0043] Specifically, a trained phoneme encoder and a trained style encoder are obtained. The trained phoneme encoder is used to extract phoneme features from the target text to obtain phoneme features, which are used to represent the text content information of the target text. The trained style encoder is used to extract style features from the reference audio to obtain style features, which are used to represent the speaker's timbre, emotion, and rhythm in the reference audio.

[0044] In order to address the impact of data non-parallelism between the target text and the real text corresponding to the reference audio on the speech synthesis results, the style encoder used in this embodiment is only used to extract the style features of the reference audio. Through training, the style features are made to exclude the text information of the reference audio, so that the speech content of the target audio comes more from the text features of the target text and is not affected by the text information of the reference audio, thereby improving the speech synthesis accuracy of the speech synthesis system in non-parallel data.

[0045] Optionally, the trained phoneme encoder includes a trained phoneme embedding layer and a trained phoneme feature extraction layer;

[0046] The trained phoneme encoder is used to extract phoneme features from the target text, and the resulting phoneme features include:

[0047] The trained phoneme embedding layer is used to embed phonemes into the target text to obtain a phoneme sequence.

[0048] The trained phoneme feature extraction layer is used to extract features from the phoneme sequence to obtain phoneme features.

[0049] Among them, a phoneme is the smallest unit of speech defined by the natural attributes of speech. It is analyzed based on the articulation actions in a syllable. One action constitutes one phoneme. Therefore, phonemes are the basis for the synthesis of target speech and play an important role in speech synthesis.

[0050] In this embodiment, the trained factor encoder is used to extract phoneme features from the target text to obtain phoneme features. In order to improve the accuracy of phoneme feature extraction, the trained phoneme encoder in this embodiment includes a trained phoneme embedding layer and a trained phoneme feature extraction layer.

[0051] Specifically, firstly, a pre-trained phoneme embedding layer is used to embed phonemes into the target text, resulting in a phoneme sequence. This phoneme sequence represents all phonemes in the target text in sequence. Then, a pre-trained phoneme feature extraction layer is used to extract features from the phoneme sequence, yielding phoneme features. These phoneme features represent the textual information of the target text based on its phonemes. Therefore, by converting the target text into a corresponding phoneme sequence and then extracting features from that sequence, the accuracy of feature extraction from the target text can be improved, thereby increasing the accuracy of the extracted phoneme features.

[0052] This embodiment uses a trained phoneme embedding layer to convert the target text into a corresponding phoneme sequence, and then uses a trained phoneme feature extraction layer to extract features from the phoneme sequence, thereby improving the accuracy of feature extraction of the target text and thus improving the accuracy of the extracted phoneme features.

[0053] Optionally, the training process for the style encoder includes:

[0054] Obtain the trained text predictor, sample reference audio, and the real text of each sample reference audio;

[0055] Style features are extracted from the sample reference audio using a style encoder to obtain the sample style features;

[0056] The trained text predictor is used to predict the text based on the style features of the sample, and the predicted text of the sample is obtained.

[0057] The model loss is calculated based on the predicted text and the real text of the samples. Based on the model loss, the style encoder is trained using gradient inversion until the model loss meets the preset training conditions, thus obtaining the trained style encoder.

[0058] In this embodiment, the trained style encoder is used only to extract the style features of the reference audio, and not to extract the text information of the reference audio.

[0059] During the training of the style encoder, a trained text predictor, sample reference audio, and the sample real text corresponding to each sample reference audio are obtained. The style encoder is used to extract style features from the sample reference audio. After obtaining the sample style features, the trained text predictor is used to predict the text of the extracted sample style features to obtain the sample predicted text contained in the sample style features.

[0060] Then, the model loss is calculated based on the sample real text and the sample predicted text. This loss is used to characterize the similarity between the sample predicted text and the sample real text implied in the sample style features. The corresponding model loss is then calculated based on the similarity. Correspondingly, the smaller the similarity between the sample predicted text and the sample real text, the larger the corresponding model loss.

[0061] Therefore, in this embodiment, in order to ensure that the trained style encoder is used only to extract the style features of the reference audio and not the text information of the reference audio, the style encoder is trained by gradient inversion until the model loss is less than the preset model loss threshold, thus obtaining the trained style encoder.

[0062] In this embodiment, during the training process of the style encoder, the extracted sample style features are predicted by the trained text predictor to obtain the sample predicted text contained in the sample style features. Then, the model loss is calculated based on the sample real text and the sample predicted text. The style encoder is trained using gradient inversion until the model loss is less than the preset model loss threshold, thus obtaining the trained style encoder. This achieves the goal of using the trained style encoder only to extract the style features of the reference audio, without extracting the text information of the reference audio, thereby improving the style encoder's accuracy in extracting style features from non-parallel data.

[0063] Optionally, training the style encoder using gradient inversion includes:

[0064] Obtain the model parameters of the style encoder, calculate the training gradient based on the model loss and model parameters, and determine the optimization direction of the model parameters based on the training gradient;

[0065] The optimization direction is reversed to obtain the reversed optimization direction, and the model parameters are iteratively adjusted according to the reversed optimization direction.

[0066] During the training process of the style encoder, it is necessary to iteratively adjust the model parameters of the style encoder in order to improve the accuracy of style feature extraction.

[0067] Gradient descent is a first-order optimization algorithm. When using gradient descent to find a local minimum of a function, it is necessary to take the partial derivative of the function at the current position to obtain the corresponding gradient, and then perform an iterative search based on the opposite direction of the gradient and a preset step size.

[0068] Therefore, in this embodiment, after calculating the model loss based on the sample real text and the sample predicted text, the model parameters of the style encoder are obtained. Then, the training gradient is calculated based on the functional relationship between the model loss and the model parameters. Since the training gradient is in vector form, the inverse direction of the training gradient can be determined as the optimization direction of the model parameters.

[0069] To ensure that the trained style encoder is used only to extract style features from the reference audio and not to extract text information from the reference audio, this embodiment reverses the optimization direction to obtain the optimization reversal direction. The model parameters are then iteratively adjusted according to the optimization reversal direction until the minimum value of the model loss is obtained. The optimal model parameters are then determined, and the trained style encoder is obtained.

[0070] This embodiment determines the optimization direction of model parameters based on gradient descent and reverses the optimization direction to obtain the optimization reversal direction. The model parameters are then iteratively adjusted according to the optimization reversal direction to obtain a trained style encoder. This ensures that the style features output by the trained style encoder do not contain the text information of the sample reference audio, thereby improving the style feature extraction accuracy of the style encoder.

[0071] The steps described above—acquiring target text and reference audio, extracting phoneme features from the target text using a trained phoneme encoder, and extracting style features from the reference audio using a trained style encoder—integrate the acquisition of target text that is inconsistent with the reference audio and the corresponding real text. Style features, which do not contain information from the reference audio, are extracted to represent the speaker's timbre, emotion, and rhythm in the reference audio. Phoneme features are also extracted from the target text to represent its content. Based on these style and phoneme features, a target audio with a style consistent with the reference audio and containing the target text as its speech content is synthesized. This improves the accuracy of speech synthesis in non-parallel data.

[0072] Step S202: Use the trained text predictor to predict the style features and obtain the predicted text.

[0073] In this embodiment, the goal of the trained style encoder is to improve the accuracy of style feature extraction from non-parallel data by extracting only the style features of the reference audio without extracting the text information of the reference audio. Therefore, after extracting the style features through the trained style encoder, it is also necessary to predict and extract the text content information contained in the style features to determine whether the style features extracted by the trained style encoder meet the target requirements.

[0074] Specifically, this embodiment obtains a trained text predictor, which, after model training, can accurately predict the text contained in the input content. Therefore, the trained text predictor is used to predict the style features to obtain the predicted text, which serves as the basis for judging whether the style features meet the speech synthesis target.

[0075] The above steps involve using a trained text predictor to predict the text of style features and obtaining the predicted text. The trained text predictor is used to predict the text contained in the style features, and the predicted text serves as the basis for judging whether the style features meet the speech synthesis goal. As a prerequisite for speech synthesis, it improves the accuracy of speech synthesis in non-parallel data.

[0076] Step S203: Obtain the real text corresponding to the reference audio, calculate the similarity between the predicted text and the real text, and detect whether the similarity is less than a preset similarity threshold.

[0077] In this embodiment, after using a trained text predictor to predict the style features and obtain the predicted text, in order to determine whether the style encoder satisfies the goal of extracting only the style features of the reference audio without extracting the text information of the reference audio, the real text corresponding to the reference audio is obtained, and the similarity between the predicted text and the real text is calculated. Correspondingly, the smaller the similarity, the less the style encoder has extracted the text information of the reference audio.

[0078] Therefore, a preset similarity threshold is obtained based on the actual application situation, and the style features extracted by the trained style encoder are judged to meet the requirements of speech synthesis by detecting whether the similarity is less than the preset similarity threshold.

[0079] Optionally, calculating the similarity between the predicted text and the real text includes:

[0080] The predicted text is converted into a predicted text vector using word vector technology, and the real text is converted into a real text vector using word vector technology.

[0081] Calculate the similarity between the predicted text vector and the real text vector, and determine it as the similarity between the predicted text and the real text.

[0082] In this embodiment, both the predicted text and the real text are in text form. To facilitate the calculation of the similarity between the predicted text and the real text, word vector technology is used to convert the predicted text and the real text into predicted text vectors and real text vectors, respectively. The similarity between the predicted text vector and the real text vector is then calculated and determined as the similarity between the predicted text and the real text.

[0083] This embodiment uses word vector technology to convert the predicted text and the real text into predicted text vectors and real text vectors respectively, so that the similarity between the predicted text vector and the real text vector is determined as the similarity between the predicted text and the real text, thereby improving the calculation efficiency of the similarity between the predicted text and the real text.

[0084] The steps described above—obtaining the real text corresponding to the reference audio, calculating the similarity between the predicted text and the real text, and detecting whether the similarity is less than a preset similarity threshold—characterize the degree to which the style encoder extracts textual information from the reference audio by calculating the similarity between the predicted text and the real text. By detecting whether the similarity is less than the preset similarity threshold, it is determined whether the style features meet the speech synthesis target and serves as a prerequisite for speech synthesis, thereby improving the accuracy of speech synthesis in non-parallel data.

[0085] Step S204: If the detected similarity is less than the similarity threshold, then according to the style features and combined with the preset phoneme attribute adaptation parameters, assign the corresponding phoneme attribute to each phoneme in the phoneme features.

[0086] Since a smaller similarity between the predicted text and the real text indicates a lower degree of textual information extraction by the style encoder from the reference audio, when a similarity greater than or equal to the similarity threshold is detected, it is determined that the style feature does not meet the speech synthesis objective. If this style feature is used for speech synthesis, the accuracy of the text content in the synthesized speech cannot be guaranteed, thus reducing the accuracy of the speech synthesis result.

[0087] Therefore, in this embodiment, when the similarity is less than the similarity threshold, after determining that the style features meet the speech synthesis target, the corresponding phoneme attributes are assigned to each phoneme in the phoneme features according to the style features and in combination with the preset phoneme attribute adaptation parameters, so as to obtain all phoneme attribute information corresponding to each phoneme, which serves as the basis for speech synthesis and improves the accuracy of the speech synthesis results.

[0088] From an acoustic perspective, the style of audio is related to the pitch, energy, and duration of phonemes. Therefore, in this embodiment, all phoneme attributes corresponding to each phoneme are set to include pitch, energy, and duration.

[0089] Specifically, if the detected similarity is less than the similarity threshold, then based on the style features and combined with the preset phoneme attribute adaptation parameters, each phoneme in the phoneme features is assigned a corresponding phoneme attribute. On the basis of the preset phoneme attribute adaptation parameters, the style features and phoneme features are combined to determine all phoneme attributes corresponding to each phoneme. Then, all phoneme attributes corresponding to all phonemes simultaneously contain the style information of the reference audio and the text information of the target text, which can be used as the basis for speech synthesis to synthesize the target speech.

[0090] In one embodiment, the preset phoneme attribute adaptation parameters are the model parameters in a preset trained phoneme attribute adapter. Style features and phoneme features are then fused to obtain a fused feature, which integrates style information from the reference audio and text information from the target text. This fused feature is then input into the trained phoneme attribute adapter for decoding, outputting the phoneme attribute corresponding to each phoneme in the phoneme feature, which serves as the basis for speech synthesis.

[0091] Optionally, based on style characteristics and combined with preset phoneme attribute adaptation parameters, each phoneme in the phoneme characteristics is assigned a corresponding phoneme attribute.

[0092] Obtain the preset phoneme attribute adaptation parameters, which include preset pitch adaptation parameters, preset energy adaptation parameters, and preset duration adaptation parameters.

[0093] Based on style characteristics and combined with preset pitch adaptation parameters, each phoneme in the phoneme characteristics is assigned a corresponding pitch, each phoneme in the phoneme characteristics is assigned a corresponding energy, and each phoneme in the phoneme characteristics is assigned a corresponding duration, based on preset duration adaptation parameters.

[0094] The pitch, energy, and duration of each phoneme in the phoneme features are obtained.

[0095] In this embodiment, since all phoneme attributes corresponding to each phoneme are set, including pitch, energy, and duration, correspondingly, preset phoneme attribute adaptation parameters are set, including preset pitch adaptation parameters, preset energy adaptation parameters, and preset duration adaptation parameters, so as to obtain the pitch, energy, and duration of each phoneme respectively.

[0096] Specifically, based on style characteristics and combined with preset pitch adaptation parameters, each phoneme in the phoneme characteristics is assigned a corresponding pitch, combined with preset energy adaptation parameters, each phoneme in the phoneme characteristics is assigned a corresponding energy, and combined with preset duration adaptation parameters, each phoneme in the phoneme characteristics is assigned a corresponding duration, thereby obtaining the pitch, energy, and duration corresponding to each phoneme in the phoneme characteristics.

[0097] In one embodiment, the preset phoneme attribute adaptation parameters are the model parameters in the preset trained phoneme attribute adapter, and the trained phoneme attribute adapter includes a trained phoneme pitch adapter, a trained phoneme energy adapter, and a trained phoneme duration adapter.

[0098] The stylistic features and phoneme features are then fused together to obtain a fused feature. This fused feature is then input into a trained phoneme pitch adapter for decoding, outputting the pitch corresponding to each phoneme in the phoneme feature. The fused feature is then input into a trained phoneme energy adapter for decoding, outputting the energy corresponding to each phoneme in the phoneme feature. Finally, the fused feature is input into a trained phoneme duration adapter for decoding, outputting the duration corresponding to each phoneme in the phoneme feature. Thus, the pitch, energy, and duration corresponding to each phoneme in the phoneme feature are obtained.

[0099] This embodiment sets preset phoneme attribute adaptation parameters, including preset pitch adaptation parameters, preset energy adaptation parameters, and preset duration adaptation parameters. By combining style features, phoneme features, and preset phoneme attribute adaptation parameters, the pitch, energy, and duration of each phoneme in the phoneme features are obtained to characterize the style information of each phoneme, thereby improving the richness of style information and thus improving the accuracy of speech synthesis results.

[0100] The above-mentioned step of assigning corresponding phoneme attributes to each phoneme in the phoneme features based on style features and preset phoneme attribute adaptation parameters if the similarity detected is less than the similarity threshold is only performed when the style features meet the speech synthesis target. In this step, the phoneme attributes corresponding to each phoneme in the phoneme features are obtained by combining style features, phoneme features and preset phoneme attribute adaptation parameters. Thus, all phoneme attributes corresponding to all phonemes simultaneously contain the style information of the reference audio and the text information of the target text, thereby improving the accuracy of the speech synthesis results.

[0101] Step S205: Use the trained audio generator to decode the phoneme attributes corresponding to all phonemes to obtain the target audio of the target text.

[0102] In this context, all phoneme attributes corresponding to all phonemes contain both the style information of the reference audio and the text information of the target text. Therefore, the trained audio generator can be used to decode the phoneme attributes corresponding to all phonemes to obtain the target audio of the target text. This target audio is the audio that is consistent with the style of the reference audio and uses the target text as its speech content.

[0103] Optionally, the trained audio generator includes a trained decoder and a trained vocoder;

[0104] The trained audio generator is used to decode the phoneme attributes corresponding to all phonemes to obtain the target audio for the target text, including:

[0105] The trained decoder is used to decode the phoneme attributes corresponding to all phonemes to obtain the target Mel spectrum;

[0106] The target Mel spectrum is decoded using a trained vocoder to obtain the target audio of the corresponding target text.

[0107] Because the human ear is more sensitive to distinguishing low-frequency signals than high-frequency signals, the perceived distance between two pairs of frequencies that are equidistant in the frequency domain may not be equal. Therefore, in text-to-speech conversion tasks, the frequency domain scale is usually adjusted based on the Mel spectrum to ensure that the human ear perceives the same distance between two pairs of frequencies that are equidistant in the new frequency domain scale, thereby improving the human ear's ability to perceive speech.

[0108] Therefore, Mel spectrum is widely used in the field of acoustics. In this embodiment, the trained audio generator includes a trained decoder and a trained vocoder. The trained decoder is used to decode the phoneme attributes corresponding to all phonemes to obtain the target Mel spectrum, and the trained vocoder is used to decode the target Mel spectrum to obtain the target audio of the corresponding target text.

[0109] In this embodiment, the trained audio generator includes a trained decoder and a trained vocoder. The trained decoder is used to decode the phoneme attributes corresponding to all phonemes to obtain the target Mel spectrum. The trained vocoder is used to decode the target Mel spectrum to obtain the target audio corresponding to the target text, thereby improving the accuracy of the target audio.

[0110] The above steps involve using a trained audio generator to decode the phoneme attributes corresponding to all phonemes to obtain the target audio for the target text. By using the trained audio generator to decode the phoneme attributes that simultaneously contain the style information of the reference audio and the text information of the target text, the target audio with the same style as the reference audio and the target text as the speech content is obtained, thus improving the accuracy of the target audio.

[0111] This invention, in its embodiments, acquires target text and reference audio. A trained phoneme encoder extracts phoneme features from the target text, yielding phoneme features. A trained style encoder extracts style features from the reference audio, yielding style features. A trained text predictor predicts text based on these style features, yielding predicted text. The corresponding real text is then obtained, and the similarity between the predicted and real text is calculated. The similarity is checked against a preset similarity threshold. If the similarity is below the threshold, then based on the style features and a preset phoneme attribute adaptation parameter, each phoneme in the phoneme features is... By assigning corresponding phoneme attributes and using a trained audio generator to decode the phoneme attributes of all phonemes, the target audio corresponding to the target text is obtained. The trained style encoder extracts only the style features of the reference audio without extracting the text information of the reference audio. Based on the style features and the target text, a target audio with the same style as the reference audio and the target text as the speech content is synthesized. This improves the accuracy of speech synthesis in non-parallel data. In the field of medical technology, this system efficiently assists medical staff in accurately handling a large amount of tedious and repetitive work, greatly improving the work efficiency and quality of medical staff.

[0112] Corresponding to the AI-based speech synthesis method in the above embodiments, Figure 3 A structural block diagram of an artificial intelligence-based speech synthesis device provided in Embodiment 2 of the present invention is given. For ease of explanation, only the parts related to the embodiments of the present invention are shown.

[0113] See Figure 3 The speech synthesis device includes:

[0114] Feature extraction module 31 is used to acquire target text and reference audio, extract phoneme features from target text using a trained phoneme encoder, and extract style features from reference audio using a trained style encoder.

[0115] Text prediction module 32 is used to predict the style features using a trained text predictor to obtain the predicted text;

[0116] The similarity calculation module 33 is used to obtain the real text corresponding to the reference audio, calculate the similarity between the predicted text and the real text, and detect whether the similarity is less than the preset similarity threshold.

[0117] The attribute determination module 34 is used to assign corresponding phoneme attributes to each phoneme in the phoneme features based on the style features and in combination with the preset phoneme attribute adaptation parameters if the detected similarity is less than the similarity threshold.

[0118] The audio synthesis module 35 is used to decode the phoneme attributes corresponding to all phonemes using a trained audio generator to obtain the target audio of the corresponding target text.

[0119] Optionally, the trained phoneme encoder includes a trained phoneme embedding layer and a trained phoneme feature extraction layer, wherein the feature extraction module 31 includes:

[0120] The phoneme embedding submodule is used to embed phonemes into the target text using the trained phoneme embedding layer to obtain a phoneme sequence.

[0121] The phoneme feature extraction submodule is used to extract features from the phoneme sequence using the trained phoneme feature extraction layer to obtain phoneme features.

[0122] Optionally, the similarity calculation module 33 mentioned above includes:

[0123] The text vector conversion submodule is used to convert predicted text into predicted text vectors and real text into real text vectors using word vector techniques.

[0124] The similarity calculation submodule is used to calculate the similarity between the predicted text vector and the real text vector, and determine the similarity between the predicted text and the real text.

[0125] Optionally, the attribute determination module 34 mentioned above includes:

[0126] The parameter acquisition submodule is used to acquire preset phoneme attribute adaptation parameters, which include preset pitch adaptation parameters, preset energy adaptation parameters, and preset duration adaptation parameters.

[0127] The attribute determination submodule is used to assign a corresponding pitch to each phoneme in the phoneme feature based on style characteristics and preset pitch adaptation parameters, assign a corresponding energy to each phoneme in the phoneme feature based on preset energy adaptation parameters, and assign a corresponding duration to each phoneme in the phoneme feature based on preset duration adaptation parameters.

[0128] The attribute acquisition submodule is used to obtain the pitch, energy, and duration of each phoneme in the phoneme features.

[0129] Optionally, the trained audio generator includes a trained decoder and a trained vocoder, and the aforementioned audio synthesis module 35 includes:

[0130] The attribute decoding submodule is used to decode the phoneme attributes corresponding to all phonemes using the trained decoder to obtain the target Mel spectrum;

[0131] The audio synthesis submodule is used to decode the target Mel spectrum using a trained vocoder to obtain the target audio of the corresponding target text.

[0132] Optionally, the feature extraction module 31 mentioned above includes:

[0133] The sample acquisition submodule is used to acquire the trained text predictor, sample reference audio, and the sample real text corresponding to each sample reference audio.

[0134] The feature extraction submodule is used to extract style features from the sample reference audio using a style encoder to obtain sample style features.

[0135] The text prediction submodule is used to predict the text based on the style features of the samples using a trained text predictor, and to obtain the predicted text of the samples.

[0136] The model training submodule is used to calculate the model loss based on the predicted text and the real text of the samples. Based on the model loss, the style encoder is trained by gradient inversion until the model loss meets the preset training conditions, and the trained style encoder is obtained.

[0137] Optionally, the above model training submodule includes:

[0138] The optimization direction determination unit is used to obtain the model parameters of the style encoder, calculate the training gradient based on the model loss and model parameters, and determine the optimization direction of the model parameters based on the training gradient.

[0139] The model parameter adjustment unit is used to reverse the optimization direction to obtain the optimization reversal direction, and then iteratively adjust the model parameters according to the optimization reversal direction.

[0140] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0141] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the diagram), a memory, and a computer program stored in the memory and executable on at least one processor, which, when executed by the processor, implements the steps in any of the above-described speech synthesis method embodiments.

[0142] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 4The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0143] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0144] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of a computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0145] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0146] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0147] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0148] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0149] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0150] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0151] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A speech synthesis method based on artificial intelligence, characterized in that, The speech synthesis method includes: Obtain target text and reference audio; use a trained phoneme encoder to extract phoneme features from the target text to obtain phoneme features; use a trained style encoder to extract style features from the reference audio to obtain style features. The style features are used to predict the text using a trained text predictor to obtain the predicted text. Obtain the real text corresponding to the reference audio, calculate the similarity between the predicted text and the real text, and detect whether the similarity is less than a preset similarity threshold; If the similarity is detected to be less than the similarity threshold, then according to the style features and in combination with the preset phoneme attribute adaptation parameters, each phoneme in the phoneme features is assigned a corresponding phoneme attribute. The trained audio generator is used to decode the phoneme attributes corresponding to all phonemes to obtain the target audio corresponding to the target text. The training process of the style encoder includes: Obtain the trained text predictor, sample reference audio, and the real text of each sample reference audio; The style encoder is used to extract style features from the sample reference audio to obtain sample style features; The trained text predictor is used to predict the text of the sample style features to obtain the sample predicted text. The similarity between the predicted text and the real text is calculated. Based on the similarity, the model loss is obtained. Based on the model loss, the style encoder is trained using gradient inversion until the model loss meets the preset training conditions, thus obtaining a trained style encoder. The smaller the similarity between the predicted text and the real text, the larger the model loss.

2. The speech synthesis method according to claim 1, characterized in that, The trained phoneme encoder includes a trained phoneme embedding layer and a trained phoneme feature extraction layer; The step of using a trained phoneme encoder to extract phoneme features from the target text to obtain phoneme features includes: The target text is embedded with phonemes using a trained phoneme embedding layer to obtain a phoneme sequence. The phoneme feature extraction layer, which has been trained, is used to extract features from the phoneme sequence to obtain phoneme features.

3. The speech synthesis method according to claim 1, characterized in that, The calculation of the similarity between the predicted text and the real text includes: The predicted text is converted into a predicted text vector using word vector technology, and the real text is converted into a real text vector using word vector technology. The similarity between the predicted text vector and the real text vector is calculated and determined as the similarity between the predicted text and the real text.

4. The speech synthesis method according to claim 1, characterized in that, The step of assigning corresponding phoneme attributes to each phoneme in the phoneme features based on the style features and in combination with preset phoneme attribute adaptation parameters includes: Obtain preset phoneme attribute adaptation parameters, which include preset pitch adaptation parameters, preset energy adaptation parameters, and preset duration adaptation parameters. Based on the style features and the preset pitch adaptation parameters, each phoneme in the phoneme features is assigned a corresponding pitch, and based on the preset energy adaptation parameters, each phoneme in the phoneme features is assigned a corresponding energy, and based on the preset duration adaptation parameters, each phoneme in the phoneme features is assigned a corresponding duration. The pitch, energy, and duration of each phoneme in the phoneme features are obtained.

5. The speech synthesis method according to claim 4, characterized in that, The trained audio generator includes a trained decoder and a trained vocoder. The step of using a trained audio generator to decode the phoneme attributes corresponding to all phonemes to obtain the target audio corresponding to the target text includes: The trained decoder is used to decode the phoneme attributes corresponding to all phonemes to obtain the target Mel spectrum; The trained vocoder is used to decode the target Mel spectrum to obtain the target audio corresponding to the target text.

6. The speech synthesis method according to claim 1, characterized in that, The step of training the style encoder using gradient inversion includes: Obtain the model parameters of the style encoder, calculate the training gradient based on the model loss and the model parameters, and determine the optimization direction of the model parameters based on the training gradient; The optimization direction is reversed to obtain the optimization reversal direction, and the model parameters are iteratively adjusted according to the optimization reversal direction.

7. A speech synthesis device based on artificial intelligence, characterized in that, The speech synthesis device includes: The feature extraction module is used to acquire target text and reference audio, extract phoneme features from the target text using a trained phoneme encoder, and extract style features from the reference audio using a trained style encoder. The text prediction module is used to predict the style features using a trained text predictor to obtain the predicted text. The similarity calculation module is used to obtain the real text corresponding to the reference audio, calculate the similarity between the predicted text and the real text, and detect whether the similarity is less than a preset similarity threshold. The attribute determination module is used to assign a corresponding phoneme attribute to each phoneme in the phoneme feature based on the style feature and in combination with the preset phoneme attribute adaptation parameters if the similarity is detected to be less than the similarity threshold. The audio synthesis module is used to decode the phoneme attributes corresponding to all phonemes using a trained audio generator to obtain the target audio corresponding to the target text. The feature extraction module includes: The sample acquisition submodule is used to acquire the trained text predictor, sample reference audio, and the sample real text corresponding to each sample reference audio. The feature extraction submodule is used to extract style features from the sample reference audio using the style encoder to obtain sample style features; The text prediction submodule is used to perform text prediction on the sample style features using the trained text predictor to obtain the sample predicted text. The model training submodule is used to calculate the similarity between the predicted sample text and the real sample text, obtain the model loss based on the similarity between the predicted sample text and the real sample text, and train the style encoder using gradient inversion based on the model loss until the model loss meets the preset training conditions to obtain the trained style encoder. The smaller the similarity between the predicted sample text and the real sample text, the larger the model loss.

8. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the speech synthesis method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the speech synthesis method as described in any one of claims 1 to 6.

Images