Speech synthesis method, device and equipment and storage medium thereof
By extracting phoneme sequences and fine-tuning processing models to adjust frame length information, the problem of speech synthesis in the existing technology requires a large amount of corpus data, and high-quality and resource-saving speech synthesis is achieved, which is suitable for industries such as financial customer service.
Patent Information
- Application Number
- CN202411081390.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-08
AI Technical Summary
The existing speech synthesis technology requires a large amount of corpus data to achieve high-quality speech synthesis during text to speech, resulting in resource consumption and noise problems.
By obtaining the audio data and text of the target object, extracting the phoneme sequence, and using speech feature extraction technology and fine-tuning processing model to adjust the frame length information of the phoneme sequence, perform speech synthesis, reduce subsequent fine-tuning operations, and save computing resources.
It realizes high-quality voice synthesis, reduces the demand for the audio data of the target object, ensures the close authenticity of voice, and is suitable for financial customer service and other industries to provide high-quality audio services.
Smart Images

Figure CN118918879A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech synthesis technology, which is applied to text-to-speech scenarios, and in particular to a speech synthesis method, apparatus, device and storage medium thereof. Background Art
[0002] With the rapid development of the Internet, especially the rapid development of the self-media industry and the financial industry, traditional speech synthesis is increasingly moving towards AI intelligent speech synthesis, and speech synthesis technology is often required, that is, outputting a specified speech with the pronunciation of another person object.
[0003] The existing speech synthesis task mainly adopts the method of splitting speech into multiple features such as content, rhythm, pitch, timbre, etc., and then adopts supervised technology and random sampling methods to extract corresponding features. However, this scheme is prone to a large amount of partial duplication in feature selection, resulting in excessive encoding and decoding, which easily introduces more noise and cannot guarantee the quality of speech synthesis. In addition, a large amount of target speaker text and corpus pairs are required, resulting in a large amount of annotation consumption. Therefore, the existing technology needs a large amount of corpus data to achieve high-quality speech synthesis in text-to-speech synthesis. Summary of the invention
[0004] The purpose of the embodiments of the present application is to propose a speech synthesis method, apparatus, device and storage medium thereof to solve the problem in the prior art that a large amount of corpus data is required to achieve high-quality speech synthesis in text-to-speech synthesis.
[0005] In order to solve the above technical problems, the present application provides a speech synthesis method, which adopts the following technical solutions:
[0006] A speech synthesis method comprises the following steps:
[0007] Acquire audio data of a target object, wherein the target object includes a real speaker;
[0008] Acquire a target text, wherein the target text includes a text to be speech synthesized;
[0009] Converting the target text into a phoneme sequence;
[0010] Extracting audio features from the audio data using speech feature extraction technology, and preliminarily determining audio frame length information occupied by different phonemes when the target object emits them based on the audio feature extraction results;
[0011] Inputting the phoneme sequence into a preset fine-tuning processing model, adjusting the sequence frame length information of the phoneme sequence based on preset reference information, and obtaining a fine-tuned phoneme sequence, wherein the preset reference information is the audio frame length information respectively occupied by the target object when uttering different phonemes;
[0012] Speech synthesis is performed according to the phoneme sequence completed by the fine-tuning.
[0013] Furthermore, the step of converting the target text into a phoneme sequence specifically includes:
[0014] Extracting syntactic features of each sentence in the target text and word features of all words in each sentence;
[0015] Determining the pronunciation phoneme corresponding to each word in the target text according to the syntactic features and the word features;
[0016] The target text is converted into a phoneme sequence according to the pronunciation phonemes.
[0017] Furthermore, the step of extracting the syntactic features of each sentence in the target text and the word features of all words in each sentence specifically includes:
[0018] Perform sentence splitting on the target text to obtain split sentences; and,
[0019] Perform word segmentation on the split sentences to obtain the words contained in each sentence;
[0020] Performing sentence pronunciation and sentence tone marking for each character in the split sentences respectively, and obtaining tone change information corresponding to the pronunciation of each character in the sentence in the entire sentence according to the marking results and a preset character pronunciation dictionary, wherein the character pronunciation dictionary marks the initial pronunciation information and initial tone information of each character before forming a word or sentence;
[0021] Based on the annotation results, obtaining the sentence pronunciation annotations and sentence tone annotations corresponding to the words respectively contained in each sentence as the pronunciation information of each word;
[0022] The step of converting the target text into a phoneme sequence according to the pronunciation phonemes specifically comprises:
[0023] Obtaining sequence position information of each word in the target text;
[0024] According to the sequence position information, the pronunciation phonemes corresponding to each word in the target text are serialized and sorted to obtain the sorting result as the phoneme sequence.
[0025] Furthermore, the step of determining the pronunciation phonemes corresponding to each word in the target text according to the syntactic features and the word features specifically includes:
[0026] According to the character pronunciation dictionary, identifying whether the current character is a polyphonic character; and
[0027] Identify whether the current word involves stress conversion in the corresponding whole sentence pronunciation according to the syntactic features and the word features;
[0028] If the current character is a non-polyphonetic character and the pronunciation of the corresponding entire sentence does not involve stress conversion, the unique pronunciation of the current character is used as the pronunciation phoneme;
[0029] If the current character is a non-polyphonetic character but the pronunciation of the corresponding entire sentence involves stress conversion, the pronunciation of the current character after stress conversion is used as the pronunciation phoneme;
[0030] If the current character is a polyphonic character and the pronunciation of the corresponding entire sentence does not involve stress conversion, identifying the pronunciation of the current character in the corresponding word according to the word feature, and using the pronunciation as the pronunciation phoneme of the current character;
[0031] If the current character is a polyphone but the corresponding pronunciation of the entire sentence involves stress conversion, the pronunciation of the current character in the corresponding word is identified according to the word features, and the stress conversion of the pronunciation is identified according to the syntactic features, and the stress conversion result of the pronunciation is used as the pronunciation phoneme.
[0032] Furthermore, the step of extracting audio features from the audio data using speech feature extraction technology and preliminarily determining the audio frame length information respectively occupied by the target object when emitting different phonemes based on the audio feature extraction result specifically includes:
[0033] Performing frame processing on the audio data to obtain continuous audio frame segments;
[0034] Extracting audio features from the continuous audio frame segments using an MFCC coefficient method or an LPC coefficient method;
[0035] Inputting the audio features into a preset acoustic feature recognition model to identify the pronunciation phonemes corresponding to each frame of audio;
[0036] By counting and sorting the pronunciation phonemes corresponding to each frame of audio, the audio frame length information occupied by the target object when uttering different phonemes is preliminarily determined.
[0037] Further, the fine-tuning processing model is composed of a phoneme encoder, a LoRA fine-tuning layer and an Adapter fine-tuning layer. The step of inputting the phoneme sequence into a preset fine-tuning processing model, adjusting the sequence frame length information of the phoneme sequence through the fine-tuning processing model, and obtaining a fine-tuned phoneme sequence specifically includes:
[0038] Using the embedding layer of the phoneme encoder to convert the phoneme sequence into a vector representation, and adding position encoding information to each phoneme in the phoneme sequence, wherein the phoneme encoder includes an encoder of a transformer structure;
[0039] Input the vector representation and the position encoding information corresponding to each phoneme into the coding sublayer to which the LoRA fine-tuning layer is added, and after encoding processing by the coding sublayer, output the phoneme sequence fine-tuned by the LoRA fine-tuning layer;
[0040] The phoneme sequence fine-tuned by the LoRA fine-tuning layer is input into the Adapter fine-tuning layer, and fine-tuning is continued to obtain a fine-tuned phoneme sequence, wherein the LoRA fine-tuning layer and the Adapter fine-tuning layer are used to adjust the frame lengths of different phonemes in the phoneme sequence according to the audio features of the target object and the phoneme mapping relationship.
[0041] Furthermore, the step of performing speech synthesis according to the fine-tuned phoneme sequence specifically includes:
[0042] Mapping the fine-tuned phoneme sequence into frame-by-frame audio features;
[0043] Inputting the frame-by-frame audio features into a decoder of a transformer structure corresponding to the phoneme encoder, and obtaining a waveform audio signal through decoding processing;
[0044] The waveform audio signal is used as a speech synthesis result and sent to a preset speech output terminal.
[0045] In order to solve the above technical problems, the embodiment of the present application also provides a speech synthesis device, which adopts the following technical solution:
[0046] A speech synthesis device, comprising:
[0047] An audio data acquisition module, used to acquire audio data of a target object, wherein the target object includes a real speaker;
[0048] A target text acquisition module, used to acquire a target text, wherein the target text includes a text to be speech synthesized;
[0049] A phoneme sequence conversion module, used for converting the target text into a phoneme sequence;
[0050] An audio feature extraction module is used to extract audio features from the audio data using a speech feature extraction technique, and preliminarily determine the audio frame length information occupied by the target object when emitting different phonemes based on the audio feature extraction result;
[0051] A phoneme sequence fine-tuning module, used to input the phoneme sequence into a preset fine-tuning processing model, and adjust the sequence frame length information of the phoneme sequence based on preset reference information to obtain a fine-tuned phoneme sequence, wherein the preset reference information is the audio frame length information respectively occupied by the target object when uttering different phonemes;
[0052] The speech synthesis module is used to perform speech synthesis according to the phoneme sequence completed by the fine-tuning.
[0053] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0054] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned speech synthesis method when executing the computer-readable instructions.
[0055] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0056] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech synthesis method as described above.
[0057] Compared with the prior art, the embodiments of the present application have the following beneficial effects:
[0058] The speech synthesis method described in the embodiment of the present application is to obtain the audio data of the target object; obtain the target text; convert the target text into a phoneme sequence; use the speech feature extraction technology to extract the audio features of the audio data, and preliminarily determine the audio frame length information of the target object when emitting different phonemes based on the audio feature extraction results; input the phoneme sequence into a preset fine-tuning processing model, adjust the sequence frame length information of the phoneme sequence through the fine-tuning processing model, and obtain the fine-tuned phoneme sequence; perform speech synthesis according to the fine-tuned phoneme sequence. The target text is converted into a phoneme sequence in the above manner, and its purpose is to ensure the initial accuracy of the phoneme sequence, minimize subsequent fine-tuning operations, and save computer processing resources. The audio frame length information of the target object when emitting different phonemes is deployed in advance as fine-tuning reference information to the fine-tuning processing model, and the phoneme sequence is fine-tuned, so as to ensure that only a small amount of audio data of the target object is required to complete high-quality speaker customization, and the converted speech is closer to the real speech. The method is applied to the financial customer service industry, and can provide customers with better quality and good sound quality service functions. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0060] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0061] Figure 2 is a flow chart of an embodiment of a speech synthesis method according to the present application;
[0062] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 203 is shown;
[0063] Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 204 is shown;
[0064] Figure 5 is a structural schematic diagram of an embodiment of a speech synthesis device according to the present application;
[0065] Figure 6 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0067] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0068] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0069] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0070] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0071] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.
[0072] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .
[0073] It should be noted that the speech synthesis method provided in the embodiments of the present application is generally executed by a terminal device, and accordingly, the speech synthesis device is generally arranged in the terminal device.
[0074] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0075] Continue to refer Figure 2 , shows a flow chart of an embodiment of a speech synthesis method according to the present application. The speech synthesis method comprises the following steps:
[0076] Step 201: Acquire audio data of a target object, wherein the target object includes a real speaker.
[0077] In this embodiment, the real speaker is, for example, a financial customer service operator or a bank voice customer service staff. The speech synthesis method described in this application only requires a small amount of audio data of the target object to complete high-quality personalized speaker customization.
[0078] Step 202: Acquire target text, wherein the target text includes text to be speech synthesized.
[0079] In this embodiment, the speech text to be synthesized refers to a text content to be converted into speech, for example, in the field of financial technology, the pre-set speech text content when the intelligent customer service answers the customer's questions. Through the speech synthesis method provided by this application, the text content can be converted into synthesized speech. The speech synthesis method can also be used in teaching auxiliary scenarios such as intelligent recitation, intelligent storytelling, and intelligent reading, such as audio books. Applying this speech synthesis method to the financial customer service industry can provide customers with better quality and better sound quality service functions.
[0080] Step 203: convert the target text into a phoneme sequence.
[0081] Continue to refer Figure 3 , Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 203 shown includes:
[0082] Step 301, extracting the syntactic features of each sentence in the target text and the word features of all words in each sentence;
[0083] Specifically, the extraction of syntactic features of each sentence in the target text and word features of all words in each sentence can be achieved by a preset grammatical feature extraction component, which includes a syntactic feature analysis subcomponent and a word feature analysis subcomponent, wherein the syntactic feature analysis subcomponent is used to perform sentence pronunciation annotation and sentence tone annotation on each word in the split sentences, and obtain the tone change information corresponding to the pronunciation of the words in each sentence in the entire sentence according to the annotation results and a preset word pronunciation dictionary, and the word feature analysis subcomponent is used to obtain the sentence pronunciation annotation and sentence tone annotation corresponding to the words respectively contained in each sentence as the pronunciation information of each word.
[0084] In this embodiment, the step of extracting the syntactic features of each sentence in the target text and the word features of all words in each sentence through a preset grammatical feature extraction component specifically includes: performing sentence segmentation on the target text to obtain the split sentences; and performing word segmentation on the split sentences to obtain the words respectively contained in each sentence; performing sentence pronunciation annotation and sentence tone annotation on each character in the split sentences respectively, and obtaining the tone change information corresponding to the pronunciation of the character in each sentence in the pronunciation of the entire sentence as the syntactic feature according to the annotation results and a preset word pronunciation dictionary, wherein the word pronunciation dictionary annotates the initial pronunciation information and initial tone information of each character when it does not constitute a word and a sentence; based on the annotation results, obtaining the sentence pronunciation annotation and sentence tone annotation corresponding to the words respectively contained in each sentence as the pronunciation information of each word, and using the pronunciation information as the word feature.
[0085] Specifically, the sentence segmentation of the target text to obtain the segmented sentences can be performed according to the punctuation marks in the target text to obtain the segmented sentences; the segmentation of the segmented sentences to obtain the words contained in each sentence can be performed according to a preset segmentation dictionary to obtain the words contained in each sentence; the grammatical feature extraction component is a grammatical feature extraction component that has been pre-trained based on large-scale natural language corpus, the syntactic feature analysis subcomponent can identify the pronunciation of the characters in each sentence and the tone change information of the pronunciation of the entire sentence, and the word feature analysis subcomponent can identify the pronunciation information of each word. Using the grammatical feature extraction component ensures the initial accuracy of the converted phoneme sequence, minimizes subsequent fine-tuning operations, and saves computer processing resources.
[0086] Step 302, determining the pronunciation phoneme corresponding to each word in the target text according to the syntactic features and the word features;
[0087] In this embodiment, the step of determining the pronunciation phonemes corresponding to each character in the target text according to the syntactic features and the word features specifically includes: identifying whether the current character is a polyphone according to the character pronunciation dictionary; and identifying whether the current character involves stress conversion in the corresponding entire sentence pronunciation according to the syntactic features and the word features; if the current character is a non-polyphone and does not involve stress conversion in the corresponding entire sentence pronunciation, the unique pronunciation of the current character is used as the pronunciation phoneme; if the current character is a non-polyphone but involves stress conversion in the corresponding entire sentence pronunciation If the current word is a polyphone, and the pronunciation of the current word does not involve stress conversion in the corresponding whole sentence pronunciation, the pronunciation of the current word in the corresponding word is identified according to the word feature, and the pronunciation is used as the pronunciation phoneme of the current word; if the current word is a polyphone, but the pronunciation of the corresponding whole sentence involves stress conversion, the pronunciation of the current word in the corresponding word is identified according to the word feature, and the stress conversion of the pronunciation is identified according to the syntactic feature, and the stress conversion result of the pronunciation is used as the pronunciation phoneme.
[0088] Specifically, the above steps are adopted to determine the pronunciation phonemes corresponding to each word in the target text, ensure the initial accuracy of the phoneme sequence, minimize subsequent fine-tuning operations, and save computer processing resources.
[0089] Step 303: convert the target text into a phoneme sequence according to the pronunciation phonemes.
[0090] In this embodiment, the step of converting the target text into a phoneme sequence according to the pronunciation phonemes specifically includes: obtaining the sequence position information of each character in the target text; according to the sequence position information, serializing and sorting the pronunciation phonemes corresponding to each character in the target text, and obtaining the sorting result as the phoneme sequence.
[0091] Specifically, by obtaining the sequence position information of each character in the target text, the pronunciation phonemes corresponding to each character in the target text are serialized and sorted according to the sequence position information, and the sorting result is obtained as the phoneme sequence, thereby ensuring the consistency of the phoneme sequence with the character sequence in the target text.
[0092] In general, step 301 to step 303 are combined with a preset grammatical feature extraction component to extract the syntactic features of each sentence in the target text and the word features of all words in each sentence. The purpose is to identify the tone change information corresponding to the pronunciation of the characters in each sentence in the pronunciation of the entire sentence, and identify whether there is a stress conversion in the character pronunciation according to the tone change information. For example, the light tone in Chinese refers to the fact that some syllables in words and sentences lose their original tones due to the influence of the previous and next syllables, thus becoming a soft and light tone, such as: the pronunciation of dad is bàba, the pronunciation of dim sum is d iǎn xin, etc. In such words, the tone of the second syllable becomes soft and fuzzy in the actual pronunciation, which belongs to the conversion from stress to light tone. When there are multiple pronunciations of a character, the final pronunciation of the character is determined in combination with the word. For example, strong has the pronunciations q iáng, q iǎng and j iàng. Assuming the word is stubborn, it is obvious that the pronunciation of strong in stubborn is j iàng. The target text is converted into a phoneme sequence in the above manner, the purpose of which is to ensure the initial accuracy of the phoneme sequence, minimize subsequent fine-tuning operations, and save computer processing resources.
[0093] Step 204: extract audio features from the audio data using speech feature extraction technology, and preliminarily determine the audio frame length information occupied by the target object when uttering different phonemes based on the audio feature extraction result.
[0094] In this embodiment, the speech feature extraction technology specifically includes audio data preprocessing technology, such as framing, noise reduction, etc., and also includes technology for extracting audio features using MFCC coefficient method or LPC coefficient method, as well as mapping relationship recognition technology between audio features and phonemes.
[0095] Continue to refer Figure 4 , Figure 4 yes Figure 2 The flowchart of a specific embodiment of step 204 shown includes:
[0096] Step 401, performing frame processing on the audio data to obtain continuous audio frame segments;
[0097] Step 402, extracting audio features from the continuous audio frame segments using an MFCC coefficient method or an LPC coefficient method;
[0098] Step 403, inputting the audio features into a preset acoustic feature recognition model to identify the pronunciation phonemes corresponding to each frame of audio, wherein the acoustic feature recognition model includes an acoustic feature recognition model based on DNN-HMM, DNN (Deep Neural Network), HMM (Hidden Markov Model);
[0099] Step 404 , by counting and sorting the pronunciation phonemes corresponding to each frame of audio, preliminarily determining the audio frame length information occupied by the target object when uttering different phonemes.
[0100] Specifically, the audio data is framed according to a preset frame length and displacement value, and the audio data is divided into a plurality of continuous audio frame segments, the purpose of which is to convert a continuous audio signal into a discrete signal that can be processed by a computer; then, audio features are extracted from each frame of audio, and the specific extraction method can use Mel-frequency cepstral coefficients (MFCC), linear predictive coding (LPC) coefficients, etc., which can capture important information in the audio signal for subsequent phoneme recognition; then, since the acoustic feature recognition model can calculate the probability that the audio feature corresponds to each phoneme under given audio features, the acoustic feature recognition model is used to identify the features of each frame of audio to obtain the most likely pronunciation phoneme of each frame of audio; since a phoneme may span multiple audio frames, that is, multiple consecutive audio frames may correspond to the same pronunciation phoneme, finally, by counting and sorting the pronunciation phonemes corresponding to each frame of audio, the audio frame length information occupied by the target object when emitting different phonemes is preliminarily determined.
[0101] Step 205: input the phoneme sequence into a preset fine-tuning processing model, and adjust the sequence frame length information of the phoneme sequence based on preset reference information to obtain a fine-tuned phoneme sequence, wherein the preset reference information is the audio frame length information occupied by the target object when emitting different phonemes.
[0102] Specifically, before executing the step of inputting the phoneme sequence into a preset fine-tuning processing model, the method also includes: pre-deploying the audio frame length information occupied by the target object when emitting different phonemes as fine-tuning reference information into the fine-tuning processing model, and then inputting the phoneme sequence into the preset fine-tuning processing model, adjusting the sequence frame length information of the phoneme sequence through the fine-tuning processing model, and obtaining a fine-tuned phoneme sequence.
[0103] Specifically, the fine-tuning processing model is composed of a phoneme encoder, a LoRA fine-tuning layer and an Adapter fine-tuning layer. The phoneme encoder adopts an encoder in a transformer structure, and the encoder in the transformer structure is composed of an embedding layer and a plurality of identical encoding sublayers stacked together, wherein each encoding sublayer is respectively composed of a multi-head self-attention layer, a feedforward neural network layer, a residual connection layer, an output layer and a normalization layer. When constructing the fine-tuning processing model, a LoRA fine-tuning layer is added to the Query matrix and the Key matrix of the multi-head attention layer respectively. Finally, the new multi-head attention layer contains "Query matrix + LoRA fine-tuning layer + Key matrix + LoRA fine-tuning layer + Value matrix". After that, an Adapter fine-tuning layer is added between the last output layer and the normalization layer of the phoneme encoder to jointly constitute the fine-tuning processing model. The Adapter fine-tuning layer is stacked by an FFT downsampling layer, a nonlinear layer and an FFT upsampling layer, and the LoRA fine-tuning layer is composed of an FFT downsampling layer and an FFT upsampling layer.
[0104] In this embodiment, the step of inputting the phoneme sequence into a preset fine-tuning processing model, adjusting the sequence frame length information of the phoneme sequence through the fine-tuning processing model, and obtaining a fine-tuned phoneme sequence specifically includes: using the embedding layer of the phoneme encoder to convert the phoneme sequence into a vector representation, and adding position coding information for each phoneme in the phoneme sequence, wherein the phoneme encoder includes an encoder of a transformer structure; inputting the vector representation and the position coding information corresponding to each phoneme into the encoding sublayer to which the LoRA fine-tuning layer is added, and outputting the phoneme sequence fine-tuned by the LoRA fine-tuning layer after encoding processing by the encoding sublayer; inputting the phoneme sequence fine-tuned by the LoRA fine-tuning layer into the Adapter fine-tuning layer, continuing fine-tuning, and obtaining a fine-tuned phoneme sequence, wherein the LoRA fine-tuning layer and the Adapter fine-tuning layer are used to adjust the frame lengths of different phonemes in the phoneme sequence according to the audio features of the target object and the phoneme mapping relationship.
[0105] Step 206: Perform speech synthesis according to the fine-tuned phoneme sequence.
[0106] In this embodiment, the step of performing speech synthesis according to the fine-tuned phoneme sequence specifically includes: mapping the fine-tuned phoneme sequence into frame-by-frame audio features; inputting the frame-by-frame audio features into a decoder of a transformer structure corresponding to the phoneme encoder, and obtaining a waveform audio signal after decoding; and sending the waveform audio signal as a speech synthesis result to a preset speech output terminal.
[0107] The present application obtains the audio data of the target object; obtains the target text; converts the target text into a phoneme sequence; uses speech feature extraction technology to extract audio features from the audio data, and preliminarily determines the audio frame length information occupied by the target object when emitting different phonemes based on the audio feature extraction results; inputs the phoneme sequence into a preset fine-tuning processing model, adjusts the sequence frame length information of the phoneme sequence through the fine-tuning processing model, and obtains a fine-tuned phoneme sequence; and performs speech synthesis according to the fine-tuned phoneme sequence. The target text is converted into a phoneme sequence in the above manner, and its purpose is to ensure the initial accuracy of the phoneme sequence, minimize subsequent fine-tuning operations, and save computer processing resources. The audio frame length information occupied by the target object when emitting different phonemes is deployed in advance as fine-tuning reference information to the fine-tuning processing model, and the phoneme sequence is fine-tuned, so as to ensure that only a small amount of audio data of the target object is required to complete high-quality speaker customization, and the converted voice is closer to the real voice. The present method is applied to the financial customer service industry, and can provide customers with better quality and good sound quality service functions.
[0108] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0109] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0110] In the embodiment of the present application, the audio data of the target object is obtained; the target text is obtained; the target text is converted into a phoneme sequence; the audio feature extraction technology is used to extract the audio feature of the audio data, and the audio frame length information of the target object when emitting different phonemes is preliminarily determined based on the audio feature extraction result; the phoneme sequence is input into a preset fine-tuning processing model, and the sequence frame length information of the phoneme sequence is adjusted by the fine-tuning processing model to obtain a fine-tuned phoneme sequence; speech synthesis is performed according to the fine-tuned phoneme sequence. The target text is converted into a phoneme sequence in the above manner, and its purpose is to ensure the initial accuracy of the phoneme sequence, minimize subsequent fine-tuning operations, and save computer processing resources. The audio frame length information of the target object when emitting different phonemes is deployed in advance as fine-tuning reference information to the fine-tuning processing model, and the phoneme sequence is fine-tuned, so as to ensure that only a small amount of audio data of the target object is required to complete high-quality speaker customization, and the converted voice is closer to the real voice. The method is applied to the financial customer service industry, and can provide customers with better quality and good sound quality service functions.
[0111] Further references Figure 5 , as a response to the above Figure 2 The present application provides an embodiment of a speech synthesis device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0112] like Figure 5 As shown, the speech synthesis device 500 described in this embodiment includes: an audio data acquisition module 501, a target text acquisition module 502, a phoneme sequence conversion module 503, an audio feature extraction module 504, a phoneme sequence fine-tuning module 505 and a speech synthesis module 506. Among them:
[0113] The audio data acquisition module 501 is used to acquire the audio data of the target object, wherein the target object includes the real speaker;
[0114] A target text acquisition module 502 is used to acquire a target text, wherein the target text includes a text to be speech synthesized;
[0115] A phoneme sequence conversion module 503, used for converting the target text into a phoneme sequence;
[0116] The audio feature extraction module 504 is used to extract audio features from the audio data using a speech feature extraction technique, and preliminarily determine the audio frame length information occupied by the target object when emitting different phonemes based on the audio feature extraction result;
[0117] The phoneme sequence fine-tuning module 505 is used to input the phoneme sequence into a preset fine-tuning processing model, and adjust the sequence frame length information of the phoneme sequence based on preset reference information to obtain a fine-tuned phoneme sequence, wherein the preset reference information is the audio frame length information respectively occupied by the target object when uttering different phonemes;
[0118] The speech synthesis module 506 is used to perform speech synthesis according to the fine-tuned phoneme sequence.
[0119] The present application obtains the audio data of the target object; obtains the target text; converts the target text into a phoneme sequence; uses speech feature extraction technology to extract audio features from the audio data, and preliminarily determines the audio frame length information occupied by the target object when emitting different phonemes based on the audio feature extraction results; inputs the phoneme sequence into a preset fine-tuning processing model, adjusts the sequence frame length information of the phoneme sequence through the fine-tuning processing model, and obtains a fine-tuned phoneme sequence; and performs speech synthesis according to the fine-tuned phoneme sequence. The target text is converted into a phoneme sequence in the above manner, and its purpose is to ensure the initial accuracy of the phoneme sequence, minimize subsequent fine-tuning operations, and save computer processing resources. The audio frame length information occupied by the target object when emitting different phonemes is deployed in advance as fine-tuning reference information to the fine-tuning processing model, and the phoneme sequence is fine-tuned, so as to ensure that only a small amount of audio data of the target object is required to complete high-quality speaker customization, and the converted voice is closer to the real voice. The present method is applied to the financial customer service industry, and can provide customers with better quality and good sound quality service functions.
[0120] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0121] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0122] To solve the above technical problems, the present application also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.
[0123] The computer device 6 includes a memory 6a, a processor 6b, and a network interface 6c that are interconnected through a system bus. It should be noted that the figure only shows a computer device 6 having components 6a-6c, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.
[0124] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0125] The memory 6a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 6a can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 6a can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (Smart Med i aCard, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 6a can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 6a is generally used to store the operating system and various application software installed on the computer device 6, such as a computer-readable instruction of a speech synthesis method, etc. In addition, the memory 6a can also be used to temporarily store various types of data that have been output or are to be output.
[0126] The processor 6b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 6b is generally used to control the overall operation of the computer device 6. In the present embodiment, the processor 6b is used to run the computer-readable instructions or process data stored in the memory 6a, such as the computer-readable instructions for running the speech synthesis method.
[0127] The network interface 6c may include a wireless network interface or a wired network interface. The network interface 6c is generally used to establish a communication connection between the computer device 6 and other electronic devices.
[0128] The computer device proposed in this embodiment belongs to the field of speech synthesis technology and is applied to text-to-speech scenarios. This application obtains audio data of a target object; obtains a target text; converts the target text into a phoneme sequence; uses speech feature extraction technology to extract audio features from the audio data, and preliminarily determines the audio frame length information occupied by the target object when emitting different phonemes based on the audio feature extraction results; inputs the phoneme sequence into a preset fine-tuning processing model, adjusts the sequence frame length information of the phoneme sequence through the fine-tuning processing model, and obtains a fine-tuned phoneme sequence; and performs speech synthesis according to the fine-tuned phoneme sequence. The target text is converted into a phoneme sequence in the above manner, the purpose of which is to ensure the initial accuracy of the phoneme sequence, minimize subsequent fine-tuning operations, save computer processing resources, and pre-deploy the audio frame length information occupied by the target object when emitting different phonemes as fine-tuning reference information into the fine-tuning processing model to perform phoneme sequence fine-tuning, thereby ensuring that high-quality speaker customization can be completed with only a small amount of audio data of the target object, and the converted voice is closer to the real voice. This method is applied to the financial customer service industry and can provide customers with higher-quality and good sound quality service functions.
[0129] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the speech synthesis method as described above.
[0130] The computer-readable storage medium proposed in this embodiment belongs to the field of speech synthesis technology and is applied to text-to-speech scenarios. This application obtains audio data of a target object; obtains a target text; converts the target text into a phoneme sequence; uses speech feature extraction technology to extract audio features from the audio data, and preliminarily determines the audio frame length information occupied by the target object when emitting different phonemes based on the audio feature extraction results; inputs the phoneme sequence into a preset fine-tuning processing model, adjusts the sequence frame length information of the phoneme sequence through the fine-tuning processing model, and obtains a fine-tuned phoneme sequence; and performs speech synthesis based on the fine-tuned phoneme sequence. The target text is converted into a phoneme sequence in the above manner, the purpose of which is to ensure the initial accuracy of the phoneme sequence, minimize subsequent fine-tuning operations, save computer processing resources, and pre-deploy the audio frame length information occupied by the target object when emitting different phonemes as fine-tuning reference information into the fine-tuning processing model to perform phoneme sequence fine-tuning, thereby ensuring that high-quality speaker customization can be completed with only a small amount of audio data of the target object, and the converted voice is closer to the real voice. This method is applied to the financial customer service industry and can provide customers with higher-quality and good sound quality service functions.
[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0132] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A speech synthesis method, characterized in that: The steps include: Acquire audio data of a target object, wherein the target object includes a real speaker; Acquire a target text, wherein the target text includes a text to be speech synthesized; Converting the target text into a phoneme sequence; Using speech feature extraction technology to extract audio features from the audio data, and preliminarily determining the audio frame length information occupied by the target object when uttering different phonemes based on the audio feature extraction results; Inputting the phoneme sequence into a preset fine-tuning processing model, adjusting the sequence frame length information of the phoneme sequence based on preset reference information, and obtaining a fine-tuned phoneme sequence, wherein the preset reference information is the audio frame length information respectively occupied by the target object when uttering different phonemes; Speech synthesis is performed according to the phoneme sequence completed by the fine-tuning.
2. The speech synthesis method according to claim 1, characterized in that: The step of converting the target text into a phoneme sequence specifically comprises: Extracting syntactic features of each sentence in the target text and word features of all words in each sentence; Determining the pronunciation phoneme corresponding to each word in the target text according to the syntactic features and the word features; The target text is converted into a phoneme sequence according to the pronunciation phonemes.
3. The speech synthesis method according to claim 2, characterized in that: The step of extracting the syntactic features of each sentence in the target text and the word features of all words in each sentence specifically includes: Perform sentence splitting on the target text to obtain split sentences; and, Perform word segmentation on the split sentences to obtain the words contained in each sentence; Performing sentence pronunciation and sentence tone marking for each character in the split sentences respectively, and obtaining tone change information corresponding to the pronunciation of each character in the sentence in the entire sentence pronunciation as the syntactic feature according to the marking results and a preset character pronunciation dictionary, wherein the character pronunciation dictionary marks the initial pronunciation information and initial tone information of each character before forming a word or sentence; Based on the annotation results, obtaining the sentence pronunciation annotation and sentence tone annotation corresponding to the words respectively contained in each sentence as the pronunciation information of each word, and using the pronunciation information as the word feature; The step of converting the target text into a phoneme sequence according to the pronunciation phonemes specifically comprises: Obtaining sequence position information of each word in the target text; According to the sequence position information, the pronunciation phonemes corresponding to each word in the target text are serialized and sorted to obtain the sorting result as the phoneme sequence.
4. The speech synthesis method according to claim 2, characterized in that: The step of determining the pronunciation phoneme corresponding to each word in the target text according to the syntactic features and the word features specifically includes: According to the character pronunciation dictionary, identifying whether the current character is a polyphonic character; and Identify whether the current word involves stress conversion in the corresponding whole sentence pronunciation according to the syntactic features and the word features; If the current character is a non-polyphonetic character and the pronunciation of the corresponding entire sentence does not involve stress conversion, the unique pronunciation of the current character is used as the pronunciation phoneme; If the current character is a non-polyphonetic character but the pronunciation of the corresponding entire sentence involves stress conversion, the pronunciation of the current character after stress conversion is used as the pronunciation phoneme; If the current character is a polyphonic character and the pronunciation of the corresponding entire sentence does not involve stress conversion, identifying the pronunciation of the current character in the corresponding word according to the word feature, and using the pronunciation as the pronunciation phoneme of the current character; If the current character is a polyphone but the corresponding pronunciation of the entire sentence involves stress conversion, the pronunciation of the current character in the corresponding word is identified according to the word features, and the stress conversion of the pronunciation is identified according to the syntactic features, and the stress conversion result of the pronunciation is used as the pronunciation phoneme.
5. The speech synthesis method according to claim 2 or 1, characterized in that: The step of extracting audio features from the audio data using a speech feature extraction technology and preliminarily determining the audio frame length information of different phonemes respectively occupied by the target object when emitting different phonemes based on the audio feature extraction results specifically includes: Performing frame processing on the audio data to obtain continuous audio frame segments; Extracting audio features from the continuous audio frame segments using an MFCC coefficient method or an LPC coefficient method; Inputting the audio features into a preset acoustic feature recognition model to identify the pronunciation phonemes corresponding to each frame of audio; By counting and sorting the pronunciation phonemes corresponding to each frame of audio, the audio frame length information occupied by the target object when uttering different phonemes is preliminarily determined.
6. The speech synthesis method according to claim 2, characterized in that: The fine-tuning processing model is composed of a phoneme encoder, a LoRA fine-tuning layer and an Adapter fine-tuning layer. The step of inputting the phoneme sequence into a preset fine-tuning processing model, adjusting the sequence frame length information of the phoneme sequence through the fine-tuning processing model, and obtaining a fine-tuned phoneme sequence specifically includes: Converting the phoneme sequence into a vector representation through an embedding layer of a phoneme encoder, and adding position encoding information to each phoneme in the phoneme sequence, wherein the phoneme encoder includes an encoder of a transformer structure; Input the vector representation and the position encoding information corresponding to each phoneme into the coding sublayer to which the LoRA fine-tuning layer is added, and after encoding processing by the coding sublayer, output the phoneme sequence fine-tuned by the LoRA fine-tuning layer; The phoneme sequence fine-tuned by the LoRA fine-tuning layer is input into the Adapter fine-tuning layer, and fine-tuning is continued to obtain a fine-tuned phoneme sequence, wherein the LoRA fine-tuning layer and the Adapter fine-tuning layer are used to adjust the frame lengths of different phonemes in the phoneme sequence according to the audio features of the target object and the phoneme mapping relationship.
7. The speech synthesis method according to claim 6, characterized in that: The step of performing speech synthesis according to the fine-tuned phoneme sequence specifically includes: Mapping the fine-tuned phoneme sequence into frame-by-frame audio features; Inputting the frame-by-frame audio features into a decoder of a transformer structure corresponding to the phoneme encoder, and obtaining a waveform audio signal through decoding processing; The waveform audio signal is used as a speech synthesis result and sent to a preset speech output terminal.
8. A speech synthesis device, characterized in that: include: An audio data acquisition module, used to acquire audio data of a target object, wherein the target object includes a real speaker; A target text acquisition module, used to acquire a target text, wherein the target text includes a text to be speech synthesized; A phoneme sequence conversion module, used for converting the target text into a phoneme sequence; An audio feature extraction module is used to extract audio features from the audio data using a speech feature extraction technique, and preliminarily determine the audio frame length information occupied by the target object when emitting different phonemes based on the audio feature extraction result; A phoneme sequence fine-tuning module, used to input the phoneme sequence into a preset fine-tuning processing model, and adjust the sequence frame length information of the phoneme sequence based on preset reference information to obtain a fine-tuned phoneme sequence, wherein the preset reference information is the audio frame length information respectively occupied by the target object when uttering different phonemes; The speech synthesis module is used to perform speech synthesis according to the phoneme sequence completed by the fine-tuning.
9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech synthesis method according to any one of claims 1 to 7 when executing the computer-readable instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech synthesis method according to any one of claims 1 to 7 are implemented.
