Prosody-based speech synthesis method, device, equipment, and medium

Abstract

The embodiments of the present application provide a speech synthesis method, apparatus, device, and medium based on rhythm, which belongs to the field of artificial intelligence technology. The method includes: obtaining the original spectrum of the original audio; preprocessing the original spectrum to obtain the target spectrum of the original audio; inputting the target spectrum, language feature vector, and speaker feature vector into a preset rhythm encoder to obtain a reference rhythm vector; inputting the language feature vector and speaker feature vector into a preset rhythm prediction model to obtain a predicted rhythm vector; obtaining a potential rhythm vector based on the reference rhythm vector and the predicted rhythm vector; obtaining the target synthesized speech based on the language feature vector to be synthesized, the speaker feature vector to be synthesized, and the potential rhythm vector. The predicted rhythm vector obtained by the rhythm prediction model only contains rhythm information that is irrelevant to the speaker and the text content, thereby improving the generation effect of the potential rhythm vector and further improving the effect of speech synthesis.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a prosody-based speech synthesis method and apparatus, device, and medium. Background Art

[0002] Currently, many audio novel platforms can accurately read article content or accurately answer user inquiries. Generally, synthesized speech is currently obtained based on text and rhythm, so the extraction of rhythm will greatly affect the effect of the synthesized speech. For example, the three attributes of audio pitch, duration, and energy are predicted and modeled separately to control rhythm. However, the synthesized speech obtained based on such rhythm is very mechanical and lacks emotion. It is difficult to achieve the effect that a human voice should achieve, and the user experience is mediocre. This undoubtedly greatly reduces the ultimate effect of audio reading, making it difficult for many people to accept and lose some users. Therefore, how to provide a rhythm-based speech synthesis method that can improve the effect of speech synthesis has become a technical problem that needs to be solved urgently. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a prosody-based speech synthesis method and apparatus, device, and medium that can improve the effect of speech synthesis.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application provides a prosody-based speech synthesis method, the method comprising:

[0005] Get the original spectrum of the original audio;

[0006] Preprocessing the original frequency spectrum to obtain a target frequency spectrum of the original audio;

[0007] Acquire a language feature vector and a speaker feature vector; wherein the language feature vector includes phoneme information;

[0008] Inputting the target spectrum, the language feature vector, and the speaker feature vector into a preset prosody encoder to obtain a reference prosody vector;

[0009] Inputting the language feature vector and the speaker feature vector into a preset prosody prediction model to obtain a predicted prosody vector;

[0010] Obtaining a language feature vector to be synthesized and a speaker feature vector to be synthesized;

[0011] Obtaining a potential prosody vector according to the reference prosody vector and the predicted prosody vector;

[0012] A target synthesized speech is obtained according to the feature vector of the language to be synthesized, the feature vector of the speaker to be synthesized, and the latent prosody vector.

[0013] In some embodiments, preprocessing the original spectrum to obtain a target spectrum of the original audio includes:

[0014] The original spectrum is filtered according to a preset frequency threshold to obtain a target spectrum of the original audio; wherein the frequency of the target spectrum is less than the frequency threshold.

[0015] In some embodiments, the language feature vector is obtained by the following steps, including:

[0016] Acquire a text sequence, wherein the text sequence includes a plurality of words;

[0017] Converting the text sequence into a phoneme sequence and a word sequence;

[0018] Concatenating the phoneme sequence and the word sequence to obtain a joint sequence of each word;

[0019] Each of the combined sequences is encoded by a preset text encoder to obtain the language feature vector.

[0020] In some embodiments, obtaining a potential prosody vector according to the reference prosody vector and the predicted prosody vector includes:

[0021] Calculate a first loss function according to the reference prosody vector and the predicted prosody vector;

[0022] Adjusting parameters of the rhythm prediction model according to the first loss function to obtain a rhythm generation model;

[0023] The language feature vector and the speaker feature vector are input into the prosody generation model to obtain a latent prosody vector.

[0024] In some embodiments, after adjusting parameters of the prosody prediction model according to the first loss function to obtain a prosody generation model, the method further includes:

[0025] Inputting the language feature vector, the speaker feature vector and the reference prosody vector into a preset decoder module to obtain a reference spectrum;

[0026] Inputting the language feature vector, the speaker feature vector and the predicted prosody vector into the decoder module to obtain a predicted spectrum;

[0027] Calculating a second loss function according to the reference spectrum and the predicted spectrum;

[0028] Parameters of the rhythm generation model are adjusted according to the second loss function to update the rhythm generation model.

[0029] In some embodiments, inputting the language feature vector, the speaker feature vector, and the reference prosody vector into a preset decoder module to obtain a reference spectrum includes:

[0030] Adding the language feature vector, the speaker feature vector, and the reference prosody vector to obtain a preliminary vector corresponding to each word;

[0031] The preliminary vector is copied according to a preset spectrum duration threshold to obtain a target vector corresponding to each of the words; wherein the spectrum duration of each of the target vectors is equal to the spectrum duration threshold;

[0032] The plurality of target vectors are decoded by the decoder module to obtain a reference spectrum.

[0033] In some embodiments, the prosody prediction model includes an attention mechanism module, a context encoder module, and a convolution module; inputting the language feature vector and the speaker feature vector into a preset prosody prediction model to obtain a predicted prosody vector includes:

[0034] Performing weight calculation on the speaker feature vector by the attention mechanism module to obtain a target speaker feature vector;

[0035] Encoding the language feature vector by the context encoder module to obtain a target language feature vector;

[0036] The target speaker feature vector and the target language feature vector are jointly encoded through a convolution module to obtain a predicted prosody vector.

[0037] To achieve the above-mentioned object, a second aspect of an embodiment of the present application provides a prosody-based speech synthesis device, the device comprising:

[0038] The original spectrum acquisition module is used to obtain the original spectrum of the original audio;

[0039] A preprocessing module, configured to preprocess the original spectrum to obtain a target spectrum of the original audio;

[0040] An original vector acquisition module, configured to acquire a language feature vector and a speaker feature vector; wherein the language feature vector includes phoneme information;

[0041] a reference prosody determination module, configured to input the target spectrum, the language feature vector, and the speaker feature vector into a preset prosody encoder to obtain a reference prosody vector;

[0042] A predicted prosody determination module, configured to input the language feature vector and the speaker feature vector into a preset prosody prediction model to obtain a predicted prosody vector;

[0043] A vector acquisition module to be synthesized, used to obtain a language feature vector to be synthesized and a speaker feature vector to be synthesized;

[0044] a latent prosody determination module, configured to obtain a latent prosody vector according to the reference prosody vector and the predicted prosody vector;

[0045] The speech synthesis module is used to obtain the target synthesized speech according to the feature vector of the language to be synthesized, the feature vector of the speaker to be synthesized and the latent prosody vector.

[0046] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer device, which includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the method described in the first aspect above is implemented.

[0047] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the first aspect above.

[0048] The prosody-based speech synthesis method, apparatus, device, and medium proposed in the present application introduce a prosody prediction model to obtain a predicted prosody vector. In the process of obtaining the latent prosody vector, the input of the predicted prosody model is the language feature vector of the text and the speaker feature vector. The latent prosody vector is obtained by combining the output of the predicted prosody model with the output of the prosody encoder, so that the output of the predicted prosody model can match the output of the prosody encoder, that is, a consistent prosody vector is obtained. It can be understood that since the speaker feature vector and the language feature vector are independent of the original audio, the predicted prosody vector obtained by the prosody generation model only contains prosody information that is independent of the speaker and the text content. In summary, the embodiments of the present application can improve the prosody generation effect and further improve the speech synthesis effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of a prosody-based speech synthesis method provided in an embodiment of the present application;

[0050] Figure 2 yes Figure 1 Flowchart of step S103 in FIG.

[0051] Figure 3 yes Figure 1 Flowchart of step S105 in FIG.

[0052] Figure 4 is another flow chart of the prosody-based speech synthesis method provided in an embodiment of the present application;

[0053] Figure 5 is another flow chart of the prosody-based speech synthesis method provided in an embodiment of the present application;

[0054] Figure 6 yes Figure 5 Flowchart of step S501 in FIG.

[0055] Figure 7 This is a block diagram of the module structure of the prosody-based speech synthesis device provided in an embodiment of the present application;

[0056] Figure 8 This is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0058] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0060] First, let’s analyze some of the terms used in this application:

[0061] Artificial Intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses theories, methods, technologies, and application systems that use digital computers or digital computer-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0062] Natural Language Processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information image processing, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0063] Information Extraction (NER): A text processing technology that extracts specified types of entity, relationship, event, and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data is all text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.

[0064] Automatic Speech Recognition (ASR): Also known as automatic speech recognition, speech recognition is an interdisciplinary subfield of computational linguistics that develops methods and techniques that enable computers to recognize and translate spoken language. It integrates knowledge and research from the fields of linguistics, computer science, and electrical engineering.

[0065] Text to Speech (TTS) is a technology that converts text into speech, giving machines the ability to speak freely like humans. It is an important part of human-computer voice interaction.

[0066] Speech synthesis systems are widely used in various scenarios in our lives, including voice dialogue systems; intelligent voice assistants such as Siri and iFlytek VoicePoint; telephone information query systems; auxiliary applications such as in-car navigation and audio e-books; language learning; real-time information broadcasting systems in airports, train stations, etc.; information acquisition and communication for people with visual or speech impairments, etc.

[0067] Currently, many audio novel platforms can accurately read the content of articles or accurately answer users' inquiries. Generally, it is necessary to obtain synthesized speech based on text and rhythm, so the extraction of rhythm will greatly affect the effect of the synthesized speech. For example, the three attributes of audio pitch, duration, and energy are predicted and modeled separately to control the rhythm. However, the synthesized speech obtained by this rhythm is very mechanical and lacks emotion. It is difficult to achieve the effect that the ups and downs of human voice should achieve, and the user experience is mediocre. This undoubtedly greatly reduces the final effect of audio reading, making it difficult for many people to accept and lose some users. Therefore, how to provide a rhythm-based speech synthesis method that can improve the effect of speech synthesis has become a technical problem that needs to be solved urgently.

[0068] Based on this, the main purpose of the embodiments of the present application is to propose a speech synthesis method and apparatus, device, and medium based on rhythm, aiming to introduce a rhythm prediction model to obtain a predicted rhythm vector. During the training process, the input of the predicted rhythm model is the language feature vector and the speaker feature vector of the text. The potential rhythm vector is obtained by combining the output result of the predicted rhythm model with the output of the rhythm encoder, so that the output of the predicted rhythm model can match the output of the rhythm encoder, that is, a consistent rhythm vector is obtained. It can be understood that since the speaker feature vector and the language feature vector are independent of the original audio, the predicted rhythm vector obtained by the rhythm generation model only contains rhythm information that is independent of the speaker and the text content. In summary, the embodiments of the present application can improve the rhythm generation effect and further improve the speech synthesis effect.

[0069] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0070] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0071] The rhythm-based speech synthesis method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The rhythm-based speech synthesis method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements a rhythm-based speech synthesis method, etc., but is not limited to the above forms.

[0072] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0073] The embodiments of the present application provide a prosody-based speech synthesis method, a prosody-based speech synthesis device, a computer device, and a storage medium, which are specifically illustrated by the following embodiments. First, the prosody-based speech synthesis method in the embodiments of the present application is described.

[0074] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0075] Figure 1 This is an optional flowchart of the prosody-based speech synthesis method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S108.

[0076] Step S101, obtaining the original spectrum of the original audio;

[0077] Step S102, preprocessing the original spectrum to obtain a target spectrum of the original audio;

[0078] Step S103, obtaining a language feature vector and a speaker feature vector; wherein the language feature vector includes phoneme information;

[0079] Step S104, inputting the target spectrum, language feature vector and speaker feature vector into a preset prosody encoder to obtain a reference prosody vector;

[0080] Step S105, inputting the language feature vector and the speaker feature vector into a preset prosody prediction model to obtain a predicted prosody vector;

[0081] Step S106, obtaining a feature vector of the language to be synthesized and a feature vector of the speaker to be synthesized;

[0082] Step S107, obtaining a latent prosody vector according to the reference prosody vector and the predicted prosody vector;

[0083] Step S108 , obtaining a target synthesized speech according to the feature vector of the language to be synthesized, the feature vector of the speaker to be synthesized, and the latent prosody vector.

[0084] In steps S101 to S108 shown in the embodiment of the present application, a reference prosody vector can be determined based on the target spectrum of the original audio. However, in the actual speech synthesis process, the audio cannot be obtained first, and thus the prosody cannot be extracted. Therefore, the embodiment of the present application introduces a prosody prediction model to obtain a predicted prosody vector. The input of the predicted prosody model is the language feature vector of the text (the voice feature vector may include phoneme information and text content) and the speaker feature vector (the speaker feature vector includes speaker encoding information). The output of the predicted prosody model is combined with the output of the prosody encoder to obtain a potential prosody encoding vector. This allows the output of the predicted prosody model to match the output of the prosody encoder, i.e., a consistent prosody vector is obtained. It is understandable that since the speaker feature vector (speaker identity) and the language feature vector (text content) are unrelated to the original audio, the predicted prosody vector obtained by the prosody generation model only contains prosody information that is unrelated to the speaker and text content. In summary, the embodiment of the present application can improve the prosody generation effect and further improve the effect of speech synthesis.

[0085] In step S101 of some embodiments, the original audio may refer to the audio of any sentence, and the original audio may be obtained from a webpage or a database. A spectrum corresponding to the original audio may be obtained by time-frequency conversion, which is recorded as the original spectrum. Specifically, the time-domain conversion refers to a Fourier transform.

[0086] In step S102 of some embodiments, the original spectrum generally contains phoneme information, speaker information, and prosody information. If the original spectrum is not preprocessed, it is difficult to directly separate the prosody vector from the original spectrum, resulting in inaccurate prosody vectors. Therefore, embodiments of the present application preprocess the original spectrum. This preprocessing includes filtering, spectrum masking, etc., with the goal of removing frequency portions of the original spectrum that are not related to prosody information. The resulting target spectrum can reduce the difficulty of subsequent prosody information separation. Specifically, taking spectrum masking as an example of preprocessing, in one example, an original spectrogram of the original spectrum is obtained; horizontal bars are added to the original spectrogram to mask the high-frequency portion of the original spectrogram to obtain a target spectrum; and / or vertical bars are added to the original spectrogram to mask the time portion of the original spectrum to obtain a target spectrum. Taking frequency filtering as an example of preprocessing, in another example, the original spectrum is filtered according to a preset frequency threshold to obtain a target spectrum of the original audio; wherein the frequency of the target spectrum is less than the frequency threshold. It's important to note that using only the target spectrum, or the low-frequency portion of the original spectrum, as input can reduce the difficulty of prosody separation. Because the target spectrum contains a nearly complete prosody, it contains much less timbre information (speaker information) and content information (language features) than the full spectrum. The resulting reference prosody vector, after encoding, contains prosody information but lacks timbre and text content information.

[0087] In step S103 of some embodiments, the text sequence may be input into a preset text encoder to obtain a language feature vector, and then a preset speaker embedder may be used to obtain a speaker feature vector, wherein the language feature vector includes phoneme information.

[0088] See also Figure 2 In some embodiments, the language feature vector of step S103 is obtained through the following steps, which may include but are not limited to steps S201 to S204.

[0089] Step S201, obtaining a text sequence, where the text sequence includes a plurality of words;

[0090] Step S202, converting the text sequence into a phoneme sequence and a word sequence;

[0091] Step S203, concatenating the phoneme sequence and the word sequence to obtain a joint sequence of each word;

[0092] Step S204: Encode each joint sequence using a preset text encoder to obtain a language feature vector.

[0093] In the embodiment of the present application, steps S201 to S204 are performed by directly concatenating the phoneme sequence and the word sequence to obtain a joint sequence of each word, and each joint sequence is encoded by a preset text encoder to obtain a language feature vector, so that the language feature vector is at the word level to reduce the difficulty of subsequent prosody separation. It should be noted that the word referred to in the embodiment of the present application specifically refers to a "word".

[0094] In step S104 of some embodiments, the prosody encoder consists of two layers, each of which is a set of convolutional layers with ReLU activation and layer normalization. The first layer derives word boundaries from the language feature vector and the speaker feature vector, and compresses the target spectrum into word-level hidden states based on the word boundaries. The second layer post-processes the word-level hidden states to ultimately obtain a reference prosody vector.

[0095] In another example, a word-level prosody encoder is proposed that uses the low-frequency part of the speech spectrum as input, filters out the high-frequency part, and also integrates the phoneme vector encoding corresponding to each word and the vector encoding of different speakers. This ensures that different words of different speakers have corresponding prosody vectors. The specific network structure of the encoder is composed of three layers of two-dimensional convolutional layers + word pooling layers + three layers of two-dimensional convolutional layers + vector quantization layers. For example, the low-frequency part of the spectrum of the original audio, the phoneme encoding vector (language feature vector) of the corresponding text, and the encoding vector (speaker feature vector) of the speaker are spliced ​​as the input of the prosody encoder, and the reference prosody vector is obtained through three layers of convolutional layers, word pooling layers, another three layers of convolutional layers, and vector quantization layers.

[0096] In step S105 of some embodiments, the prosody prediction model includes a predicted prosody factor. An initial predicted prosody factor can be calculated using a language feature vector and a speaker feature vector. The initial predicted prosody factor is then adjusted based on a loss function to obtain a target predicted prosody factor. It should be noted that the loss function includes a first loss function and a second loss function. The first loss function primarily characterizes the similarity or difference between a reference prosody vector and a predicted prosody vector. By adjusting the prosody prediction factor using the first loss function, the potential prosody vector generated by the prosody generation model obtained is irrelevant to the subsequent speech synthesis effect, thereby enhancing the learning ability of the reference prosody vector. This makes the potential prosody vector obtained by the prosody generation model more optimal, can improve the prosody generation effect, and further can improve the speech synthesis effect.

[0097] See also Figure 3 In some embodiments, the prosody prediction model of step S105 includes an attention mechanism module, a context encoder module, and a convolution module; step S105 may include but is not limited to steps S301 to S303.

[0098] Step S301: weighting the speaker feature vector using an attention mechanism module to obtain a target speaker feature vector.

[0099] Step S302: Encode the language feature vector through a context encoder module to obtain a target language feature vector;

[0100] Step S303 : jointly encode the target speaker feature vector and the target language feature vector through a convolution module to obtain a predicted prosody vector.

[0101] In steps S301 to S303 shown in the embodiment of the present application, the prosody prediction model and the prosody encoder are jointly trained in pre-training, which is separated from speech synthesis. That is, the effect of speech synthesis is not considered, and only the reference prosody vector is extracted from the target spectrum of the original audio, and the predicted prosody vector is obtained from the text and speaker information. The predicted prosody vector can be adjusted according to the reference prosody vector, and whether the predicted prosody vector at this time is suitable for subsequent speech synthesis is not considered. Therefore, the prosody prediction model of the embodiment of the present application introduces an attention mechanism, a context encoder module and a convolution module, which can enhance the learning of the context dependency relationship between the input text sequence and the target spectrum.

[0102] In step S107 of some embodiments, a latent prosody vector may be obtained based on the similarity between the reference prosody vector and the predicted prosody vector. For example, if the similarity is greater than a first similarity threshold, the predicted prosody vector is used as the latent prosody vector. If the similarity is less than or equal to the first similarity threshold, the predicted prosody vector is weighted based on the similarity to obtain the latent prosody vector. If the similarity is less than a second similarity threshold, the reference prosody vector is used as the latent prosody vector. It should be noted that the first similarity threshold is greater than the second similarity threshold.

[0103] See also Figure 4 In some other embodiments, step S107 may include but is not limited to steps S401 to S403.

[0104] Step S401, calculating a first loss function based on the reference prosody vector and the predicted prosody vector;

[0105] Step S402, adjusting parameters of the rhythm prediction model according to the first loss function to obtain a rhythm generation model;

[0106] Step S403: Input the language feature vector and the speaker feature vector into a prosody generation model to obtain a latent prosody vector.

[0107] In steps S401 to S403 of the present embodiment, a loss function is calculated based on the reference prosody vector and the predicted prosody vector. The parameters of the prosody prediction model are adjusted based on the loss function to obtain a prosody generation model. The prosody generation model then outputs a latent prosody vector, thereby obtaining a more optimal prosody vector.

[0108] See also Figure 5 In some embodiments, after step S402, the method of this embodiment further includes updating the rhythm generation model, which may specifically include but is not limited to steps S501 to S504.

[0109] Step S501: inputting the language feature vector, speaker feature vector and reference prosody vector into a preset decoder module to obtain a reference spectrum;

[0110] Step S502: input the language feature vector, speaker feature vector and predicted prosody vector into a decoder module to obtain a predicted spectrum;

[0111] Step S503, calculating a second loss function according to the reference spectrum and the predicted spectrum;

[0112] Step S504: Adjust parameters of the rhythm generation model according to the second loss function to update the rhythm generation model.

[0113] In steps S501 to S504 illustrated in the embodiment of the present application, the second loss function primarily represents the similarity or difference between the reference spectrum and the predicted spectrum. The prosody prediction factor is adjusted using the second loss function. The resulting latent prosody vector generated by the prosody generation model is correlated with the subsequent speech synthesis performance. This improves the performance of synthesized speech obtained using the latent prosody vector during training, but reduces the learning ability of the reference prosody vector, impacting the generation performance of the latent prosody vector. Therefore, in this embodiment, the predicted prosody factor can be trained using a hybrid method using the first and second loss functions, resulting in a more optimized latent prosody vector obtained by the prosody generation model, which can improve the prosody generation performance and, in turn, speech synthesis performance.

[0114] See also Figure 6 In some embodiments, the reference spectrum of step S501 is obtained by the following steps, which may include but are not limited to steps S601 to S603:

[0115] Step S601, adding the language feature vector, the speaker feature vector and the reference prosody vector to obtain a preliminary vector corresponding to each word;

[0116] Step S602: copying the preliminary vector according to a preset spectrum duration threshold to obtain a target vector corresponding to each word; wherein the spectrum duration of each target vector is equal to the spectrum duration threshold;

[0117] Step S603: Decode the multiple target vectors through a decoder module to obtain a reference spectrum.

[0118] In the steps S501 to S503 shown in the embodiment of the present application, first, the speech feature vector is the feature vector obtained after text encoding, the speaker feature vector is the timbre vector extracted by X-vector that can characterize different speakers, and the reference prosody vector is a latent variable representation obtained by the prosody encoder. The three vectors are added together to obtain a preliminary vector. Since the audio spectrum corresponding to each word has a different duration, the corresponding vector must be copied according to the predicted duration to form a new variable sequence. Therefore, the number of spectrum frames corresponding to the preliminary vector is adjusted by a duration predictor. At this time, the preliminary vector will be copied according to the number of frames, and the target vector obtained after the duration adjustment is input into the decoder for decoding, and finally the reference spectrum is obtained. It should be noted that the decoder is implemented using a 4-layer two-dimensional convolutional layer in the embodiment of the present application, and the output of the decoder is passed through a linear layer to obtain the corresponding spectrum feature representation, that is, the reference spectrum.

[0119] In some embodiments, the predicted spectrum in step S502 is obtained by the following steps, which may include but are not limited to:

[0120] Add the language feature vector, speaker feature vector and predicted prosody vector to obtain the preliminary prediction vector corresponding to each word;

[0121] The preliminary prediction vector is copied according to a preset spectrum duration threshold to obtain a target prediction vector corresponding to each of the words; wherein the spectrum duration of each target prediction vector is equal to the spectrum duration threshold;

[0122] The decoder module decodes multiple target prediction vectors to obtain the predicted spectrum.

[0123] Specifically, the decoder in the embodiment of the present application is implemented using a four-layer two-dimensional convolutional layer, and the output of the decoder is passed through a linear layer to obtain the corresponding spectrum feature representation, that is, the predicted spectrum.

[0124] See also Figure 7 The embodiment of the present application also provides a prosody-based speech synthesis device, which can implement the above-mentioned prosody-based speech synthesis method. Figure 7This is a block diagram of the module structure of the prosody-based speech synthesis device provided in an embodiment of the present application. The device includes: an original spectrum acquisition module 701, a preprocessing module 702, an original vector acquisition module 703, a reference prosody determination module 704, a predicted prosody determination module 705, a to-be-synthesized vector acquisition module 706, a potential prosody determination module 707, and a speech synthesis module 708. Among them, the original spectrum acquisition module 701 is used to obtain the original spectrum of the original audio; the preprocessing module 702 is used to preprocess the original spectrum to obtain the target spectrum of the original audio; the original vector acquisition module 703 is used to obtain the language feature vector and the speaker feature vector; wherein the language feature vector includes phoneme information; the reference prosody determination module 704 is used to input the target spectrum, the language feature vector and the speaker feature vector into a preset prosody encoder to obtain a reference prosody vector; the predicted prosody determination module 705 is used to input the language feature vector and the speaker feature vector into a preset prosody prediction model to obtain a predicted prosody vector; the to-be-synthesized vector acquisition module 706 is used to obtain the to-be-synthesized language feature vector and the to-be-synthesized speaker feature vector; the latent prosody determination module 707 is used to obtain the latent prosody vector based on the reference prosody vector and the predicted prosody vector; the speech synthesis module 708 is used to obtain the target synthesized speech based on the to-be-synthesized language feature vector, the to-be-synthesized speaker feature vector and the latent prosody vector.

[0125] In another embodiment, the prosody-based speech synthesis device further includes a reference spectrum determination module, a predicted spectrum determination module, a spectrum loss calculation module, and an update module. The reference spectrum determination module is configured to input a language feature vector, a speaker feature vector, and a reference prosody vector into a preset decoder module to obtain a reference spectrum; the predicted spectrum determination module is configured to input the language feature vector, the speaker feature vector, and the predicted prosody vector into the decoder module to obtain a predicted spectrum; the spectrum loss calculation module is configured to calculate a second loss function based on the reference spectrum and the predicted spectrum; and the update module is configured to adjust parameters of the prosody generation model based on the second loss function to update the prosody generation model.

[0126] It should be noted that the specific implementation of the prosody-based speech synthesis device is basically the same as the specific embodiment of the above-mentioned prosody-based speech synthesis method, and will not be repeated here.

[0127] The present application also provides a computer device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communicating between the processor and the memory. When the program is executed by the processor, the prosody-based speech synthesis method described above is implemented. The computer device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0128] See also Figure 8 , Figure 8The hardware structure of a computer device according to another embodiment is shown. The computer device includes:

[0129] The processor 801 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0130] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called by the processor 801 to execute the prosody-based speech synthesis method of the embodiments of this application.

[0131] Input / output interface 803, used to implement information input and output;

[0132] Communication interface 804, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0133] Bus 805 , which transmits information between various components of the device (e.g., processor 801 , memory 802 , input / output interface 803 , and communication interface 804 );

[0134] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0135] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned prosody-based speech synthesis method.

[0136] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0137] The prosody-based speech synthesis method, prosody-based speech synthesis device, computer equipment and storage medium provided in the embodiments of the present application introduce a prosody prediction model to obtain a predicted prosody vector. During the training process, the input of the predicted prosody model is the language feature vector and the speaker feature vector of the text. The loss function can be calculated based on the output of the predicted prosody model and the output of the prosody encoder. The parameters of the predicted prosody model are adjusted according to the loss function so that the output of the predicted prosody model can match the output of the prosody encoder, that is, a consistent prosody vector is obtained. After the training is completed, a prosody generation model is obtained. It can be understood that since the speaker feature vector and the language feature vector are independent of the original audio, the predicted prosody vector obtained by the prosody generation model only contains prosody information that is independent of the speaker and the text content. In summary, the embodiments of the present application can improve the prosody generation effect. It can further improve the effect of speech synthesis.

[0138] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0139] It will be understood by those skilled in the art that Figure 1-6 The technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or a combination of certain steps, or different steps.

[0140] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0141] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0142] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0143] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0146] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0147] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0148] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A prosody-based speech synthesis method, characterized in that: The method comprises: Get the original spectrum of the original audio; Preprocessing the original frequency spectrum to obtain a target frequency spectrum of the original audio; Acquire a language feature vector and a speaker feature vector; wherein the language feature vector includes phoneme information; Inputting the target spectrum, the language feature vector, and the speaker feature vector into a preset prosody encoder to obtain a reference prosody vector; Inputting the language feature vector and the speaker feature vector into a preset prosody prediction model to obtain a predicted prosody vector; Obtaining a language feature vector to be synthesized and a speaker feature vector to be synthesized; Calculate a first loss function according to the reference prosody vector and the predicted prosody vector; Adjusting parameters of the rhythm prediction model according to the first loss function to obtain a rhythm generation model; Inputting the language feature vector and the speaker feature vector into the prosody generation model to obtain a latent prosody vector; A target synthesized speech is obtained according to the feature vector of the language to be synthesized, the feature vector of the speaker to be synthesized, and the latent prosody vector.

2. The method according to claim 1, characterized in that The preprocessing of the original spectrum to obtain the target spectrum of the original audio includes: The original spectrum is filtered according to a preset frequency threshold to obtain a target spectrum of the original audio; wherein the frequency of the target spectrum is less than the frequency threshold.

3. The method according to claim 1, characterized in that The language feature vector is obtained by the following steps, including: Acquire a text sequence, wherein the text sequence includes a plurality of words; Converting the text sequence into a phoneme sequence and a word sequence; Concatenating the phoneme sequence and the word sequence to obtain a joint sequence of each word; Each of the combined sequences is encoded by a preset text encoder to obtain the language feature vector.

4. The method according to claim 3, characterized in that After adjusting parameters of the prosody prediction model according to the first loss function to obtain a prosody generation model, the method further includes: Inputting the language feature vector, the speaker feature vector and the reference prosody vector into a preset decoder module to obtain a reference spectrum; Inputting the language feature vector, the speaker feature vector and the predicted prosody vector into the decoder module to obtain a predicted spectrum; Calculating a second loss function according to the reference spectrum and the predicted spectrum; Parameters of the rhythm generation model are adjusted according to the second loss function to update the rhythm generation model.

5. The method according to claim 4, characterized in that The step of inputting the language feature vector, the speaker feature vector, and the reference prosody vector into a preset decoder module to obtain a reference spectrum includes: Adding the language feature vector, the speaker feature vector, and the reference prosody vector to obtain a preliminary vector corresponding to each word; The preliminary vector is copied according to a preset spectrum duration threshold to obtain a target vector corresponding to each of the words; wherein the spectrum duration of each of the target vectors is equal to the spectrum duration threshold; The plurality of target vectors are decoded by the decoder module to obtain a reference spectrum.

6. The method according to claim 1, characterized in that The prosody prediction model includes an attention mechanism module, a context encoder module, and a convolution module; the step of inputting the language feature vector and the speaker feature vector into a preset prosody prediction model to obtain a predicted prosody vector includes: Performing weight calculation on the speaker feature vector by the attention mechanism module to obtain a target speaker feature vector; Encoding the language feature vector by the context encoder module to obtain a target language feature vector; The target speaker feature vector and the target language feature vector are jointly encoded through a convolution module to obtain a predicted prosody vector.

7. A prosody-based speech synthesis device, characterized in that: The device comprises: The original spectrum acquisition module is used to obtain the original spectrum of the original audio; A preprocessing module, configured to preprocess the original spectrum to obtain a target spectrum of the original audio; An original vector acquisition module, configured to acquire a language feature vector and a speaker feature vector; wherein the language feature vector includes phoneme information; a reference prosody determination module, configured to input the target spectrum, the language feature vector, and the speaker feature vector into a preset prosody encoder to obtain a reference prosody vector; A predicted prosody determination module, configured to input the language feature vector and the speaker feature vector into a preset prosody prediction model to obtain a predicted prosody vector; A vector acquisition module to be synthesized, used to obtain a language feature vector to be synthesized and a speaker feature vector to be synthesized; a latent prosody determination module, configured to calculate a first loss function based on the reference prosody vector and the predicted prosody vector; adjust parameters of the prosody prediction model based on the first loss function to obtain a prosody generation model; and input the language feature vector and the speaker feature vector into the prosody generation model to obtain a latent prosody vector; The speech synthesis module is used to obtain the target synthesized speech according to the feature vector of the language to be synthesized, the feature vector of the speaker to be synthesized and the latent prosody vector.

8. A computer device, characterized in that: The computer device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A storage medium, which is a computer-readable storage medium and is used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of claims 1 to 6.

Images