Speech synthesis method and related equipment

Abstract

The present disclosure belongs to the field of artificial intelligence technology and relates to machine learning technology. Specifically, the present disclosure relates to a speech synthesis method and related equipment. The method includes: obtaining target text information and its target style emotion information; using a shared encoder to perform text encoding processing on the target text information to obtain target encoding features of the target text information; determining a target advanced decoder corresponding to the target style emotion information from K advanced decoders corresponding to K types of style emotion information, K is an integer greater than 1, and the K types of style emotion information include the target style emotion information; using the target advanced decoder to decode the target encoding features to obtain target acoustic features corresponding to the target style emotion information; performing acoustic encoding processing on the target acoustic features to obtain target audio information corresponding to the target text information and the target style emotion information. This method can improve the performance of speech synthesis with target style emotion information.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology and machine learning technology. Specifically, the present disclosure relates to a speech synthesis method, a speech synthesis device, a computer-readable medium, an electronic device, and a computer program product. Background Art

[0002] With the rapid development of artificial intelligence technology and intelligent hardware devices (such as smartphones, smart speakers, etc.), voice interaction technology has been increasingly used as a natural way of interaction. As an important part of voice interaction technology, speech synthesis technology has also made great progress. Speech synthesis technology is also known as text-to-speech technology (TTS). Its function is to convert text information generated by the computer itself or input externally into fluent speech that users can understand and play it out.

[0003] In some applications such as audio reading, in addition to requiring the synthesized audio to be understandable, smooth and natural, users also have higher requirements for the style (e.g., narration, dialogue, etc.) and emotions (e.g., joy, anger, sadness, happiness, etc.) of the synthesized audio. However, the speech synthesis solutions for different styles and emotions in the related art still have the problem that the speech synthesis performance cannot meet the requirements, such as the problem of poor synthesis accuracy and stability due to insufficient audio of one or some emotions, and the problem that different styles and emotions share the same decoder, which weakens the characteristics of each style and emotion, and the problem of additional computing resources introduced by sharing the same decoder.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a speech synthesis method, a speech synthesis device, a computer-readable medium, an electronic device and a computer program product, which can at least to some extent overcome the technical problem of low performance of speech synthesis suitable for different styles and emotions.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of an embodiment of the present disclosure, a speech synthesis method is provided, which includes: acquiring target text information and its target style emotion information; performing text encoding processing on the target text information using a shared encoder to obtain target encoding features of the target text information; determining a target advanced decoder corresponding to the target style emotion information from K advanced decoders corresponding to K types of style emotion information, where K is an integer greater than 1, and the K types of style emotion information include the target style emotion information; performing decoding processing on the target encoding features using the target advanced decoder to obtain target acoustic features corresponding to the target style emotion information; and performing acoustic encoding processing on the target acoustic features to obtain target audio information corresponding to the target text information and the target style emotion information.

[0008] According to one aspect of an embodiment of the present disclosure, a speech synthesis device is provided, which includes: an acquisition unit, used to acquire target text information and its target style emotion information; a processing unit, used to perform text encoding processing on the target text information using a shared encoder to obtain target encoding features of the target text information; the processing unit is also used to determine a target advanced decoder corresponding to the target style emotion information from K advanced decoders corresponding to K types of style emotion information, K is an integer greater than 1, and the K types of style emotion information include the target style emotion information; the processing unit is also used to decode the target encoding features using the target advanced decoder to obtain target acoustic features corresponding to the target style emotion information; the processing unit is also used to perform acoustic encoding processing on the target acoustic features to obtain target audio information corresponding to the target text information and the target style emotion information.

[0009] In some embodiments of the present disclosure, based on the above technical solution, the processing unit is also used to: determine a target style emotion embedding vector corresponding to the target style emotion information from K styles of emotion embedding vectors corresponding to K styles of emotion information; fuse the target coding feature and the target style emotion embedding vector to obtain a target fused feature vector; use the target advanced decoder to decode the target fused feature vector to obtain target advanced decoding information of the target text information; post-process the target advanced decoding information based on the target style emotion information to obtain the target acoustic feature corresponding to the target style emotion information.

[0010] In some embodiments of the present disclosure, based on the above technical solution, the processing unit is also used to: determine a target duration predictor corresponding to the target style emotion information from K duration predictors corresponding to K styles of emotion information; use the target duration predictor to perform duration prediction processing on the target coding feature to obtain the target duration information of the target text information; use the target duration information to upsample the target coding feature; and fuse the upsampled target coding feature with the target style emotion embedding vector to obtain the target fused feature vector.

[0011] In some embodiments of the present disclosure, based on the above technical solution, the processing unit is also used to: use a shared preliminary decoder to perform preliminary decoding processing on the target fusion feature vector to obtain target preliminary decoding information of the target text information; use the target advanced decoder to perform advanced decoding processing on the target preliminary decoding information to obtain the target advanced decoding information.

[0012] In some embodiments of the present disclosure, based on the above technical solution, the processing unit is also used to: determine a target post-processing module corresponding to the target style emotion information from K post-processing modules corresponding to K types of style emotion information; and use the target post-processing module to post-process the target advanced decoding information to obtain the target acoustic features.

[0013] In some embodiments of the present disclosure, based on the above technical solution, the K types of style emotion information include first style emotion information and second style emotion information. The acquisition unit is further used to acquire first training text information and first training audio information with the first style emotion information, and second training text information and second training audio information with the second style emotion information. The processing unit is also used to use the shared encoder to perform text encoding processing on the first training text information and the second training text information respectively to obtain a first training encoding feature of the first training text information and a second training encoding feature of the second training text information; determine a first advanced decoder corresponding to the first style emotion information from K advanced decoders, and determine a second advanced decoder corresponding to the second style emotion information from K advanced decoders; use the first advanced decoder to decode the first training encoding feature to obtain a first training acoustic feature corresponding to the first style emotion information, and use the second advanced decoder to decode the second training encoding feature to obtain a second training acoustic feature corresponding to the second style emotion information; perform acoustic encoding processing on the first training acoustic feature to obtain a first predicted audio information corresponding to the first training text information and the first style emotion information, and perform acoustic encoding processing on the second training acoustic feature to obtain a second predicted audio information corresponding to the second training text information and the second style emotion information; train the shared encoder, the first advanced decoder and the second advanced decoder based on the first training audio information and the first predicted audio information, and the second training audio information and the second predicted audio information.

[0014] In some embodiments of the present disclosure, based on the above technical solution, K types of style emotion information correspond to K types of style emotion embedding vectors, and the K types of style emotion embedding vectors include a first style emotion embedding vector corresponding to the first style emotion information and a second style emotion embedding vector corresponding to the second style emotion information. Wherein, the processing unit is also used to: subject the first training coding feature and the first style emotion embedding vector corresponding to the first style emotion information to fusion processing to obtain a first fusion feature vector, and subject the second training coding feature and the second style emotion embedding vector corresponding to the second style emotion information to fusion processing to obtain a second fusion feature vector; use the first advanced decoder to decode the first fusion feature vector to obtain the first advanced decoding information of the first training text information, and use the second advanced decoder to decode the second fusion feature vector to obtain the second advanced decoding information of the second training text information; post-process the first advanced decoding information based on the first style emotion information to obtain the first training acoustic feature, and post-process the second advanced decoding information based on the second style emotion information to obtain the second training acoustic feature.

[0015] In some embodiments of the present disclosure, based on the above technical solution, the processing unit is also used to: determine a first duration predictor corresponding to the first style emotion information from K duration predictors corresponding to K styles of emotion information, and determine a second duration predictor corresponding to the second style emotion information from the K duration predictors; use the first duration predictor to perform duration prediction processing on the first training coding feature to obtain the first duration information of the first training text information, and use the second duration predictor to perform duration prediction processing on the second training coding feature to obtain the second duration information of the second training text information; use the first duration information to upsample the first training coding feature, and use the second duration information to upsample the second training coding feature; fuse the upsampled first training coding feature with the first style emotion embedding vector to obtain the first fused feature vector, and fuse the upsampled second training coding feature with the second style emotion embedding vector to obtain the second fused feature vector.

[0016] In some embodiments of the present disclosure, based on the above technical solution, the processing unit is also used to: use a shared preliminary decoder to perform preliminary decoding processing on the first fused feature vector and the second fused feature vector, respectively, to obtain first preliminary decoding information of the first training text information and second preliminary decoding information of the second training text information; use the first advanced decoder to perform advanced decoding processing on the first preliminary decoding information to obtain the first advanced decoding information, and use the second advanced decoder to perform advanced decoding processing on the second preliminary decoding information to obtain the second advanced decoding information.

[0017] In some embodiments of the present disclosure, based on the above technical solution, the processing unit is also used to: determine a first post-processing module corresponding to the first style emotion information from K post-processing modules corresponding to K types of style emotion information, and determine a second post-processing module corresponding to the second style emotion information from the K post-processing modules; use the first post-processing module to post-process the first advanced decoding information to obtain the first training acoustic feature, and use the second post-processing module to post-process the second advanced decoding information to obtain the second training acoustic feature.

[0018] According to one aspect of an embodiment of the present disclosure, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the speech synthesis method in the above technical solution is implemented.

[0019] According to one aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the speech synthesis method in the above technical solution by executing the executable instructions.

[0020] According to one aspect of the present disclosure, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the methods provided in the above-mentioned various optional implementations.

[0021] In the technical solution provided by the embodiments of the present disclosure, when acquiring the target text information and its target style emotion information, on the one hand, by using a shared encoder to perform text encoding processing on the target text information to obtain the target encoding features of the target text information, the shared encoder can be used to perform text encoding processing on the target text information with any target style emotion information, thereby improving the accuracy and stability of speech synthesis; on the other hand, by determining the target advanced decoder corresponding to the target style emotion information from the K advanced decoders corresponding to the K types of style emotion information, K is an integer greater than 1, and the K types of style emotion information include the target style emotion information; using the target advanced decoder to decode the target encoding features to obtain the target acoustic features corresponding to the target style emotion information, and performing acoustic encoding processing on the target acoustic features to obtain the target audio information corresponding to the target text information and the target style emotion information. Since different style emotion information corresponds to different advanced decoders, the timbre characteristics and rhythmic information of different style emotion information can be retained. At the same time, when decoding the target coding features corresponding to the target style emotion information, it only needs to pass through the target advanced decoder corresponding to the required target style emotion information, and the advanced decoders corresponding to other style emotion information do not need to be calculated, thereby reducing the amount of calculation and saving computing resources.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0024] Figure 1 The following schematically shows an exemplary system architecture diagram of the technical solution of the present disclosure in an application scenario.

[0025] Figure 2 An exemplary system architecture of the technical solution of the present disclosure in another application scenario is schematically shown.

[0026] Figure 3 The flowchart schematically shows the steps of a speech synthesis method provided in an exemplary embodiment of the present disclosure.

[0027] Figure 4 The flowchart schematically shows the steps of a speech synthesis method provided in another exemplary embodiment of the present disclosure.

[0028] Figure 5 The following schematically shows a flowchart of the steps of a speech synthesis method provided in another exemplary embodiment of the present disclosure.

[0029] Figure 6 The following schematically shows a flowchart of the steps of a speech synthesis method provided in another exemplary embodiment of the present disclosure.

[0030] Figure 7 The following schematically shows a flowchart of the steps of a speech synthesis method provided in another exemplary embodiment of the present disclosure.

[0031] Figure 8 The following schematically shows a flowchart of the steps of a speech synthesis method provided in another exemplary embodiment of the present disclosure.

[0032] Fig. 9 The figure schematically shows a speech synthesis method provided in an exemplary embodiment of the present disclosure.

[0033] Fig.10 The schematic diagram of the structure of an encoder provided in an exemplary embodiment of the present disclosure is schematically shown.

[0034] Fig.11 The schematic diagram of the structure of a deformer module provided in an exemplary embodiment of the present disclosure is schematically shown.

[0035] Fig.12 The schematic diagram shows the structure of a duration predictor provided in an exemplary embodiment of the present disclosure.

[0036] Fig.13 The block diagram schematically shows the composition of a speech synthesis device provided in an exemplary embodiment of the present disclosure.

[0037] Fig.14 The structure diagram of a computer system suitable for implementing the electronic device of the embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0038] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more comprehensive and complete and will fully convey the concept of the example embodiments to those skilled in the art.

[0039] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0040] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0041] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0042] First, some terms involved in this disclosure are explained.

[0043] Fundamental frequency: In sound, fundamental frequency refers to the frequency of the fundamental tone in a complex tone, represented by the symbol FO. Among the several tones that make up a complex tone, the fundamental tone has the lowest frequency and the greatest intensity. The height of the fundamental frequency determines the height of a tone. The so-called frequency of speech usually refers to the frequency of the fundamental tone.

[0044] Vocoder: Vocoder is derived from the abbreviation of Voice Encoder, also known as speech signal analysis and synthesis system. Its function is to convert acoustic features into sound.

[0045] HMM (Hidden Markov Model): Hidden Markov Model is a statistical analysis model used to describe a Markov process with hidden unknown parameters. In a hidden Markov model, the state is not directly visible, but some variables (observations) affected by the state are visible.

[0046] Before describing the technical solutions such as the speech synthesis method, speech synthesis device, computer-readable medium, electronic device, and computer program product provided by the present invention, a brief description is first given of the cloud technology and artificial intelligence (AI) technology involved in the technical solutions of the present invention.

[0047] Cloud computing refers to the delivery and use model of IT (Internet Technology) infrastructure, which means obtaining required resources through the network in an on-demand and easily scalable manner. This service can be related to IT and software, the Internet, or other services. Cloud computing is the product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing.

[0048] With the development of the Internet, real-time data streams, and the diversification of connected devices, as well as the demand for search services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Different from the previous parallel distributed computing, the emergence of cloud computing will promote revolutionary changes in the entire Internet model and enterprise management model from a conceptual perspective.

[0049] Artificial intelligence cloud service is generally referred to as AIaaS (AI as a Service). This is the service mode of a mainstream artificial intelligence platform. Specifically, the AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to opening an AI theme mall: all developers can access one or more artificial intelligence services provided by the platform through the API (Application Programming Interface). Some senior developers can also use the AI ​​framework and AI infrastructure provided by the platform to deploy and operate their own cloud artificial intelligence services.

[0050] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0051] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0052] The key technologies of speech technology include automatic speech recognition (ASR), text-to-speech (TTS) and voiceprint recognition. Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction, among which speech has become one of the most promising human-computer interaction methods in the future.

[0053] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0054] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0055] The application scenarios of the present disclosure are wide-ranging. A speech synthesis solution with multiple branches applicable to multiple styles (such as narration, dialogue, etc.) and multiple emotions (such as joy, anger, sadness, happiness, etc.) (which can be called a multi-branch multi-style emotional speech synthesis service) can be configured as a cloud service, which can be used as a basic technology to empower users who use the cloud service, and the solution can also be used for personalized scenarios in vertical fields. For example, it can be applied to scenarios such as intelligent reading of reading APP (application), intelligent customer service, news broadcasting, and intelligent device interaction, to achieve intelligent speech synthesis in various scenarios.

[0056] Figure 1 The following schematically shows an exemplary system architecture diagram of the technical solution of the present disclosure in an application scenario.

[0057] like Figure 1 As shown, the system architecture 100 may include a client 110, a network 120, and a server 130. The client 110 may include various terminal devices such as smart phones, smart robots, smart speakers, tablet computers, laptops, desktop computers, etc. The server 130 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The network 120 may be a communication medium of various connection types that can provide a communication link between the client 110 and the server 130, such as a wired communication link, a wireless communication link, etc.

[0058] According to implementation requirements, the technical solution provided by the embodiments of the present disclosure may be applied to the client 110 or the server 130, or may be implemented jointly by the client 110 and the server 130, and the present disclosure does not make any special limitation on this.

[0059] For example, various smart devices such as smart robots and smart phones can access the multi-branch multi-style emotional speech synthesis service on the cloud server through a wireless network. After the client 110 accesses the server 130, it selects the required target style emotional information and sends the target text information to be synthesized to the server 130. After the server 130 performs rapid synthesis, it can send the corresponding synthesized target audio information to the client 110 in the form of streaming or whole sentence return. A complete speech synthesis process may include, for example:

[0060] The client 110 uploads the target text information to be synthesized and the required target style and emotion information (which can be selected from the style and emotion options displayed on the client 110 or directly input) to the server 130, and the server 130 performs corresponding regularization processing after receiving the text;

[0061] After receiving the target text information, the server 130 performs corresponding regularization processing, for example, regularization processing, such as converting Chinese characters into correct pinyin, removing punctuation marks and converting them into corresponding pauses, etc.;

[0062] The server 130 inputs the regularized target text information into the branch corresponding to the multi-branch multi-style emotional speech synthesis service (including the branch of the target advanced decoder), quickly synthesizes the target text information and the target style emotional information corresponding to the target audio information, and completes post-processing operations such as audio compression;

[0063] The server 130 returns the target audio information to the client 110 in a streaming or whole sentence manner;

[0064] After receiving the target audio information, the client 110 can play the target audio information in a smooth and natural voice.

[0065] In the above speech synthesis process, the speech synthesis service provided by the server 130 has a small delay, and the client 110 can basically obtain the return result immediately. The user can hear the required content in a short time, freeing his eyes, and the interaction is natural and convenient.

[0066] Figure 2 The exemplary system architecture of the disclosed technical solution in another application scenario is schematically shown. The system architecture and process are mainly applied to the vertical fields of speech synthesis services such as novel reading and news broadcasting, which require customized style and emotional information.

[0067] The process of implementing customized audio synthesis services under this system architecture can mainly include:

[0068] The front-end demander 210 submits a demand list for speech synthesis required for its product, such as K styles of emotional information and other various requirements.

[0069] After receiving the list from the demander, the backend service provider 220 collects K types of emotional information and their corresponding training audio information according to the required K types of emotional information styles, and trains the corresponding speech synthesis model 230 .

[0070] The backend service provider 220 uses the speech synthesis model 230 to synthesize the sample. After delivering the sample to the frontend demander 210 for verification and confirmation, the customized speech synthesis model 230 can be deployed online.

[0071] The application of the front-end demand side 210 (such as a reading APP, a news client, etc.) sends the target text information and the required target style emotion information of the audio to be synthesized to the speech synthesis model 230 deployed on the back-end service side 220;

[0072] The user of the front-end demand side 210 can hear the text content corresponding to the target text information read aloud with the corresponding target style emotion information in the application program. The specific audio synthesis process is similar to Figure 1 The online synthesis service used in the system architecture shown is the same.

[0073] In this application scenario, after the front-end demander 210 provides the demand, the back-end service provider 220 can train the speech synthesis model according to various styles of emotional information and their corresponding training audio information during training, thereby improving the accuracy and stability of the trained speech synthesis model. When in use, only the branch corresponding to the target style emotional information in the speech synthesis model needs to be used during decoding, thereby reducing the amount of calculation.

[0074] The technical solution provided by the present disclosure is described in detail below in conjunction with specific implementation methods.

[0075] Figure 3 The flowchart of the steps of the speech synthesis method provided in an exemplary embodiment of the present disclosure is schematically shown. The execution subject of the speech synthesis method can be various terminal devices such as smart phones and smart speakers as clients, or various server devices such as physical servers and cloud servers as servers. Figure 3 As shown, the speech synthesis method may mainly include the following steps S310 to S350:

[0076] Step S310: Obtain target text information and target style emotion information.

[0077] In the embodiment of the present disclosure, the target text information may be any text to be synthesized into speech, for example, Figure 1 The client 110 in the embodiment uploads to the server 130. The target text information can be in one language or a combination of multiple languages. In the following embodiment, Chinese is used as an example, but the present disclosure is not limited thereto.

[0078] In the disclosed embodiment, the style and emotion information refers to information used to determine which style and / or which emotion to use for voice expression, such as whether to use a narration style or a dialogue style, whether to use a happy emotion or a sad emotion, etc. The target style and emotion information is information specified by the user as to which style and emotion the target text information is to be synthesized. For example, different style options and different emotion options can be displayed on the screen of the client 110, and the style option and / or emotion option selected by the user is used as the target style and emotion information and uploaded to the server 130.

[0079] Step S320: Perform text encoding processing on the target text information using a shared encoder to obtain target encoding features of the target text information.

[0080] In the disclosed embodiments, a shared encoder refers to an encoder that can be used to perform text encoding processing on target text information having a plurality of different target style emotion information. For example, regardless of whether the target style emotion information is narration, dialogue, or any one of joy, anger, sorrow, and happiness, the shared encoder can be used to perform text encoding processing to obtain target encoding features. In the training stage, the shared encoder is an encoder trained using a plurality of (e.g., K types, K being an integer greater than 1) different styles of emotion information and their corresponding training text information and training audio information. In the training stage, the training text information corresponding to different styles of emotion information is all processed by the shared encoder for text encoding to obtain corresponding training encoding features.

[0081] Step S330: Determine a target advanced decoder corresponding to the target style emotion information from K advanced decoders corresponding to K styles emotion information, where K is an integer greater than 1, and the K styles emotion information include the target style emotion information.

[0082] In the disclosed embodiment, during the decoding process of training, different advanced decoders are trained for different styles of emotion information. For example, K advanced decoders are trained for K types of style emotion information. That is, the encoders are shared, but the advanced decoders are independent for each style of emotion information, which can also be called multi-branch. Therefore, when in use, first determine the target advanced decoder corresponding to the target style emotion information from the K advanced decoders.

[0083] Step S340: Utilize the target advanced decoder to decode the target coding feature to obtain a target acoustic feature corresponding to the target style emotion information.

[0084] Then, the target encoding features obtained by encoding with the shared encoder may be decoded using a pre-trained target advanced decoder to obtain target acoustic features corresponding to the target style emotion information.

[0085] The acoustic features in the embodiments of the present disclosure (including the target acoustic features and the acoustic features mentioned in other embodiments) may be, for example, features with timbre characteristics and sound content presented in the form of spectrograms or other forms. Spectrum refers to the representation of a time domain signal in the frequency domain, which can be obtained by Fourier transforming the sound signal. The result is two images with amplitude and phase as the vertical axis and frequency as the horizontal axis. In the application of speech synthesis technology, phase information is often omitted, and only the amplitude information corresponding to different frequencies is retained.

[0086] Step S350: Perform acoustic coding processing on the target acoustic features to obtain target audio information corresponding to the target text information and the target style emotion information.

[0087] In this step, the target acoustic features can be input into a vocoder, which converts and processes the target acoustic features to form target audio information that can be output and played through an audio output device such as a speaker.

[0088] In the speech synthesis method provided by the embodiment of the present disclosure, when acquiring the target text information and its target style emotion information, on the one hand, by using a shared encoder to perform text encoding processing on the target text information to obtain the target encoding features of the target text information, the shared encoder can be used to perform text encoding processing on the target text information with any target style emotion information, thereby improving the accuracy and stability of speech synthesis; on the other hand, by also determining the target advanced decoder corresponding to the target style emotion information from K advanced decoders corresponding to K types of style emotion information, K is an integer greater than 1, and the K types of style emotion information include the target style emotion information; using the target advanced decoder to decode the target encoding features to obtain the target acoustic features corresponding to the target style emotion information, and performing acoustic encoding processing on the target acoustic features to obtain the target audio information corresponding to the target text information and the target style emotion information. Since different style emotion information corresponds to different advanced decoders, the timbre characteristics and rhythmic information of different style emotion information can be retained. At the same time, when decoding the target coding features corresponding to the target style emotion information, it only needs to pass through the target advanced decoder corresponding to the required target style emotion information, and the advanced decoders corresponding to other style emotion information do not need to be calculated, thereby reducing the amount of calculation and saving computing resources.

[0089] Speech synthesis technology converts text into corresponding audio content through certain rules or model algorithms. Traditional speech synthesis technology is mainly based on splicing methods or statistical parameter methods. As deep learning continues to make breakthroughs in the field of speech recognition, deep learning has begun to be introduced into the field of speech synthesis.

[0090] During the actual development process, the amount of training audio information and training text information of different styles and emotions of the same speaker that can be obtained is very unbalanced. Relatively stable narration-style training audio information and training text information are easier to obtain, while training audio information and training text information containing obvious emotions are more difficult to obtain. For example, the ratio of the amount of narration-style training audio information and training text information to the amount of training audio information and training text information for a single emotion can be up to 1000:1.

[0091] The following embodiment takes the basic structure of the end-to-end speech synthesis model as an example, and uses a variety of different style emotion information for training, and introduces a style emotion embedding (embedding) vector extracted by an unrestricted extraction method after the shared encoder to indicate which style or emotion each training audio information corresponds to. The multi-branch multi-style multi-emotion speech synthesis model obtained in this way can simultaneously integrate multiple types of styles and emotions, and can use the types with sufficient training data and the types with insufficient training data to participate in the training at the same time, so as to alleviate the problems caused by the incomplete coverage of the training audio information and training text information of some emotion types and the low recording quality to a certain extent. On the other hand, in addition to using the style emotion embedding vector to distinguish different style emotions, the advanced decoder in such a multi-branch multi-style multi-emotion speech synthesis model is independent for different style emotion information, which can solve the problem that the model tends to be an average model, and retain the characteristics of each style emotion (such as timbre, rhythm and emotional intensity, etc.) without being weakened. In addition, since there is no need to introduce style emotion embedding vectors at different positions of the model, no additional computational overhead is incurred.

[0092] Combine the following Figures 4 to 12 The implementation methods of some steps in the above embodiments are described in detail.

[0093] Figure 4 The flowchart schematically shows the steps of a speech synthesis method provided in another exemplary embodiment of the present disclosure.

[0094] like Figure 4 As shown above Figure 3 The difference of the embodiment is that the above step S340 may further include the following steps S341 to S344:

[0095] Step S341: Determine a target style emotion embedding vector corresponding to the target style emotion information from the K style emotion embedding vectors corresponding to the K style emotion information.

[0096] In the disclosed embodiment, embedding vectors corresponding to different styles of emotions can be pre-extracted as K style emotion embedding vectors, and stored in association with the corresponding style emotions. When the target style emotion information is obtained, the corresponding target style emotion embedding vector can be determined from the stored K style emotion embedding vectors.

[0097] In the embodiment of the present disclosure, data of the same style emotion in the training data can be used to extract a style emotion embedding vector (style embedding), and then the mean of the style embeddings of the same style emotion is taken as the embedding vector of the style emotion, but the present disclosure is not limited to this.

[0098] Step S342: The target encoding feature and the target style emotion embedding vector are subjected to a fusion process to obtain a target fused feature vector.

[0099] In the disclosed embodiment, the target encoding feature obtained by encoding with the shared encoder is fused, for example, concatenated, with the target style emotion embedding vector determined above to obtain a target fused feature vector.

[0100] Step S343: Utilize the target advanced decoder to decode the target fused feature vector to obtain target advanced decoding information of the target text information.

[0101] Step S344: Post-process the target advanced decoding information based on the target style emotion information to obtain target acoustic features corresponding to the target style emotion information.

[0102] The speech synthesis method provided by the embodiment of the present disclosure obtains a target fused feature vector by fusing the target coding features obtained by encoding with the target style emotion embedding vector, thereby determining which style or emotion the target text information corresponds to. In this way, when the target advanced decoder decodes the target fused feature vector, it can obtain the target acoustic features corresponding to the target style emotion information.

[0103] Figure 5 The following schematically shows a flow chart of the steps of a speech synthesis method provided in another exemplary embodiment of the present disclosure. Figure 5 As shown above Figure 4 The difference of the embodiment is that the above step S342 may further include the following steps S3421 to S3424:

[0104] Step S3421: Determine a target duration predictor corresponding to the target style emotion information from the K duration predictors corresponding to the K styles emotion information.

[0105] In the disclosed embodiment, corresponding duration predictors are set for different style emotion information. For example, if there are K types of style emotion information, K duration predictors are set, and each duration predictor is used to predict the duration information of the text corresponding to the style emotion information (which can be the target text information or the training text information). The corresponding duration information is used to upsample the number of frames of the encoding feature (which can be the target encoding feature or the training encoding feature) output by the shared encoder to the number of frames of the corresponding acoustic feature, so that the length of the corresponding text is consistent with the length of the acoustic feature.

[0106] Step S3422: Use the target duration predictor to perform duration prediction processing on the target coding feature to obtain target duration information of the target text information.

[0107] Step S3423. Use the target duration information to upsample the target coding features.

[0108] In the disclosed embodiment, the target duration information predicted and output by the target duration predictor is used to upsample the number of frames of the target coding feature encoded and output by the shared encoder to the number of frames of the target acoustic feature, so that the length of the target text information is consistent with the length of the target acoustic feature.

[0109] Step S3424: The upsampled target encoding feature is fused with the target style emotion embedding vector to obtain the target fused feature vector.

[0110] The speech synthesis method provided by the embodiment of the present disclosure, on the one hand, uses a target duration predictor to predict the target duration information of the target text information, and then uses the target duration information to upsample the target coding features, so that the length of the target text information is consistent with the length of the target acoustic feature; on the other hand, independent duration predictors are set for different styles of emotional information, which can ensure the accuracy of the duration information predicted for different styles of emotional information.

[0111] Figure 6 The following schematically shows a flow chart of the steps of a speech synthesis method provided in another exemplary embodiment of the present disclosure. Figure 6 As shown above Figure 4 The difference of the embodiment is that the above step S343 may further include the following steps S3431 to S3432:

[0112] Step S3431. Use a shared preliminary decoder to perform preliminary decoding processing on the target fused feature vector to obtain target preliminary decoding information of the target text information.

[0113] In the disclosed embodiment, the coding features output by the shared encoder (including target coding features and may also include training coding features) will be decoded at least twice. Therefore, the first decoding process is called preliminary decoding process or preliminary decoding process, and the result obtained by the preliminary decoding process is called preliminary decoding information or preliminary decoding information (including target preliminary decoding information), and the corresponding decoder is called preliminary decoder or preliminary decoder; the second decoding process is called advanced decoding process or secondary decoding process, and the corresponding decoder is called advanced decoder or secondary decoder.

[0114] In the disclosed embodiment, since different styles of emotion information share the same preliminary decoder or primary decoder for primary decoding processing, it can also be called a shared primary decoder.

[0115] Step S3432: Utilize the target advanced decoder to perform advanced decoding processing on the target preliminary decoding information to obtain the target advanced decoding information.

[0116] The speech synthesis method provided in the embodiment of the present disclosure fuses the upsampled target coding feature with the target style emotion embedding vector to obtain a target fused feature vector, performs preliminary decoding processing using a shared primary decoder to obtain target primary decoding information, and inputs the target primary decoding information into a target advanced decoder corresponding to the target style emotion information for corresponding advanced decoding processing to obtain target advanced decoding information, which helps to obtain a stable target primary decoding result for secondary decoding.

[0117] Figure 7 The following schematically shows a flow chart of the steps of a speech synthesis method provided in another exemplary embodiment of the present disclosure. Figure 7 As shown above Figure 4 The difference of the embodiment is that the above step S344 may further include the following steps S3441 to S3442:

[0118] Step S3441: Determine a target post-processing module corresponding to the target style emotion information from the K post-processing modules corresponding to the K types of style emotion information.

[0119] In the disclosed embodiment, during the training phase, K post-processing modules may be trained corresponding to K types of style emotion information, that is, the post-processing modules in the multi-branch multi-style multi-emotion speech synthesis model may also be multi-branched.

[0120] Step S3442: Utilize the target post-processing module to post-process the target advanced decoding information to obtain the target acoustic features.

[0121] The post-processing in the embodiments of the present disclosure may include processing such as audio compression.

[0122] The speech synthesis method provided by the embodiment of the present disclosure, on the one hand, uses the target post-processing module corresponding to the target style emotion information to post-process the target advanced decoding information, which can be beneficial to retain the timbre characteristics and rhythmic information of different styles of emotions; on the other hand, during the use stage, it is only necessary to follow the branch of the target post-processing module corresponding to the target style emotion information, which can reduce the amount of calculation.

[0123] Figure 8 The following schematically shows a flow chart of the steps of a speech synthesis method provided in another exemplary embodiment of the present disclosure. Figure 8 As shown above Figure 3 The difference of the embodiment is that the K types of style emotion information may include first style emotion information and second style emotion information, for example, the first style emotion information is a narration style, and the second style emotion information is an emotion of joy, anger, sadness, happiness, etc. Figure 8 The embodiment uses two different style emotion information as an example to illustrate the training process of the multi-branch multi-style multi-emotion speech synthesis model provided by the embodiment of the present disclosure, but the present disclosure is not limited to this. In the actual training process, training can be performed according to the required number of style emotion types.

[0124] In the disclosed embodiment, several styles and emotions can be predefined, such as narration style, dialogue style, happiness, anger, sadness, joy, etc., and then the training data of the corresponding styles and emotions can be recorded and the corresponding model can be trained. Since the training data is labeled with the corresponding style and emotion, different paths are taken according to the label during training, and the corresponding path can be taken as needed during online use, thereby determining the target advanced decoder and the target post-processing module.

[0125] Figure 8 The method provided in the embodiment may further include the following steps S810 to S860:

[0126] Step S810: Acquire first training text information and first training audio information with first style emotion information, and second training text information and second training audio information with second style emotion information.

[0127] The first training audio information with the first style emotion information can be used as a label of the first training text information, and the second training audio information with the second style emotion information can be used as a label of the second training text information.

[0128] Step S820: Use the shared encoder to perform text encoding processing on the first training text information and the second training text information respectively to obtain a first training encoding feature of the first training text information and a second training encoding feature of the second training text information.

[0129] In the disclosed embodiment, the first training text information may be input into a shared encoder to output a first training coding feature; the second training text information may also be input into the shared encoder to output the second training coding feature.

[0130] Step S830: Determine a first advanced decoder corresponding to the first style emotion information from the K advanced decoders, and determine a second advanced decoder corresponding to the second style emotion information from the K advanced decoders.

[0131] Step S840. Use the first advanced decoder to decode the first training coding feature to obtain a first training acoustic feature corresponding to the first style emotion information, and use the second advanced decoder to decode the second training coding feature to obtain a second training acoustic feature corresponding to the second style emotion information.

[0132] In the disclosed embodiment, K advanced decoders correspond one-to-one to K style emotion information. For example, assume that the first advanced decoder is used for secondary decoding processing of the first training coding feature corresponding to the first style emotion information to obtain the first training acoustic feature; the second advanced decoder is used for secondary decoding processing of the second training coding feature corresponding to the second style emotion information to obtain the second training acoustic feature.

[0133] Step S850. Perform acoustic encoding processing on the first training acoustic features to obtain first predicted audio information corresponding to the first training text information and the first style and emotion information, and perform acoustic encoding processing on the second training acoustic features to obtain second predicted audio information corresponding to the second training text information and the second style and emotion information.

[0134] In the disclosed embodiment, the first training acoustic feature is processed by a vocoder to obtain the first predicted audio information, and the second training acoustic feature is processed by a vocoder to obtain the second predicted audio information.

[0135] Step S860: Train the shared encoder, the first advanced decoder and the second advanced decoder based on the first training audio information and the first predicted audio information, and the second training audio information and the second predicted audio information.

[0136] In the disclosed embodiment, a loss function, such as a cross entropy loss function, an MSE (Mean Square Error) loss function, etc., can be constructed using the first predicted audio information and the first training audio information, the second predicted audio information and the second training audio information to train the shared encoder and the first advanced decoder and the second advanced decoder to obtain trained shared encoders, first advanced decoders and second advanced decoders. The trained shared encoder can be used to perform text encoding processing on the target text information to obtain target encoding features; the trained first advanced decoder and / or second advanced decoder can be used as a target advanced decoder to perform secondary decoding processing on the target encoding features to obtain target acoustic features. On the one hand, by adopting a shared encoder, training text information (including the first training text information and the second training text information) using different styles of emotion information (including the first style emotion information and the second style emotion information mentioned as an example) all participate in the training of the shared encoder, and categories with less training data (such as emotions such as joy, anger, sadness, and happiness) can share the benefits of wide coverage and relatively stable recording of training text information of categories with large training data (such as narration style), so that the prediction performance of the multi-branch multi-style multi-emotion model obtained through training is more accurate and stable; on the other hand, since different styles of emotions have independent advanced decoders (including the first advanced decoder and the second advanced decoder mentioned as an example), it can ensure that the timbre characteristics and rhythmic information of different styles of emotions are retained.

[0137] Fig. 9 The schematic diagram of the speech synthesis method provided in an exemplary embodiment of the present disclosure is schematically shown. The core of the present invention "a multi-branch multi-style emotional speech synthesis system" is divided into two parts, and its flow chart is as follows Figure 2 As shown:

[0138] like Fig. 9 As described above, the multi-branch multi-style multi-emotion speech synthesis model provided by the embodiment of the present disclosure may include a first part and a second part.

[0139] The first part is the fusion sharing part, which may include an encoder 910 and a preliminary decoder 930. Since different styles of emotional information share one encoder 910 and preliminary decoder 930, it can also be called a shared encoder 910 and a shared preliminary decoder 930 / shared preliminary decoder.

[0140] The second part is a multi-branch part, and different styles of emotions have independent duration predictors, secondary decoders, and post-processing modules. For example, style A has an independent duration predictor A, secondary decoder A, and post-processing module A, style B has an independent duration predictor B, secondary decoder B, and post-processing module B, emotion M has an independent duration predictor M, secondary decoder M, and post-processing module M, and emotion N has an independent duration predictor N, secondary decoder N, and post-processing module N. For example, if style A is used as the first style emotion information and emotion N is used as the second style emotion information, duration predictor A can be used as the first duration predictor, duration predictor N can be used as the second duration predictor, secondary decoder A can be used as the first advanced decoder, secondary decoder N can be used as the second advanced decoder, post-processing module A can be used as the first post-processing module, and post-processing module N can be used as the second post-processing module.

[0141] During training, the training text information corresponding to different styles of emotion information is respectively input into the encoder 910 to obtain the hidden coding representation of the corresponding style of emotion information as the training coding feature of the corresponding style of emotion information. For example, the first training text information corresponding to style A is input into the encoder 910, and the hidden coding A is output as the first training coding feature. The training text information corresponding to style B is input into the encoder 910, and the hidden coding B is output. The training text information corresponding to emotion M is input into the encoder 910, and the hidden coding M is output. The second training text information corresponding to emotion N is input into the encoder 910, and the hidden coding N is output as the second training coding feature.

[0142] Then, the hidden code representations corresponding to the different styles of emotion information are input into the duration predictors corresponding to the different styles of emotion information to obtain the corresponding duration information. For example, the hidden code A corresponding to style A is input into duration predictor A to obtain the first duration information. The hidden code B corresponding to style B is input into duration predictor B to obtain the duration information corresponding to style B. The hidden code M corresponding to emotion M is input into duration predictor M to obtain the duration information corresponding to emotion M. The hidden code N corresponding to emotion N is input into duration predictor N to obtain the second duration information corresponding to emotion N.

[0143] The corresponding style extractor 920 can be used to extract the corresponding style emotion embedding vectors of different style emotions. For example, the style extractor 920 corresponding to style A is used to process the hidden code A, and extract the style emotion embedding vector of style A as the first style emotion embedding vector; the style extractor 920 corresponding to style B is used to process the hidden code B, and extract the style emotion embedding vector B of style B; the style extractor 920 corresponding to emotion M is used to process the hidden code M, and extract the style emotion embedding vector M of emotion M; the style extractor 920 corresponding to emotion N is used to process the hidden code N, and extract the style emotion embedding vector of emotion N as the second style emotion embedding vector.

[0144] Then, the number of frames of the hidden code representation of the corresponding style emotion information is upsampled to the number of frames of the corresponding acoustic feature by using the duration information corresponding to the different style emotion information, so that the length of the corresponding training text information is consistent with the length of the corresponding acoustic feature. For example, the number of frames of the hidden code A is upsampled to the number of frames of the first training acoustic feature (acoustic feature A) using the first duration information, the number of frames of the hidden code B is upsampled to the number of frames of the acoustic feature B using the duration information corresponding to the style B, the number of frames of the hidden code M is upsampled to the number of frames of the acoustic feature M using the duration information corresponding to the emotion M, and the number of frames of the hidden code N is upsampled to the number of frames of the acoustic feature N (the second acoustic feature) using the second duration information.

[0145] Afterwards, the upsampled hidden code representation and the corresponding style-emotion embedding vector are fused. For example, each upsampled hidden code A is vector-summed with the style-emotion embedding vector of style A (first style-emotion embedding vector) to obtain a first fused feature vector, which is used to indicate that the first training text information corresponds to the first style-emotion information. Each upsampled hidden code B is vector-summed with the style-emotion embedding vector of style B to obtain a fused feature vector of style B, which is used to indicate that the training text information of style B corresponds to style B. Each upsampled hidden code M is vector-summed with the style-emotion embedding vector of emotion M to obtain a fused feature vector of emotion M, which is used to indicate that the training text information of emotion M corresponds to emotion M. Each upsampled hidden code N is vector-summed with the style-emotion embedding vector of emotion N (second style-emotion embedding vector) to obtain a second fused feature vector, which is used to indicate that the second training text information corresponds to the second style-emotion information.

[0146] After that, the upsampled hidden coding representation and the corresponding style emotion embedding vector are fused and input into the preliminary decoder 930 to obtain the corresponding preliminary decoding result. For example, the first fused feature vector is input into the preliminary decoder 930, and each hidden decoding A is output as the first preliminary decoding information; the fused feature vector of style B is input into the preliminary decoder 930, and each hidden decoding B is output; the fused feature vector of emotion M is input into the preliminary decoder 930, and each hidden decoding M is output; the second fused feature vector is input into the preliminary decoder 930, and each hidden decoding N is output as the second preliminary decoding information.

[0147] Then, each preliminary decoding result is input into the secondary decoder and post-processing module corresponding to different styles and emotions to obtain the corresponding acoustic features (also referred to as training acoustic features). For example, each hidden code A is input into the secondary decoder A (first advanced decoder) to obtain the first advanced decoding information, and the first advanced decoding information is input into the post-processing module A (first post-processing module) to obtain the acoustic feature A (first training acoustic feature). Each hidden code N is input into the secondary decoder N (second advanced decoder) to obtain the second advanced decoding information, and the second advanced decoding information is input into the post-processing module N (second post-processing module) to obtain the acoustic feature N (second training acoustic feature).

[0148] When testing or using online, the target text information to be synthesized is input into the encoder 910, the encoding result is used as the target encoding feature, and the target encoding feature is input into the target duration predictor corresponding to the target style emotion information to obtain the target duration information. The target encoding feature is upsampled with the target duration information and fused with the corresponding target emotion style embedding vector and then input into the preliminary decoder 930 to obtain the target primary decoding information. The target primary decoding information is input into the secondary decoder (as the target advanced decoder) corresponding to the required target style emotion information for processing to obtain the target advanced decoding information, and the target advanced decoding information is then input into the post-processing module (as the target post-processing module) corresponding to the required target style emotion information to obtain the target acoustic feature corresponding to the required target style emotion information. That is to say, in actual use, it is only necessary to pass through the branches corresponding to the fusion shared part and the required target style emotion information, and the remaining branches do not need to be calculated.

[0149] The preliminary decoder and each secondary decoder in the embodiment of the present disclosure may have the same structure or different structures. In the following examples, examples with the same structure are used for illustration. For example, both the preliminary decoder and the secondary decoder may use a non-autoregressive single layer RNN as a decoder. Non-autoregressive here means that the decoder does not rely on the predicted or real acoustic features of the autoregressive feedback. In this way, without introducing auxiliary strategies such as scheduled sampling and maximum mutual information, the effects of exposure bias and local information preference can be eliminated. In addition, the decoding efficiency can be significantly improved.

[0150] In the disclosed embodiment, the duration predictor is used to connect the encoder and the non-autoregressive RNN decoder, which can avoid the instability caused by the attention model. Non-attentive means that the encoder and the decoder are not connected through the attention model. This non-attentive non-autoregressive hybrid model has the advantage of giving full play to the advantages of each component.

[0151] The multi-branch multi-style multi-emotion speech synthesis model provided by the embodiment of the present disclosure includes a shared encoder and a shared preliminary decoder shared by different styles of emotions, as well as independent duration predictors, secondary decoders and post-processing modules for different styles of emotions. This structure, which includes a fusion shared part and a multi-branch part, can use the training text information and training audio information of multiple styles and emotions of different numbers of speakers to train the model, so that different training text information of different styles of emotional information can all participate in the training of the shared encoder, which can fully utilize the characteristics of wide coverage and high recording quality of the training text information of the categories with richer training data, and help improve the synthesis accuracy and stability of the categories with less training data. At the same time, the shared preliminary decoder helps to obtain a stable preliminary decoding result for secondary decoding. On the other hand, the timbre and rhythmic information of various styles of emotions can be retained. In addition, in actual use, it is only necessary to follow the branch corresponding to the required target style emotional information without introducing redundant calculations.

[0152] The encoder in the disclosed embodiment can be a model obtained by training various types of neural networks such as Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM) or Gate Recurrent Unit (GRU). CNN is a feedforward neural network whose neurons can respond to units within the receptive field; CNN is usually composed of multiple convolutional layers and a fully connected layer at the top, which reduces the number of parameters of the model by sharing parameters, making it widely used in image and speech recognition. RNN is a type of recursive neural network that takes sequence data as input, performs recursion in the direction of sequence evolution, and all nodes (recurrent units) are connected in a chain. LSTM is a recurrent neural network. It adds a unit to the algorithm to determine whether the information is useful or not. An input gate, a forget gate, and an output gate are placed in one unit. After the information enters the LSTM, it is judged whether it is useful based on the rules. Only information that meets the algorithm certification will be retained, and information that does not meet the certification will be forgotten through the forget gate. LSTM is suitable for processing and predicting important events with relatively long intervals and delays in time series. GRU is a type of recurrent neural network. Like LSTM, GRU is also proposed to solve problems such as long-term memory and gradients in back propagation. Compared with LSTM, GRU has one less "gate" inside and fewer parameters than LSTM. In most cases, it can achieve the same effect as LSTM and effectively reduce calculation time. The following is Fig.10 The structure of the encoder shown is for illustrative purposes only, but the present disclosure is not limited thereto.

[0153] like Fig.10 As shown, the encoder 910 may include an embedding layer 911, a convolutional processor 912, a text feature extractor 913, and a multi-head attention module 914.

[0154] The text extractor 913 may further include at least one transformer block, such as a transformer block 9131-1, a transformer block 9131-2, a transformer block 9131-3, and a transformer block 9131-4.

[0155] Taking the data input to the encoder 910 (such as target text information, training text information) as Chinese as an example, the data input to the encoder 910 can be first converted into a pinyin sequence (sequence), which contains tone and prosody information, and then converted into a corresponding embedding sequence (embeddingsequence) after being processed by the embedding layer 911. The convolution processor 912 can have multiple 1-dimensional convolution layers (Conv-1D layers) for processing the embedding sequence. Then, the output of the convolution processor 912 is input to the text extractor 913, which includes a stacked deformer module.

[0156] like Fig.11 As shown, each deformer module can further include a multi-head attention unit 91311, an anti-overfitting (Dropout) unit 91312, a normalization (Normalize) unit 91313, a convolution (convolution, which can be expressed as Conv-1Dx2) layer 91314, an anti-overfitting (Dropout) unit 91315 and a normalization (Normalize) unit 91316. The input information of the deformer module is processed by the Multi-head attention unit 91311, and the output information of the Multi-head attention unit 91311 is input to the anti-overfitting (Dropout) unit 91312. The output information of the anti-overfitting (Dropout) unit 91312 is residual-added with the input information of the deformer module, and the result of the residual summation is then processed by the layer normalization of the normalization unit 91313. The output information of the normalization unit 91313 is processed by the convolution layer 91314, and the output information of the convolution layer 91314 is input to the anti-overfitting (Dropout) unit 91315. The output information of the anti-overfitting (Dropout) unit 91315 is residual-added with the output information of the normalization unit 91313, and the result of the residual summation is then processed by the layer normalization of the normalization unit 91316.

[0157] Fig.10In the embodiment, multiple deformer modules are densely connected through bypass connections. For example, the input information of the deformer module 9131-1 and the output information of the deformer module 9131-1 are summed as the input information of the deformer module 9131-2, the input information of the deformer module 9131-2 and the output information of the deformer module 9131-2 are summed as the input information of the deformer module 9131-3, and the input information of the deformer module 9131-3 and the output information of the deformer module 9131-3 are summed as the input information of the deformer module 9131-4. This design allows multiple deformer modules to be "densely" interconnected, that is, not just the connection between layers and adjacent layers, but for each deformer module, its input information is related to the input information of all the previous deformer modules, and its own output information is also used as the input information of all subsequent deformer modules. This dense connection method can alleviate the problem of gradient disappearance, enhance feature propagation, encourage feature reuse, and reduce the number of parameters. From the perspective of backpropagation, the supervision signal from the top layer can be better transmitted back to the lower layer, making the model easier to train. From the perspective of feature fusion, dense connections enable features of lower layers to be reused by higher layers. Because the features extracted from the transformer modules at different layers of the encoder have different representation meanings, the reuse of these features can enhance the final encoding ability. The use of element-wise addition in feature fusion can keep the encoding dimensions between different transformer modules unchanged, thereby maintaining the computational complexity. In this way, features from different layers are processed indiscriminately, which can be regarded as a coarse fuse. This is a relatively rough fusion of the features learned by the transformer modules through dense bypass connections, because the model cannot control the proportion of these features during the fusion process. Fig.10 Q is used to represent the residual sum of the output information of the transformer module 9131-4 and the input information of the transformer module 9131-4, and Q is input to the multi-head attention module 914.

[0158] In the embodiment of the present disclosure, a multi-head attention module 914 is introduced to perform fine fusion. Fig.10The coarse fusion feature is represented by Q, and the output information of the deformer module 9131-1, the deformer module 9131-2, the deformer module 9131-3, and the deformer module 9131-4 are also input to the multi-head attention module 914 respectively. Fig.10 In the example, the output information of the deformer module 9131-1, the deformer module 9131-2, the deformer module 9131-3, and the deformer module 9131-4 are used as the key of attention, and the corresponding attention head is called value, so it can be abbreviated as K&V. The final hidden code (hidden text representation) is obtained by combining the output information of different deformer modules and the weights learned through the multi-head attention layer. Fine fusion can be regarded as a further adjustment of the model to the features in coarse fuse. The model can use fine fusion to determine how much additional information the coarse fused features need to obtain from each deformer module. Fig.10 The encoder 910 in the embodiment can extract text encoding with strong representation capabilities, which is beneficial to improving the naturalness and robustness of the synthesized speech.

[0159] Fig.12 The schematic diagram shows the structure of a duration predictor provided in an exemplary embodiment of the present disclosure.

[0160] like Fig.12 As shown, Fig. 9 Any duration predictor 1200 in the embodiment may include a 1D-convolution network 1210, a bidirectional GRU 1220, and a linear unit 1230 to process the hidden code output by the encoder 910 and output corresponding duration information.

[0161] The 1D-convolution network 1210 may further include Conv-1D&BN 1211, Conv-1D&BN1212, and Conv-1D&BN 1213, where BN is short for Batch Normalization, which is used to keep the input of each layer of the neural network in the same distribution during the deep neural network training process. The 1D-convolution network 1210 may be used to capture local position-related information in hidden codes. Considering that the duration information is related to the global context information, a bidirectional RNN layer is added to the convolution network, such as a bidirectional GRU 1220, and the output of the bidirectional GRU 1220 is linearly mapped by a linear unit 1230 to become a scalar.

[0162] During the training phase, the gradient of the duration predictor is prevented from backpropagating to the encoder to avoid affecting the learning of the encoder.

[0163] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0164] The following introduces an apparatus embodiment of the present disclosure, which can be used to execute the speech synthesis method in the above-mentioned embodiment of the present disclosure. Fig.13 The block diagram of the speech synthesis device in the embodiment of the present disclosure is schematically shown. Fig.13 As shown, the speech synthesis device 1300 may mainly include: an acquisition unit 1310 and a processing unit 1320 .

[0165] The acquisition unit 1310 may be used to acquire target text information and target style emotion information thereof.

[0166] The processing unit 1320 may be configured to perform text encoding processing on the target text information using a shared encoder to obtain target encoding features of the target text information.

[0167] The processing unit 1320 may also be configured to determine a target advanced decoder corresponding to the target style emotion information from K advanced decoders corresponding to K styles emotion information, where K may be an integer greater than 1, and the K styles emotion information include the target style emotion information.

[0168] The processing unit 1320 may also be configured to use the target advanced decoder to decode the target coding feature to obtain a target acoustic feature corresponding to the target style emotion information.

[0169] The processing unit 1320 may also be configured to perform acoustic coding processing on the target acoustic feature to obtain target audio information corresponding to the target text information and the target style emotion information.

[0170] In an exemplary embodiment, the processing unit 1320 can also be used to: determine a target style emotion embedding vector corresponding to the target style emotion information from K styles of emotion embedding vectors corresponding to K styles of emotion information; fuse the target coding feature and the target style emotion embedding vector to obtain a target fused feature vector; use the target advanced decoder to decode the target fused feature vector to obtain target advanced decoding information of the target text information; post-process the target advanced decoding information based on the target style emotion information to obtain the target acoustic feature corresponding to the target style emotion information.

[0171] In an exemplary embodiment, the processing unit 1320 can also be used to: determine a target duration predictor corresponding to the target style emotion information from K duration predictors corresponding to K styles of emotion information; use the target duration predictor to perform duration prediction processing on the target coding feature to obtain the target duration information of the target text information; use the target duration information to upsample the target coding feature; and fuse the upsampled target coding feature with the target style emotion embedding vector to obtain the target fused feature vector.

[0172] In an exemplary embodiment, the processing unit 1320 can also be used to: use a shared preliminary decoder to perform preliminary decoding processing on the target fusion feature vector to obtain target preliminary decoding information of the target text information; use the target advanced decoder to perform advanced decoding processing on the target preliminary decoding information to obtain the target advanced decoding information.

[0173] In an exemplary embodiment, the processing unit 1320 can also be used to: determine a target post-processing module corresponding to the target style emotion information from K post-processing modules corresponding to K types of style emotion information respectively; and use the target post-processing module to post-process the target advanced decoding information to obtain the target acoustic feature.

[0174] In an exemplary embodiment, the K types of style emotion information include first style emotion information and second style emotion information. The acquisition unit 1310 may also be used to acquire first training text information and first training audio information with the first style emotion information, and second training text information and second training audio information with the second style emotion information. The processing unit 1320 can also be used to use the shared encoder to perform text encoding processing on the first training text information and the second training text information respectively to obtain a first training encoding feature of the first training text information and a second training encoding feature of the second training text information; determine a first advanced decoder corresponding to the first style emotion information from K advanced decoders, and determine a second advanced decoder corresponding to the second style emotion information from K advanced decoders; use the first advanced decoder to decode the first training encoding feature to obtain a first training acoustic feature corresponding to the first style emotion information, and use the second advanced decoder to decode the second training encoding feature to obtain a second training acoustic feature corresponding to the second style emotion information; perform acoustic encoding processing on the first training acoustic feature to obtain a first predicted audio information corresponding to the first training text information and the first style emotion information, and perform acoustic encoding processing on the second training acoustic feature to obtain a second predicted audio information corresponding to the second training text information and the second style emotion information; train the shared encoder, the first advanced decoder and the second advanced decoder based on the first training audio information and the first predicted audio information, and the second training audio information and the second predicted audio information.

[0175] In an exemplary embodiment, K types of style emotion information may correspond to K types of style emotion embedding vectors, and the K types of style emotion embedding vectors may include a first style emotion embedding vector corresponding to the first style emotion information and a second style emotion embedding vector corresponding to the second style emotion information. The processing unit 1320 may also be used to: subject the first training coding feature and the first style emotion embedding vector corresponding to the first style emotion information to a fusion process to obtain a first fusion feature vector, and subject the second training coding feature and the second style emotion embedding vector corresponding to the second style emotion information to a fusion process to obtain a second fusion feature vector; use the first advanced decoder to decode the first fusion feature vector to obtain the first advanced decoding information of the first training text information, and use the second advanced decoder to decode the second fusion feature vector to obtain the second advanced decoding information of the second training text information; post-process the first advanced decoding information based on the first style emotion information to obtain the first training acoustic feature, and post-process the second advanced decoding information based on the second style emotion information to obtain the second training acoustic feature.

[0176] In an exemplary embodiment, the processing unit 1320 can also be used to: determine a first duration predictor corresponding to the first style emotion information from K duration predictors corresponding to K styles of emotion information, and determine a second duration predictor corresponding to the second style emotion information from the K duration predictors; use the first duration predictor to perform duration prediction processing on the first training coding feature to obtain the first duration information of the first training text information, and use the second duration predictor to perform duration prediction processing on the second training coding feature to obtain the second duration information of the second training text information; use the first duration information to upsample the first training coding feature, and use the second duration information to upsample the second training coding feature; fuse the upsampled first training coding feature with the first style emotion embedding vector to obtain the first fused feature vector, and fuse the upsampled second training coding feature with the second style emotion embedding vector to obtain the second fused feature vector.

[0177] In an exemplary embodiment, the processing unit 1320 can also be used to: use a shared preliminary decoder to perform preliminary decoding processing on the first fused feature vector and the second fused feature vector, respectively, to obtain first preliminary decoding information of the first training text information and second preliminary decoding information of the second training text information; use the first advanced decoder to perform advanced decoding processing on the first preliminary decoding information to obtain the first advanced decoding information, and use the second advanced decoder to perform advanced decoding processing on the second preliminary decoding information to obtain the second advanced decoding information.

[0178] In an exemplary embodiment, the processing unit 1320 can also be used to: determine a first post-processing module corresponding to the first style emotion information from K post-processing modules corresponding to K types of style emotion information, and determine a second post-processing module corresponding to the second style emotion information from the K post-processing modules; use the first post-processing module to post-process the first advanced decoding information to obtain the first training acoustic feature, and use the second post-processing module to post-process the second advanced decoding information to obtain the second training acoustic feature.

[0179] The specific details of the speech synthesis device provided in each embodiment of the present disclosure have been described in detail in the corresponding method embodiment and will not be repeated here.

[0180] Fig.14 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present disclosure is shown.

[0181] It should be noted that Fig.14 The computer system 1400 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0182] like Fig.14 As shown, the computer system 1400 includes a central processing unit (CPU) 1401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1402 or the program loaded from the storage part 1408 to the random access memory (RAM) 1403. Various programs and data required for system operation are also stored in the RAM 1403. The CPU 1401, the ROM 1402 and the RAM 1403 are connected to each other through a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0183] The following components are connected to the I / O interface 1405: an input section 1406 including a keyboard, a mouse, etc.; an output section 1407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1408 including a hard disk, etc.; and a communication section 1409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the I / O interface 1405 as needed. A removable medium 1411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1410 as needed so that a computer program read therefrom is installed into the storage section 1408 as needed.

[0184] In particular, according to an embodiment of the present disclosure, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1409, and / or installed from a removable medium 1411. When the computer program is executed by a central processing unit (CPU) 1401, various functions defined in the system of the present disclosure are executed.

[0185] It should be noted that the computer-readable medium shown in the embodiment of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0186] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0187] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0188] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0189] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0190] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A speech synthesis method, characterized in that: include: Obtain target text information and target style emotion information; Using a shared encoder to perform text encoding processing on the target text information to obtain target encoding features of the target text information, the shared encoder refers to an encoder used for text encoding processing of target text information with a plurality of different target style emotion information, and in the training stage, the shared encoder is an encoder trained using K different styles of emotion information and their corresponding training text information and training audio information; Determining a target advanced decoder corresponding to the target style emotion information from K advanced decoders corresponding to K styles of emotion information, where K is an integer greater than 1, and the K styles of emotion information include the target style emotion information; Using the target advanced decoder to decode the target coding feature to obtain a target acoustic feature corresponding to the target style emotion information; Performing acoustic coding processing on the target acoustic features to obtain target audio information corresponding to the target text information and the target style emotion information; The target advanced decoder is used to decode the target coding feature to obtain a target acoustic feature corresponding to the target style emotion information, including: Determine a target style emotion embedding vector corresponding to the target style emotion information from the K style emotion embedding vectors corresponding to the K style emotion information; The target encoding feature and the target style emotion embedding vector are subjected to fusion processing to obtain a target fusion feature vector; Using the target advanced decoder to decode the target fused feature vector to obtain target advanced decoding information of the target text information; The target advanced decoded information is post-processed based on the target style emotion information to obtain a target acoustic feature corresponding to the target style emotion information.

2. The speech synthesis method according to claim 1, characterized in that: The target encoding feature and the target style emotion embedding vector are subjected to fusion processing to obtain a target fusion feature vector, including: Determine a target duration predictor corresponding to the target style emotion information from the K duration predictors corresponding to the K styles emotion information; Using the target duration predictor to perform duration prediction processing on the target coding feature to obtain target duration information of the target text information; Upsampling the target coding feature using the target duration information; The upsampled target encoding feature is fused with the target style emotion embedding vector to obtain the target fused feature vector.

3. The speech synthesis method according to claim 1, characterized in that: Using the target advanced decoder to decode the target fused feature vector to obtain target advanced decoding information of the target text information includes: Using a shared preliminary decoder to perform preliminary decoding processing on the target fused feature vector to obtain target preliminary decoding information of the target text information; The target advanced decoder is used to perform advanced decoding processing on the target preliminary decoding information to obtain the target advanced decoding information.

4. The speech synthesis method according to claim 1, characterized in that: Post-processing the target advanced decoding information based on the target style emotion information to obtain a target acoustic feature corresponding to the target style emotion information includes: Determine a target post-processing module corresponding to the target style emotion information from the K post-processing modules corresponding to the K styles emotion information respectively; The target post-processing module is used to post-process the target advanced decoding information to obtain the target acoustic feature.

5. The speech synthesis method according to claim 1, characterized in that: The K types of style emotion information include first style emotion information and second style emotion information; wherein the method further includes: Acquire first training text information and first training audio information with first style emotion information, and second training text information and second training audio information with second style emotion information; Using the shared encoder to perform text encoding processing on the first training text information and the second training text information respectively, so as to obtain a first training encoding feature of the first training text information and a second training encoding feature of the second training text information; Determine a first advanced decoder corresponding to the first style emotion information from the K advanced decoders, and determine a second advanced decoder corresponding to the second style emotion information from the K advanced decoders; Decoding the first training coding feature using the first advanced decoder to obtain a first training acoustic feature corresponding to the first style emotion information, and decoding the second training coding feature using the second advanced decoder to obtain a second training acoustic feature corresponding to the second style emotion information; Performing acoustic coding processing on the first training acoustic features to obtain first predicted audio information corresponding to the first training text information and the first style and emotion information, and performing acoustic coding processing on the second training acoustic features to obtain second predicted audio information corresponding to the second training text information and the second style and emotion information; The shared encoder, the first advanced decoder, and the second advanced decoder are trained based on the first training audio information and the first predicted audio information, and the second training audio information and the second predicted audio information.

6. The speech synthesis method according to claim 5, characterized in that: K types of style emotion information correspond to K types of style emotion embedding vectors, where the K types of style emotion embedding vectors include a first style emotion embedding vector corresponding to the first style emotion information and a second style emotion embedding vector corresponding to the second style emotion information; The method further comprises: decoding the first training coding feature using the first advanced decoder to obtain a first training acoustic feature corresponding to the first style emotion information, and decoding the second training coding feature using the second advanced decoder to obtain a second training acoustic feature corresponding to the second style emotion information, including: The first training coding feature and the first style emotion embedding vector corresponding to the first style emotion information are subjected to fusion processing to obtain a first fused feature vector, and the second training coding feature and the second style emotion embedding vector corresponding to the second style emotion information are subjected to fusion processing to obtain a second fused feature vector; Using the first advanced decoder to decode the first fused feature vector to obtain first advanced decoding information of the first training text information, and using the second advanced decoder to decode the second fused feature vector to obtain second advanced decoding information of the second training text information; The first advanced decoded information is post-processed based on the first style emotion information to obtain the first training acoustic feature, and the second advanced decoded information is post-processed based on the second style emotion information to obtain the second training acoustic feature.

7. The speech synthesis method according to claim 6, characterized in that: The first training coding feature and the first style emotion embedding vector corresponding to the first style emotion information are subjected to fusion processing to obtain a first fused feature vector, and the second training coding feature and the second style emotion embedding vector corresponding to the second style emotion information are subjected to fusion processing to obtain a second fused feature vector, including: Determine a first duration predictor corresponding to the first style emotion information from the K duration predictors corresponding to the K types of style emotion information, and determine a second duration predictor corresponding to the second style emotion information from the K duration predictors; Performing duration prediction processing on the first training coding feature using the first duration predictor to obtain first duration information of the first training text information, and performing duration prediction processing on the second training coding feature using the second duration predictor to obtain second duration information of the second training text information; Upsampling the first training coding feature using the first duration information, and upsampling the second training coding feature using the second duration information; The first upsampled training coding feature is fused with the first style emotion embedding vector to obtain the first fused feature vector, and the second upsampled training coding feature is fused with the second style emotion embedding vector to obtain the second fused feature vector.

8. The speech synthesis method according to claim 6, characterized in that: Using the first advanced decoder to decode the first fused feature vector to obtain first advanced decoding information of the first training text information, and using the second advanced decoder to decode the second fused feature vector to obtain second advanced decoding information of the second training text information, including: Using a shared preliminary decoder, respectively, perform preliminary decoding processing on the first fused feature vector and the second fused feature vector to obtain first preliminary decoding information of the first training text information and second preliminary decoding information of the second training text information; The first advanced decoder is used to perform advanced decoding processing on the first preliminary decoded information to obtain the first advanced decoded information, and the second advanced decoder is used to perform advanced decoding processing on the second preliminary decoded information to obtain the second advanced decoded information.

9. The speech synthesis method according to claim 6, characterized in that: Post-processing the first advanced decoded information based on the first style emotion information to obtain the first training acoustic feature, and post-processing the second advanced decoded information based on the second style emotion information to obtain the second training acoustic feature, including: Determine a first post-processing module corresponding to the first style emotion information from the K post-processing modules corresponding to the K types of style emotion information, and determine a second post-processing module corresponding to the second style emotion information from the K post-processing modules; The first advanced decoded information is post-processed by the first post-processing module to obtain the first training acoustic feature, and the second advanced decoded information is post-processed by the second post-processing module to obtain the second training acoustic feature.

10. A speech synthesis device, characterized in that: include: An acquisition unit, used to acquire target text information and target style emotion information; A processing unit, configured to perform text encoding processing on the target text information using a shared encoder to obtain target encoding features of the target text information, wherein the shared encoder refers to an encoder used for performing text encoding processing on target text information having a plurality of target style emotion information, and in a training phase, the shared encoder is an encoder trained using K types of different style emotion information and their corresponding training text information and training audio information; The processing unit is further used to determine a target advanced decoder corresponding to the target style emotion information from K advanced decoders corresponding to K types of style emotion information, K is an integer greater than 1, and the K types of style emotion information include the target style emotion information; The processing unit is further used to decode the target coding feature using the target advanced decoder to obtain a target acoustic feature corresponding to the target style emotion information; The processing unit is further used to perform acoustic coding processing on the target acoustic feature to obtain target audio information corresponding to the target text information and the target style emotion information; The processing unit is further used to: determine a target style emotion embedding vector corresponding to the target style emotion information from the K style emotion embedding vectors corresponding to the K style emotion information; perform fusion processing on the target coding feature and the target style emotion embedding vector to obtain a target fusion feature vector; Using the target advanced decoder to decode the target fused feature vector to obtain target advanced decoding information of the target text information; The target advanced decoded information is post-processed based on the target style emotion information to obtain a target acoustic feature corresponding to the target style emotion information.

11. The speech synthesis device according to claim 10, characterized in that: The processing unit is further used to: determine a target duration predictor corresponding to the target style emotion information from the K duration predictors corresponding to the K styles emotion information; perform duration prediction processing on the target coding feature using the target duration predictor to obtain the target duration information of the target text information; and upsample the target coding feature using the target duration information; The upsampled target encoding feature is fused with the target style emotion embedding vector to obtain the target fused feature vector.

12. The speech synthesis device according to claim 10, characterized in that: The processing unit is also used to: use the shared preliminary decoder to perform preliminary decoding processing on the target fusion feature vector to obtain target preliminary decoding information of the target text information; use the target advanced decoder to perform advanced decoding processing on the target preliminary decoding information to obtain the target advanced decoding information.

13. The speech synthesis device according to claim 10, characterized in that: The processing unit is also used to: determine a target post-processing module corresponding to the target style emotion information from K post-processing modules corresponding to K styles of emotion information respectively; and use the target post-processing module to post-process the target advanced decoding information to obtain the target acoustic feature.

14. The speech synthesis device according to claim 10, characterized in that: The K types of style emotion information include first style emotion information and second style emotion information; The acquisition unit is further used to: acquire first training text information and first training audio information with first style emotion information, and second training text information and second training audio information with second style emotion information; The processing unit is also used to: use the shared encoder to perform text encoding processing on the first training text information and the second training text information respectively to obtain a first training encoding feature of the first training text information and a second training encoding feature of the second training text information; determine a first advanced decoder corresponding to the first style emotion information from K advanced decoders, and determine a second advanced decoder corresponding to the second style emotion information from K advanced decoders; use the first advanced decoder to decode the first training encoding feature to obtain a first training acoustic feature corresponding to the first style emotion information, and use the second advanced decoder to decode the second training encoding feature to obtain a second training acoustic feature corresponding to the second style emotion information; perform acoustic encoding processing on the first training acoustic feature to obtain a first predicted audio information corresponding to the first training text information and the first style emotion information, and perform acoustic encoding processing on the second training acoustic feature to obtain a second predicted audio information corresponding to the second training text information and the second style emotion information; and train the shared encoder, the first advanced decoder, and the second advanced decoder based on the first training audio information and the first predicted audio information, and the second training audio information and the second predicted audio information.

15. The speech synthesis device according to claim 14, characterized in that: K types of style emotion information correspond to K types of style emotion embedding vectors, where the K types of style emotion embedding vectors include a first style emotion embedding vector corresponding to the first style emotion information and a second style emotion embedding vector corresponding to the second style emotion information; Wherein, the processing unit is also used for: subjecting the first training coding feature and the first style emotion embedding vector corresponding to the first style emotion information to fusion processing to obtain a first fused feature vector, and subjecting the second training coding feature and the second style emotion embedding vector corresponding to the second style emotion information to fusion processing to obtain a second fused feature vector; using the first advanced decoder to decode the first fused feature vector to obtain the first advanced decoding information of the first training text information, and using the second advanced decoder to decode the second fused feature vector to obtain the second advanced decoding information of the second training text information; post-processing the first advanced decoding information based on the first style emotion information to obtain the first training acoustic feature, and post-processing the second advanced decoding information based on the second style emotion information to obtain the second training acoustic feature.

16. The speech synthesis device according to claim 15, characterized in that: The processing unit is also used to: determine a first duration predictor corresponding to the first style emotion information from K duration predictors corresponding to K styles of emotion information, and determine a second duration predictor corresponding to the second style emotion information from the K duration predictors; use the first duration predictor to perform duration prediction processing on the first training coding feature to obtain the first duration information of the first training text information, and use the second duration predictor to perform duration prediction processing on the second training coding feature to obtain the second duration information of the second training text information; use the first duration information to upsample the first training coding feature, and use the second duration information to upsample the second training coding feature; fuse the upsampled first training coding feature with the first style emotion embedding vector to obtain the first fused feature vector, and fuse the upsampled second training coding feature with the second style emotion embedding vector to obtain the second fused feature vector.

17. The speech synthesis device according to claim 15, characterized in that: The processing unit is also used to: use a shared preliminary decoder to perform preliminary decoding processing on the first fused feature vector and the second fused feature vector respectively to obtain first preliminary decoding information of the first training text information and second preliminary decoding information of the second training text information; use the first advanced decoder to perform advanced decoding processing on the first preliminary decoding information to obtain the first advanced decoding information, and use the second advanced decoder to perform advanced decoding processing on the second preliminary decoding information to obtain the second advanced decoding information.

18. The speech synthesis device according to claim 15, characterized in that: The processing unit is further used to: determine a first post-processing module corresponding to the first style emotion information from the K post-processing modules corresponding to the K types of style emotion information, and determine a second post-processing module corresponding to the second style emotion information from the K post-processing modules; The first advanced decoded information is post-processed by the first post-processing module to obtain the first training acoustic feature, and the second advanced decoded information is post-processed by the second post-processing module to obtain the second training acoustic feature.

19. A computer readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

20. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 9 by executing the executable instructions.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to claims 1 to 9 is implemented.

Images