Method for converting text data into acoustic features, electronic device and computer program

Abstract

A method for converting text data into acoustic features, an electronic device and a computer-readable storage medium, which relate to the technical field of artificial intelligence, the method includes the steps of: for each text data block among a plurality of text data blocks, obtaining hidden text features corresponding to the text data block and hidden text features corresponding to at least one text data block subsequent to the text data block by a neural network model, the hidden text features being a non-final output feature vector generated by the neural network model; performing a coarse fusion process on the hidden text features corresponding to the text data block and the hidden text features corresponding to the at least one text data block subsequent to the text data block to obtain at least one coarse fusion text feature; performing a fine fusion process on the at least one coarse fusion text feature to obtain a fine fusion text feature corresponding to the text data block; adjusting a length corresponding to the fine fusion text feature; and converting the fine fusion text feature after the length adjustment into an acoustic feature (S203).

Description

[Technical field]

[0001] This application claims priority to a Chinese patent application filed with the China Patent Office on April 19, 2021, bearing application number 202110419629X and entitled "Method for converting text data into acoustic features," the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the technical field of artificial intelligence, and more particularly, to a method for converting text data into acoustic features, an electronic device, and a computer-readable storage medium. [Background technology]

[0003] Currently, Text-To-Speech (TTS) technology has been proposed to convert text data into voice. TTS technology intelligently converts text into a natural voice stream by simultaneously using linguistics and psychology to design a neural network. However, current TTS technology still has problems such as unnatural phonological rhythms in the synthesized voice, mechanical sounds, and missing words and phrases, and there is still a clear difference between the voice synthesized by TTS technology and the voice read by a human.

[0004] Currently, neural network models related to TTS technology are generally based on an attention mechanism. The attention mechanism inputs the sequence encoded by the encoder and the attention weights encoded by the attention layer to the decoder, thereby improving the text data processing ability of the TTS technology and improving the efficiency of the neural network model. The architectures of the encoder and decoder in the neural network model using the attention mechanism are generally similar, for example, both are based on the transformer architecture. Currently, it has also been proposed to improve the quality of synthesized voice by using an autoregressive neural network model. Summary of the Invention [Problem to be solved by the invention]

[0005] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure provide a method, an electronic device, and a computer-readable storage medium for converting text data into acoustic features. [Means for solving the problem]

[0006] An embodiment of the present disclosure provides a method for converting text data into acoustic features, executed by an electronic device, where the text data includes a plurality of continuous text data blocks, the method including: for each text data block of the plurality of text data blocks, obtaining hidden text features corresponding to the text data block and hidden text features corresponding to at least one text data block subsequent to the text data block by a neural network model, the hidden text features being non-final output feature vectors generated from the neural network model; performing a coarse fusion process on the hidden text features corresponding to the text data block and the hidden text features corresponding to the at least one text data block subsequent to the text data block to obtain at least one coarse fusion text feature; performing a fine fusion process on the at least one coarse fusion text feature to obtain a fine fusion text feature corresponding to the text data block; adjusting a length corresponding to the fine fusion text feature; and converting the fine fusion text feature after the length adjustment into an acoustic feature.

[0007] An embodiment of the present disclosure discloses an electronic device, comprising one or more processors and one or more memories, wherein the memories store computer-executable programs, which, when executed by the processors, perform the above-mentioned methods.

[0008] An embodiment of the present disclosure provides a computer-readable storage medium having stored thereon computer instructions, which, when executed by a processor, implement the above method.

[0009] According to another aspect of the present disclosure, there is provided a computer program product or computer program, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium, the computer instructions being read by a processor of a computing device from the computer readable medium and executed by the processor to cause the computing device to perform a method provided by any of the above aspects or various preferred implementations of any of the above aspects. Effect of the Invention

[0010] An embodiment of the present disclosure provides a method for converting text data into acoustic features, which uses a dense-fuse encoder that can densely connect multiple transformers and a multi-head attention layer to learn a fusion process of text features corresponding to text data, thereby improving the accuracy and richness of extracted acoustic features and robustly generating natural synthetic speech.

[0011] The embodiments of the present disclosure further use a length adjuster as a replacement for an attention layer between the encoder and decoder to overcome the effects of instability caused by the attention layer and robustly generate natural synthetic speech.

[0012] The embodiments of the present disclosure further reduce the adverse effects of exposure bias and local information preference by using a non-autoregressive single-layer RNN structure decoder, thereby improving the decoding efficiency of the decoder and the accuracy of the decoded acoustic features.

[0013] In order to more clearly describe the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for the description of the embodiments. The drawings described below are exemplary embodiments of the present disclosure. [Brief description of the drawings]

[0014] [Figure 1] FIG. 2 is an exemplary schematic diagram of a reasoning scenario for an acoustic model according to an embodiment of the present disclosure. [Diagram 2] 1 is a flowchart of a method for converting text data into acoustic features according to an embodiment of the present disclosure. [Diagram 3] FIG. 1 is a schematic diagram of an apparatus for converting text data into acoustic features according to an embodiment of the present disclosure. [Figure 4] FIG. 2 is a schematic diagram of an encoder according to an embodiment of the present disclosure. [Diagram 5] FIG. 2 is a schematic diagram of a transformer in an encoder according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a schematic diagram of a length adjuster according to an embodiment of the present disclosure. [Figure 7] FIG. 2 is a schematic diagram of a length predictor according to an embodiment of the present disclosure. [Figure 8A] FIG. 2 is another schematic diagram of an acoustic model according to an embodiment of the present disclosure. [Figure 8B] 4 is another flowchart of a method according to an embodiment of the present disclosure. [Figure 8C] FIG. 8B is a schematic diagram of a training flow for the acoustic model of FIG. 8A. [Figure 9A] FIG. 2 is another schematic diagram of an acoustic model according to an embodiment of the present disclosure. [Figure 9B] 4 is another flowchart of a method according to an embodiment of the present disclosure. [Figure 9C] FIG. 9B is a schematic diagram of a training flow for the acoustic model of FIG. 9A. [Figure 10] FIG. 2 is a schematic diagram of another training process of an acoustic model according to an embodiment of the present disclosure. [Figure 11] 1 is an exemplary structure of a length extractor according to an embodiment of the present disclosure. [Figure 12]FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the present disclosure. [Figure 13] FIG. 1 is a schematic diagram of an exemplary computing device architecture in accordance with an embodiment of the present disclosure. [Figure 14] 1 is a schematic diagram of a storage medium according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, exemplary embodiments according to the present disclosure are described in detail below with reference to the drawings. Obviously, the described embodiments are not all embodiments of the present disclosure, but are only some embodiments of the present disclosure, and the present disclosure is not limited to the exemplary embodiments described herein.

[0016] In the present specification and drawings, steps and elements that are essentially the same or similar are indicated by the same or similar reference numerals, and duplicate descriptions of these steps and elements are omitted. In addition, in the description of the present disclosure, the terms "first", "second", etc. are not intended to indicate or imply relative importance or order, but are merely intended to distinguish the description.

[0017] The attention mechanism slows down the convergence of the model, so the generated sound features are not stable enough, and the robustness is insufficient when generating long-length sound data. The same architecture of the encoder and decoder may cause the acoustic features to not match the text conditions extracted from the text data (e.g., it is necessary to ensure that the synthesized speech has no word leakage), or the text features extracted from the long text may not be accurate enough. The autoregressive neural network model may weaken the dependency between the text conditions and the synthesized speech, leading to the situation of word / phrase leakage.

[0018] Therefore, there is a need to further improve TTS technology to robustly synthesize speech that more closely resembles human reading.

[0019] In order to conveniently describe the present disclosure, the following introduces some concepts relevant to the present disclosure.

[0020] The present disclosure provides a method for converting the text data into acoustic features by using an acoustic model, and the encoder, decoder and length adjuster mentioned below are all components of the acoustic model.

[0021] The acoustic model of the present disclosure is based on artificial intelligence (AI). Artificial intelligence is a theory, method, technology, and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, sense the environment, and acquire and use knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology of computer science that is intended to understand the essence of intelligence and create new smart machines that can respond in a manner similar to human intelligence. For example, the acoustic model of the present disclosure translates a variety of different languages ​​to be similar to the way humans browse and understand these languages. Artificial intelligence studies the design principles and implementation methods of various smart machines, and the acoustic model of the present disclosure has the function of understanding a variety of different languages ​​and translating them into other languages.

[0022] Artificial intelligence technology is a broad field, including both hardware and software technologies. Artificial intelligence software technology mainly includes computer vision technology, natural language processing, and machine learning / deep learning.

[0023] In some embodiments, the acoustic model in the present disclosure employs natural language processing (NLP) technology. Natural language processing technology is an important direction in the fields of computer science and artificial intelligence, and can realize various theories and methods for effectively communicating between humans and computers using natural language. Natural language processing is a science that integrates linguistics, computer science and mathematics. Thus, based on the natural language processing technology, the acoustic model in the present disclosure analyzes input text data, extracts features in the text data, and generates audio data in the manner of human text reading.

[0024] In some embodiments, the natural language processing technology employed in the embodiments of the present disclosure may further be based on machine learning (ML) and deep learning. Machine learning is a multidisciplinary field that involves multiple disciplines, such as probability theory, statistics, approximation theory, convex analysis, and computational complexity theory. Natural language processing technology uses machine learning to study how computers can simulate or realize the behavior of human language learning, acquire new knowledge or skills by analyzing existing, categorized text data, and reorganize existing knowledge structures to continuously improve their own performance. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0025] In some embodiments, any acoustic model applied to the embodiments of the present disclosure may be an artificial intelligence model, in particular, an artificial intelligence neural network model. Generally, an artificial intelligence neural network model is implemented as an acyclic graph, with neurons arranged in different layers. Generally, a neural network model includes an input layer and an output layer separated by at least one hidden layer. The hidden layer converts inputs received by the input layer into a representation useful for generating outputs in the output layer. Network nodes are fully connected to nodes in adjacent layers via edges, and there are no edges between nodes within each layer. Data received at nodes in the input layer of the neural network is propagated to nodes in the output layer via any one of a hidden layer, an activation layer, a pooling layer, a convolutional layer, etc. The inputs and outputs of the neural network model may take various forms, and the present disclosure is not limited thereto.

[0026] The solutions provided by the embodiments of the present disclosure relate to technologies such as artificial intelligence, natural language processing, and machine learning, and are specifically described in the following embodiments.

[0027] The acoustic model of the embodiment of the present disclosure is specifically integrated in an electronic device, which may be a device such as a terminal or a server. For example, the acoustic model is integrated in a terminal. The terminal may be, but is not limited to, a mobile phone, a tablet, a laptop, a desktop computer, a personal computer (PC), a smart speaker, or a smart watch. For example, the acoustic model may be integrated in a server. The server may be an independent physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), big data, and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected by wired or wireless communication, which is not limited by the present disclosure.

[0028] Here, the device that performs reasoning using the acoustic model of the embodiment of the present disclosure may be a terminal, a server, or even a system composed of a terminal and a server.

[0029] Here, the method of converting text data into acoustic features using the acoustic model of the embodiment of the present disclosure may be executed on a terminal, may be executed on a server, or may be jointly executed by the terminal and the server.

[0030] The acoustic model provided by the embodiments of the present disclosure may further relate to artificial intelligence cloud services in the cloud technology field. Cloud technology is a hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or a local network to realize data calculation, storage, processing and sharing. Cloud technology is a collective term for network technology, information technology, integration technology, platform management technology, application technology, etc. that are applied based on the cloud computing business mode, and forms a resource pool, and is flexible and convenient to apply according to needs. Cloud computing technology is becoming an important support. The background service of the technology network system requires a large amount of computing and storage resources, such as video websites, picture websites and more portal sites. With the advanced development and application of the Internet industry, in the future, each product will have its own recognition mark, and each will be transmitted to the background system for logic processing, and different levels of data may be processed separately, and each type of industry data requires powerful system support, which can only be realized by cloud computing.

[0031] AI cloud services are commonly referred to as AIaaS (AI as a Service). This is the service mode of the current mainstream AI platform. Specifically, the AIaaS platform divides some of the more common AI services and provides independent or packaged services in the cloud. This service mode is similar to the opening of an AI theme mall, where any developer can access and use one or more AI services provided by the platform through an application programming interface (API), and some senior developers can even use the AI ​​framework and AI infrastructure provided by the platform to configure, operate and maintain their own dedicated cloud AI services.

[0032] FIG. 1 is an exemplary schematic diagram of an acoustic model reasoning scenario 100 according to an embodiment of the present disclosure.

[0033] Currently, there are many types of reading applications. A user installs a reading application on his / her user terminal, and instructs the reading application on the text data that needs to be converted into audio data. The user terminal then transmits a text data conversion request to the server of the application via a network, receives the converted audio data corresponding to the text data, and then plays the audio data.

[0034] The server receives the text data to be converted, converts the text data according to an acoustic model, obtains audio data, and then feeds back the audio data (e.g., audio data corresponding to the text data in FIG. 1) to the user.

[0035] The user scores the audio data. For example, if the user thinks that the audio data and the text data correspond well, there are no obvious missing words or phrases, and the effect is close to that of a human reading aloud, the user will give the audio data a high score, and the server will select the text data-audio data pair as a positive sample for training the acoustic model in real time. If the user gives the audio data a low score, the server will select the text data-audio data pair as a negative sample for training the acoustic model in real time.

[0036] Of course, the server may also obtain samples for training the acoustic model in other ways, for example, the server crawls the audio of human text readings and corresponding texts existing in the Internet environment, and trains the acoustic model with such human text readings, for example, referring to FIG. 1, the server obtains text from a database for training the acoustic model.

[0037] Currently, the acoustic model that converts text data to audio data may be an autoregressive neural network model with an attention mechanism (e.g., Tacotron) or a non-autoregressive feedforward neural network model with a duration predictor (e.g., Fastspeech).

[0038] Tacotron is an example of an autoregressive neural network model with attention mechanism. Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, RJ Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio and other researchers introduced Tacotron in detail in the article "Tacotron: Towards end-to-end speech synthesis". In this article, Tacotron merges the generation of linguistic features and acoustic features into one network, and uses a large amount of data for end-to-end training, simplifying the traditional synthesis process and generating highly natural speech that is close to human quality.

[0039] However, Tacotron cannot robustly process out-of-domain text (text different from the type / field of the training text) and can only generate humanoid voices for in-domain text (text similar to the type / field of the training text). For example, Tacotron cannot process test texts with different text lengths from the training texts well. Also, for example, if the training texts are concentrated in the news field and the test texts are concentrated in the daily conversation field, the voice of the test texts synthesized by Tacotron is not natural enough. There are several reasons why Tacotron's robustness is weak. First, Tacotron does not have a soft attention mechanism to prevent skipping, overlapping, and mispronunciation. Second, Tacotron determines whether speech synthesis is complete by predicting whether the current frame is a stop frame. Therefore, if Tacotron mispredicts the stop frame, it will cause early cutoff and stop delay. Third, Tacotron adopts a supervised training mechanism, which will cause a mismatch between the training period and the reasoning period (also called exposure bias). Tacotron also employs an autoregressive decoder, whose local information preferences weaken the dependency between predicted acoustic features and textual conditions, leading to failure of the synthesized speech.

[0040] Fastspeech is an example of a non-autoregressive feedforward neural network model with duration predictor. In the article "Fastspeech: Fast, robust and controllable text to speech", researchers such as Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao and T.-Y. Liu use a duration predictor instead of an attention module to eliminate the robustness problems of attention disable and freeze-frame prediction. However, compared with Tacotron, the quality of the synthesized speech by Fastspeech still has a gap.

[0041] In addition, the types of Tacotron encoders and decoders are similar, and both are based on a transformer mechanism. The types of Fastspeech encoders and decoders are also based on a transformer mechanism. With encoders and decoders of the same architecture, the acoustic features may not match the text conditions extracted from the text data, or the text features extracted from long texts may not be accurate enough.

[0042] In view of this, the present disclosure provides a method for converting text data into acoustic features, the text data including a plurality of text data blocks, the method including: for each text data block among the plurality of text data blocks, performing a coarse fusion process on hidden text features corresponding to the text data block and hidden text features corresponding to at least one subsequent text data block of the text data block to obtain at least one coarse fusion text feature; performing a fine fusion process on the at least one coarse fusion text feature to obtain a fine fusion text feature corresponding to the text data block; adjusting a length corresponding to the fine fusion text feature; and converting the fine fusion text feature after the length adjustment into an acoustic feature. The method performs the coarse fusion process and the fine fusion process on the hidden text feature to learn a fusion process of text features corresponding to the text data, thereby improving the accuracy and richness of the extracted acoustic features, and robustly generating natural synthetic speech.

[0043] The embodiment of the present disclosure further uses a length adjuster as a replacement for the attention layer between the encoder and the decoder to overcome the influence of instability caused by the attention layer and robustly generate natural synthetic speech. The embodiment of the present disclosure further uses a non-autoregressive single-layer RNN structure decoder to reduce the adverse influence of exposure bias and local information preference, and improve the decoding efficiency of the decoder and the accuracy of the decoded acoustic features.

[0044] In the following, in combination with FIGS. 2 to 14, an embodiment of the present disclosure will be introduced in detail, in which the accuracy and richness of the extracted acoustic features are improved, so that the acoustic model robustly generates natural synthetic speech.

[0045] 2 is a flowchart of a method 200 for converting text data into acoustic features according to an embodiment of the present disclosure. The method 200 according to an embodiment of the present disclosure is applicable to any electronic device. Here, the electronic device may be different types of hardware devices, such as a personal digital assistant (PDA), an audio / video device, a mobile phone, an MP3 player, a personal computer, a laptop computer, a server, etc. For example, the electronic device may be the server in FIG. 1, a user terminal, or a system composed of a server and a user terminal, etc. As can be understood by those skilled in the art, the present disclosure is not limited thereto.

[0046] 2, in step S201, for each text data block among a plurality of continuous text data blocks constituting text data, a hidden text feature corresponding to the text data block and a hidden text feature corresponding to at least one text data block following the text data block are obtained by a neural network model, the hidden text feature is a feature vector generated by the neural network model and outputted as a non-final output, a coarse fusion process is performed on the hidden text feature corresponding to the text data block and the hidden text feature corresponding to at least one text data block following the text data block to obtain at least one coarse fusion text feature, and a fine fusion process is performed on the at least one coarse fusion text feature to obtain a fine fusion text feature corresponding to the text data block. In step S202, a length corresponding to the fine fusion text feature is adjusted. In step S203, the fine fusion text feature after length adjustment is converted into an acoustic feature.

[0047] For example, the text data block described herein may be any element constituting the text to be read aloud in Fig. 1, such as a word, a phrase, a paragraph, a chapter, etc. The present disclosure is not limited to the length and language type of the text data block, and for example, the text data block may include text information in English, Chinese, Hindi, Russian, Japanese, Korean, etc., such as "Hello Dad" in Chinese and "Hello World" in English.

[0048] In some embodiments, step S201 is performed by various neural network models capable of extracting text features from text data. In some embodiments, the neural network model is implemented as an acyclic graph, with neurons arranged in different layers. The neural network model includes an input layer and an output layer separated by at least one hidden layer. The hidden layer converts inputs received by the input layer into a representation useful for generating outputs in the output layer. Network nodes are fully connected to nodes in adjacent layers via edges, and there are no edges between nodes within each layer. Data received at nodes in the input layer of the neural network is propagated to nodes in the output layer through any one of multiple hidden layers, multiple activation layers, multiple pooling layers, multiple convolution layers, etc. The inputs and outputs of the neural network model may take various forms, to which the present disclosure is not limited.

[0049] For example, the hidden text feature corresponding to the text data block (or the hidden text feature corresponding to at least one text data block subsequent to the text data block) may be any non-final output feature vector generated in the process of generating the coarse fused text feature and the fine fused text feature, for example, the output of a hidden layer, or a convolutional layer, or an activation layer, or a pooling layer in the neural network model that performs step S201.

[0050] For example, the coarse fusion process for the hidden text feature corresponding to the text data block (and / or the hidden text feature corresponding to at least one text data block subsequent to the text data block) may instruct the hidden text feature to only undergo simple linear processing and not pass the hidden text feature through any learnable / trainable hidden layers. The fine fusion process for the at least one coarse fusion feature may instruct the at least one coarse fusion feature to pass through at least one learnable / trainable hidden layer. The coarse fusion process retains information in a hidden text feature to a large extent, while the fine fusion process performs learnable processing on multiple coarse fusion text features to extract more information at a richer level.

[0051] Generally, the length of an acoustic feature is larger than that of a text feature, so by performing length extension on the text feature, the text feature can be better decoded by the acoustic feature. Of course, in some cases, reducing the length of the text feature can improve the computation speed. Thus, in step S202, the method 200 adjusts the length corresponding to the thin fusion text feature so that the decoder can decode it more easily.

[0052] The acoustic features converted in step S203 incorporate various information, such as prosodic information, phoneme information, tone information, participle information, emotion information, text information, and semantic information. For example, the acoustic features may be feature vectors directly decoded by a vocoder. For example, the acoustic features may be Mel Frequency Cepstral Coefficients (MFCC) of the speech to be synthesized.

[0053] Since the finely fused text features are generated by performing multi-level and multi-length deep fusion of various text features (e.g., hidden text features and coarsely fused features), the acoustic features generated based on the finely fused text features are highly accurate and rich, and can synthesize speech that is closer to a human voice.

[0054] Thus, the embodiments of the present disclosure provide a method for converting text data into acoustic features, which performs coarse and fine fusion processes on hidden text features corresponding to different text data blocks to improve the accuracy and richness of the converted acoustic features, based on which the embodiments of the present disclosure robustly generate natural synthetic speech.

[0055] 3 is a schematic diagram of an apparatus 300 for converting text data into acoustic features according to an embodiment of the present disclosure. Hereinafter, the apparatus 300 is also referred to as an acoustic model 300. Hereinafter, the structure of a neural network model for performing the method 200 is described using the acoustic model of FIG. 3 as an example, and it is understood by those skilled in the art that the present disclosure is not limited thereto. As shown in FIG. 3, the acoustic model includes an encoder 301, a length adjuster 302 and a decoder 303.

[0056] For example, the encoder 301 executes the above step S201, that is, the encoder 301 is arranged to, for each text data block among a plurality of consecutive text data blocks, perform a coarse fusion process on the hidden text features corresponding to the text data block and the hidden text features corresponding to at least one text data block subsequent to the text data block to obtain at least one coarse fusion text feature, and perform a fine fusion process on the at least one coarse fusion text feature to obtain a fine fusion text feature corresponding to the text data block.

[0057] Furthermore, the encoder 301 is a dense-fuse encoder with a transformer structure. In some embodiments, the encoder 301 densely connects multiple basic transformers with bypass connections, and fuses the outputs of each transformer with different information in a manner that cannot be learned (e.g., coarse fusion process). The encoder 301 further includes one multi-head attention layer, which further densely connects multiple basic transformers, and fuses the outputs of each hidden layer with different information in a manner that can be learned (fine fusion process). In this way, richer and more accurate information is fused in the fine-fused text features.

[0058] A highly integrated encoder with a transformer structure can capture more global information, is faster to train, more efficient, and achieves rapid parallelism through a self-attention mechanism (e.g., a multi-head attention layer). In some embodiments, each transformer in the encoder 301 outputs hidden text features corresponding to a block of text data.

[0059] In some embodiments, a hidden text feature corresponding to a text data block is obtained by fusing first prediction information and second prediction information of the text data block, the first prediction information being predicted based on the text data block, and the second prediction information being predicted based on at least one text data block subsequent to the text data block.

[0060] For example, the first prediction information may be pinyin information, phoneme information, semantic information, emotion information, etc., directly extracted based on the text data block. The second prediction information may be emotion information, speech rate information, etc., extracted based on the subsequent text data block. For example, the text to be recited is "It's a very nice day today". When generating a hidden text feature corresponding to the word "today", it is necessary to consider the possible influence of the subsequent words "weather" and "very nice". For example, "very nice weather" expresses a happy emotion, so the voice synthesized based on the hidden text feature corresponding to "today" should be light and natural, not heavy. Of course, as can be understood by those skilled in the art, more or less information may be integrated into the hidden text feature corresponding to the text data block, and the present disclosure is not limited thereto.

[0061] In some embodiments, the coarse fusion process is performed by an encoder, the encoder including N transformers and N adders, and the step of performing the coarse fusion process on the hidden text features corresponding to the text data block and the hidden text features corresponding to at least one subsequent text data block of the text data block further includes the step of generating N coarse fusion text features based on the hidden text features corresponding to the text data block and the hidden text features corresponding to at least one subsequent text data block of the text data block by the N transformers and the N adders, where N is a positive integer. The process of the coarse fusion process will be further described below with reference to FIG. 4, and the present disclosure will not be repeated here. Of course, as can be understood by those skilled in the art, the coarse fusion process may include more or less processing methods, and the present disclosure is not limited thereto.

[0062] In some embodiments, the step of performing a fine fusion process on the at least one coarse fusion text feature further includes the steps of taking one or more of the coarse fusion text features of the at least one coarse fusion text feature as a search feature, taking the hidden text feature corresponding to the text data block and the hidden text feature corresponding to at least one subsequent text data block of the text data block as a value feature and a key feature, respectively, and predicting a fine fusion text feature corresponding to the text data block based on the search feature, the value feature and the key feature. The process of the fine fusion process will be further described below with reference to FIG. 4, and the present disclosure will not be detailed here. Of course, as can be understood by those skilled in the art, the fine fusion process may include more or less processing methods, and the present disclosure is not limited thereto.

[0063] For example, the length adjuster 302 performs step S202 to solve the problem of mismatched lengths between the thin-fused text features and the acoustic features to be generated. That is, the length adjuster 302 is configured to adjust the length corresponding to the thin-fused text features. The length adjuster replaces the attention layer between the encoder 301 and the decoder 303 in the conventional acoustic model, and solves the effect of instability caused by the attention layer to robustly generate natural synthetic speech.

[0064] For example, the decoder 303 performs the above step S203. That is, the decoder 303 is configured to convert the thin-fused text features after length adjustment into acoustic features. In some embodiments, the decoder is a non-autoregressive single-layer loop recurrent neural network (RNN) structure. For example, the input of the decoder 303 is the thin-fused text features after length adjustment, and the output of the decoder is acoustic features. Compared with the autoregressive decoder with a transformer structure in the traditional acoustic model, the decoder with a non-autoregressive single-layer RNN structure is more advantageous in generating text conditions and can reduce the adverse effects of exposure bias and local information preference.

[0065] The decoder 303 is not trained in a strongly supervised manner, but only feeds back the length-adjusted thin-fused text features to the decoder, which places high requirements on the thin-fused text features. The above encoder 301 can meet these requirements.

[0066] In this way, the embodiment of the present disclosure uses an advanced fusion encoder that can densely connect multiple transformers and a multi-head attention layer to learn a fusion process of text features corresponding to text data, improve the accuracy and richness of the extracted acoustic features, and robustly generate natural synthetic speech. The embodiment of the present disclosure further uses a length adjuster as a substitute for the attention layer between the encoder and the decoder to solve the influence of instability caused by the attention layer, and robustly generate natural synthetic speech. The embodiment of the present disclosure further uses a decoder with a non-autoregressive single-layer RNN structure to reduce the adverse influence of exposure bias and local information preference, thereby improving the decoding efficiency of the decoder and the accuracy of the decoded acoustic features.

[0067] FIG 4 is a schematic diagram of an encoder 301 according to an embodiment of the present disclosure. The encoder 301 includes multiple transformers. FIG 5 is a schematic diagram of a transformer in the encoder 301 according to an embodiment of the present disclosure. As can be understood by those skilled in the art, the structures in FIG 4 and FIG 5 are only examples, and the present disclosure is not limited thereto.

[0068] Referring to Fig. 4, in some embodiments, the encoder 301 includes N transformers, N adders, and one multi-head attention layer. The output of the nth transformer and the output of the n-1th adder are both connected to the input of the nth adder, and the output of the nth adder is connected to the input of the n+1th transformer, and the input of the multi-head attention layer includes the outputs of the N transformers and the output of the Nth adder, each transformer outputs one hidden text feature, each adder outputs one coarse fused text feature, and the multi-head attention layer of the transformer outputs a fine fused text feature corresponding to the text data block, where 1 < n < N, and N is a positive integer greater than 1. In some embodiments, the encoder further includes an embedding layer and a one-dimensional convolutional network, examples of which may be seen in Fig. 4.

[0069] For the m-th text data block, the operation related to the rough fusion process in step S201 (i.e., performing rough fusion process on the hidden text features corresponding to the text data block and the hidden text features corresponding to at least one subsequent text data block of the text data block to obtain rough fusion text features) may be performed in the following manner:

[0070] From the Nth transformer to the 2nd transformer and from the Nth adder to the 2nd adder, N-1 hidden text features and N-1 rough fused text features are generated.

[0071] For example, for an nth transformer and an nth adder (2≦n≦N), first, use the nth transformer to generate a hidden text feature corresponding to the m+Nnth text data block based on the n-1th coarsely fused text feature. Use the nth adder to generate an nth coarsely fused text feature based on the hidden text feature corresponding to the m+Nnth text data block and the n-1th coarsely fused text feature. For example, the nth coarsely fused text feature is a sum of the hidden text feature corresponding to the m+Nnth text data block and the n-1th coarsely fused text feature. Use the embedding layer to generate an embedding representation vector corresponding to the m+Nth text data block based on the m+Nth text data block. Use the one-dimensional convolutional network to generate a context feature vector including a relative position feature of the m+Nth text data block based on the embedding representation vector corresponding to the m+Nth text data block. Use a first transformer to generate a hidden text feature corresponding to the m+Nth text data block based on the context feature vector. A first adder is used to generate a first coarse-fused text feature based on the m+Nth context feature vector and the hidden text feature corresponding to the m+N-1th text data block, where the first coarse-fused text feature is a sum of the m+Nth context feature vector and the hidden text feature corresponding to the m+N-1th text data block.

[0072] For the mth text data block, the operation related to the fine fusion processing in step S201 (i.e., performing the fine fusion processing on the at least one coarse fusion text feature to obtain a fine fusion text feature corresponding to the text data block) may be performed in the following manner:

[0073] For example, using a multi-head attention layer, the Nth coarsely fused text feature output from the Nth adder is used as the search feature, and the hidden text features corresponding to the mth text data block to the hidden text features corresponding to the m+Nth text data block are used as the value features and key features to generate a finely fused text feature corresponding to the mth text data block.

[0074] FIG. 4 illustrates an encoder 301 that includes four transformers (e.g., the fourth transformer, the third transformer, the second transformer, and the first transformer shown), four adders (

number

[0075] Take the situation of N=4 and m=2 in Fig. 4 as an example. Suppose the encoder 301 encodes a word sequence {a, b, c, d, e, f}, where the word corresponding to the m-th text data block is "b".

[0076] The multi-head attention layer takes the fourth coarsely fused text feature output from the fourth adder as a search feature, and takes the hidden text features corresponding to the second text data block to the hidden text features corresponding to the sixth text data block as value features and key features to generate a finely fused text feature corresponding to the second text data block. For example, the multi-head attention sub-layer calculates the attention distribution of the value feature (V) based on the search feature (Q), and adds the attention distribution to the key feature (K) to calculate a finely fused text feature that highly integrates the text information of "b", "c", "d", and "e", including the self-attention information of "b", "c", "d", and "e".

[0077] In addition, a fourth Transformer is used to generate hidden text features corresponding to the second text data block ("b") based on the third coarse fused text features, where the first prediction information and the second prediction information are fused into the hidden text features corresponding to "b", where the first prediction information is predicted based on the text data block "b", and the second prediction information is predicted based on the text data blocks "c", "d", and "e".

[0078] A fourth adder is used to generate a fourth coarse fused text feature based on the hidden text feature corresponding to the second text data block ("b") and the third coarse fused text feature, the fourth coarse fused text feature being a sum of the hidden text feature corresponding to the second text data block and the third coarse fused text feature.

[0079] A third Transformer is used to generate hidden text features corresponding to a third text data block ("c") based on the second coarse fused text features. The hidden text features corresponding to "c" are fused with the first prediction information and the second prediction information, where the first prediction information is predicted based on the text data block "c" and the second prediction information is predicted based on the text data blocks "d" and "e".

[0080] A third adder is used to generate a third coarse fused text feature based on the hidden text feature corresponding to the third text data block ("c") and the second coarse fused text feature, the third coarse fused text feature being a sum of the hidden text feature corresponding to the third text data block and the second coarse fused text feature.

[0081] A second Transformer is used to generate hidden text features corresponding to the fourth text data block ("d") based on the first coarse fused text features, where the hidden text features corresponding to "d" are fused with the first prediction information and the second prediction information, where the first prediction information is predicted based on the text data block "d" and the second prediction information is predicted based on the text data block "e".

[0082] A second adder is used to generate a second coarse-fused text feature based on the hidden text feature corresponding to the fourth text data block ("d") and the first coarse-fused text feature, the second coarse-fused text feature being a sum of the hidden text feature corresponding to the fourth text data block and the first coarse-fused text feature.

[0083] A first Transformer is used to generate hidden text features corresponding to the fifth text data block ('e') based on the context feature vector corresponding to 'e', ​​and the hidden text features corresponding to 'e' are fused with first prediction information, and the first prediction information is predicted based on the text data block 'e'.

[0084] A first adder is used to generate a first coarse fused text feature based on the fifth context feature vector and the hidden text feature corresponding to the fourth text data block, the first coarse fused text feature being the sum of the fifth context feature vector and the hidden text feature corresponding to the fourth text data block ("d").

[0085] An embedding layer is used to generate an embedding representation vector corresponding to the fifth text data block ("e") based on the fifth text data block ("e"). Then, a one-dimensional convolutional network is used to generate a context feature vector including the relative position feature of the fifth text data block ("e") based on the embedding representation vector corresponding to the fifth text data block ("e"). The one-dimensional convolutional network captures the relative position information of the text data block ("e") in the entire word sequence through a convolution operation.

[0086] The structures of the first to fourth Transformers can be seen in FIG. 5. Hereinafter, the manner in which the Transformers predict hidden text features corresponding to text data block “e” based on a context feature vector corresponding to the text data block “e” will be explained with reference to FIG. 5.

[0087] As shown in Figure 5, each transformer includes one multi-head attention sub-layer, multiple dropout layers, multiple sub-adders, multiple normal layers, one-dimensional convolutional sub-networks, etc. The dropout layer prevents some neurons in the prologue hidden layer from randomly operating, thereby avoiding overfitting.

[0088] The multi-head attention sub-layer in the first Transformer performs deep extraction and fusion on the context feature vector corresponding to the text data block "e" using the context feature vector corresponding to the text data block "e" as the search feature, value feature, and key feature. For example, the multi-head attention sub-layer calculates the attention distribution of the key feature based on the search feature, and adds the attention distribution to the value feature to calculate a sub-feature vector containing self-attention information of "e". Then, the Transformer further processes the sub-feature vector, for example, by multiple dropout layers, multiple sub-adders, multiple normal layers, one-dimensional convolutional sub-networks, etc., to perform further feature extraction and fusion on the context feature vector corresponding to "e" and the sub-feature vector containing self-attention information of "e", thereby predicting hidden text features corresponding to "e".

[0089] The multi-head attention sub-layer in the first Transformer uses the first coarsely fused text feature as a search feature, a value feature, and a key feature to perform deep extraction and fusion on the first coarsely fused text feature to predict a hidden text feature corresponding to "d". The first coarsely fused text feature is the sum of the context feature vector of "e" and the hidden text feature of the fourth text data block ("d") output from the first Transformer, so that the hidden text feature predicted by this method for "d" contains information of "e" and information of "d". In this way, the deep fusion and extraction processes of other Transformers can be inferred, and the present disclosure will not be detailed here.

[0090] In some implementations, all the above feature vectors (hidden text features, coarse fused text features, fine fused text features) can be 256-dimensional or 1024-dimensional numerical vectors, and can be custom aligned based on the alignment accuracy of the acoustic model.

[0091] From the perspective of feature fusion, the features output from the lower layer transformer are reused by the upper layer transformer using the dense connection method of Fig. 4. Because the features extracted by different transformers have different information, the feature reuse method of Fig. 4 enhances the final display ability of the encoder 301. From the perspective of training, the supervisory signal of the upper layer transformer is better posted back to the lower layer transformer, which realizes an effect similar to deep supervisory (strong supervisory), making it easier to train the encoder.

[0092] The adder is used by the encoder 301 to directly calculate the coarsely fused text feature, which keeps the length of the coarsely fused text feature unchanged during the iteration process, and the weights of the two features superimposed by the adder are similar, so that the process of the coarse fusion process is faster.

[0093] The encoder 301 further uses a multi-head attention layer to realize the combined weighted superposition of multiple text feature vectors, and in this learnable manner, combines different hidden text feature vectors with the information fused into the coarse fused text feature vector to obtain a fine fused text feature vector with rich information.

[0094] In this way, the embodiments of the present disclosure use an advanced fusion encoder that can densely connect multiple transformers and a multi-head attention layer to learn a fusion process of text features corresponding to text data, improve the accuracy and richness of the extracted acoustic features, and robustly generate natural synthetic speech.

[0095] Figure 6 is a schematic diagram of a length adjuster 302 according to an embodiment of the present disclosure. Figure 7 is a schematic diagram of a length predictor according to an embodiment of the present disclosure.

[0096] As shown in FIG. 6, the length adjuster 302 includes a length predictor 3021 and a length regularizer 3022 .

[0097] The above step S202 further includes the steps of: predicting the length of the acoustic feature corresponding to the thin-fused text feature by a length predictor 3021 in the length adjuster 302; and adjusting the length corresponding to the thin-fused text feature by a length regularizer 3022 in the length adjuster 302 based on the length of the acoustic feature corresponding to the thin-fused text feature and the thin-fused text feature, thereby generating a length-adjusted thin-fused text feature.

[0098] As shown in FIG. 7, the length predictor 3021 includes a plurality of cascaded 1D convolutional neural networks 701, a bilateral gate cycle unit 702, and a linear layer 703, in which the 1D convolutional neural network 701 extracts relative position features in the thin fused text features, the bilateral gate cycle unit 702 generates global context features based on the relative position features, and the linear layer 703 generates the length of the acoustic feature corresponding to the thin fused text features based on the global context features.

[0099] In addition, the number of the cascaded multiple one-dimensional convolutional neural networks 701 is K (K>0), and the step of predicting the length of the acoustic feature corresponding to the thin-fused text feature by the length predictor 3021 in the length adjuster 302 further includes: extracting a first relative position feature in the thin-fused text feature based on the thin-fused text feature by a first one-dimensional convolutional neural network; generating a Kth relative position feature by a second one-dimensional convolutional neural network to a Kth one-dimensional convolutional neural network, where 2≦k≦K for the kth length prediction layer, and generating a kth relative position feature based on the k-1th relative position feature by the one-dimensional convolutional neural network in the kth length prediction layer; generating a global context feature based on the Kth relative position feature by the bilateral gate cycle unit 702, and generating the length of the acoustic feature corresponding to the thin-fused text feature based on the global context feature by the linear layer 703.

[0100] FIG. 7 shows an example of a length predictor, which includes three cascaded one-dimensional convolutional neural networks 701, one bilateral gate cycle unit 702, and one linear layer 703. The first one-dimensional convolutional neural network takes the above-mentioned thin fusion text feature as input and outputs a first relative position feature in the thin fusion text feature, which includes other information about the text data block included in the thin fusion text feature in addition to the relative position feature in the thin fusion text feature. The second one-dimensional convolutional neural network takes the first relative position feature as input and outputs the second relative position feature, and the third one-dimensional convolutional neural network takes the second relative position feature as input and outputs the third relative position feature. Next, the bilateral gate cycle unit generates a global context feature based on the third relative position feature. Finally, the linear layer generates the length of the acoustic feature corresponding to the thin fusion text feature based on the global context feature.

[0101] In this way, the thin-fused text features are processed by a one-dimensional convolutional neural network, which can extract local position-related information in the thin-fused text features well. Since the length of the acoustic feature is not only related to the relative position information of the text data block, but also to global context information, a bilateral gate cycle unit is used to extract the related information, and the information is mapped to a scalar through a linear layer.

[0102] In this way, the length adjuster 302 solves the problem of the mismatch in length between the thin fusion text features and the acoustic features to be generated. The length adjuster 302 replaces the attention layer between the encoder and decoder in the traditional acoustic model, and solves the effect of instability caused by the attention layer to robustly generate natural synthetic speech.

[0103] 8A is another schematic diagram of an acoustic model according to an embodiment of the present disclosure, FIG 8B is another flowchart of a method according to an embodiment of the present disclosure, and FIG 8C is a schematic diagram of a training flow of the acoustic model of FIG 8A.

[0104] As shown in FIG. 8A, the acoustic model 300 may further include a synthesis layer and a linear layer. In some embodiments, the acoustic model 300 may further include a vocoder. Thus, referring to FIG. 8B, the method 800 may further include steps S204 to S205. In step S204, a synthesis layer combines a plurality of acoustic features corresponding to a plurality of text data blocks to generate a combined acoustic feature, and in step S205, a linear layer linearly adjusts the combined acoustic feature.

[0105] In some embodiments, the linearly adjusted combined acoustic features are directly converted into audio data by a vocoder and output. In some embodiments, the vocoder may be a lightweight neural network that can use LPC-Net technology. As can be appreciated by those skilled in the art, the vocoder may also be other types of vocoders, and the present disclosure is not limited thereto.

[0106] In some embodiments, the encoder, the length adjuster, the decoder, the synthesis layer and the linear layer are trained based on a plurality of training sample sets, each training sample in the plurality of training sample sets including training text data and a target acoustic feature corresponding to the training text data.

[0107] In some embodiments, some of the training samples in the multiple sets of training sample sets may be recorded in a studio by a professional Chinese female speaker. The text content in the training sample set covers multiple fields, the average phrase length is 70 characters, and the number of training phrases is 9600. The number of training phrases is 9600. In some embodiments, another part of the training samples in the multiple sets of training sample sets may be composed of 12000 audio files extracted from online audiobooks, and the audio files may be recorded by an actor with a good rhythmic sense. The text may be a novel, and the average phrase length is 86 characters. The above audio is sampled at 16 kHz and quantized with 16 bits. In some embodiments, another part of the training samples in the multiple sets of training sample sets may be dialogue samples selected from life recordings, for example, covering fields such as politics, sports, entertainment, literature, cooking, etc. These dialogue samples include a large number of hot words and long paragraphs.

[0108] In some embodiments, training the encoder, decoder, synthesis layer, and linear layer includes converting the training text data into the linearly adjusted combined acoustic features by the encoder, decoder, synthesis layer, and linear layer (which in some embodiments further use a length adjuster), calculating a value corresponding to a first target function based on the linearly adjusted combined acoustic features and a target acoustic feature corresponding to the training text data, and adjusting parameters of neurons in the encoder, decoder, synthesis layer, and linear layer based on the value corresponding to the first target function to converge on the value corresponding to the first target function.

[0109] In some embodiments, the first target function indicates the similarity between the linearly adjusted combined acoustic features and the target acoustic features corresponding to the training text data. That is, the training of the acoustic model is a process of maximizing the similarity between the linearly adjusted combined acoustic features and the target acoustic features corresponding to the training text data. The first target function is also called a first loss function.

[0110] Thus, the above training process completes the training of the acoustic model. Any text can be passed through the encoder, the length adjuster, the decoder, the synthesis layer, the linear layer and the first vocoder to obtain a highly natural voice.

[0111] 9A is another schematic diagram of an acoustic model according to an embodiment of the present disclosure, FIG 9B is another flowchart of a method according to an embodiment of the present disclosure, and FIG 9C is a schematic diagram of a training flow of the acoustic model of FIG 9A.

[0112] As shown in FIG. 9A, the acoustic model 300 may further include a post-processing network. In some embodiments, the acoustic model 300 may further include a second vocoder. Thus, as shown in FIG. 9B, the method 900 may further include steps S204 to S207. In step S204, a synthesis layer combines a plurality of acoustic features corresponding to a plurality of text data blocks to generate a combined acoustic feature, in step S205, a linear layer linearly adjusts the combined acoustic feature, in step S206, a post-processing network performs sequence adjustment on the linearly adjusted combined acoustic feature, and in step S207, a vocoder converts the sequence-adjusted combined acoustic feature into audio data and outputs it.

[0113] In some embodiments, the encoder, the length adjuster, the decoder, the synthesis layer, the linear layer and the post-processing network are trained based on a plurality of training sample sets, each training sample in the plurality of training sample sets includes training text data and a target acoustic feature corresponding to the training text data. The plurality of training sample sets may be similar to the embodiments shown in Figures 8A-8C, and it is understood by those skilled in the art that the present disclosure is not limited thereto.

[0114] In some embodiments, training the encoder, the decoder, the synthesis layer, the linear layer, and the post-processing network includes converting the training text data into the sequence-adjusted combined acoustic features by the encoder, the decoder, the synthesis layer, the linear layer, and the post-processing network (which in some embodiments further uses a length adjuster), calculating a value corresponding to a second target function based on the sequence-adjusted combined acoustic features and a target acoustic feature corresponding to the training text data, and adjusting parameters of neurons in the encoder, the decoder, the synthesis layer, the linear layer, and the post-processing network based on the value corresponding to the second target function to converge on the value corresponding to the second target function.

[0115] In some embodiments, the second target function indicates the similarity between the sequence-adjusted combined acoustic features and the target acoustic features corresponding to the training text data. That is, the training of the acoustic model is a process of maximizing the similarity between the sequence-adjusted combined acoustic features and the target acoustic features corresponding to the training text data. The second target function is also called a second loss function.

[0116] In some embodiments, training the encoder, the length adjuster, the decoder, the synthesis layer, the linear layer, and the post-processing network further comprises calculating a weighted sum of values ​​corresponding to the first target function and values ​​corresponding to a second target function, and adjusting parameters of neurons in the encoder, the length adjuster, the decoder, the synthesis layer, the linear layer, and the post-processing network to converge the weights and corresponding values.

[0117] Thus, the above training process completes the training of the acoustic model, and any text can be passed through the encoder, the length adjuster, the decoder, the synthesis layer, the linear layer, the post-processing network and the vocoder to obtain a highly natural voice.

[0118] 10 is a schematic diagram of another training process of an acoustic model according to an embodiment of the present disclosure. FIG 11 is an exemplary structure of a length extractor according to an embodiment of the present disclosure.

[0119] See FIG. 10, the length extractor aids in training the length predictor. See FIG. 11 for an exemplary structure of the length extractor. The length extractor is composed of a length extraction encoder, an attention network, and a length extraction decoder. The time length extracted by the length extractor is used to upsample the output of the length extraction encoder, and is matched to the length of the target acoustic feature during the training process. The output of the length extractor is used as a learning label for the length predictor. Furthermore, since the length extractor is only used to train the length predictor, not the encoder 301, there is no need to post back attenuated gradients to the encoder. In some embodiments, the length extraction decoder is arranged as a two-layer LSTM network (long short-term memory network) to better capture context information. In some embodiments, the length extractor may further include a feed-forward network to adjust the output of the length extraction decoder.

[0120] In some embodiments, the length extraction encoder has a structure similar to that of the encoder, and the decoder is a two-layer autoregressive loop neural network, and the length extractor is trained by the plurality of training sample sets before training the length predictor, and the training of the length extractor includes: generating a thin-fused text feature corresponding to at least one text data block in the training text data by a length extraction encoder in the length extractor; predicting a length of an acoustic feature corresponding to the thin-fused text feature based on the thin-fused text feature by the attention network and the length extraction decoder; calculating a value corresponding to a fourth target function based on the length of the acoustic feature corresponding to the training text data and the length of the acoustic feature corresponding to the predicted thin-fused text feature; and adjusting parameters of neurons in the length extractor based on the value corresponding to the fourth target function to converge the value corresponding to the fourth target function.

[0121] In some embodiments, the fourth target function indicates the similarity between the lengths of the acoustic features corresponding to the training text data and the lengths of the acoustic features corresponding to the predicted thin-fusion text features, i.e., the training of the length extractor is a process of maximizing the similarity between the lengths of the target acoustic features and the lengths of the predicted acoustic features, also called the fourth target function or the fourth loss function.

[0122] After completing the training of the length extractor, the training of a length predictor may be further assisted, where the length predictor is trained based on a plurality of training sample sets, each training sample in the plurality of training sample sets includes training text data and a length of an acoustic feature corresponding to the training text data.

[0123] The training of the length predictor includes the steps of: generating, by a trained length extractor, a length of a target acoustic feature corresponding to at least one text data block in the training text data based on the training text data and lengths of the acoustic features corresponding to the training text data; generating, by a length predictor, a length of a predicted acoustic feature corresponding to at least one text data block in the training text data based on the training text data; calculating a value corresponding to a third target function based on the length of the target acoustic feature and the length of the predicted acoustic feature corresponding to the at least one text data block in the training text data; and adjusting parameters of neurons in the length predictor based on the value corresponding to the third target function to converge the value corresponding to the third target function.

[0124] In some embodiments, the third target function indicates the similarity between the length of the target acoustic feature and the length of the predicted acoustic feature, i.e., training the length predictor is a process of maximizing the similarity between the length of the target acoustic feature and the length of the predicted acoustic feature. The third target function is also called a third loss function.

[0125] Thus, the acoustic model provided by the embodiment of the present disclosure has a fast training and convergence speed and produces good prediction results.

[0126] Below, a comparison is shown of the naturalness of sound between a reading app using the method of an embodiment of the present disclosure and a reading app using Fastspeech or Tacotron. [Table 1] The mean opinion score (MOS) and comparative mean choice score (CMOS) are obtained as follows: In other words, every MOS test invites two groups of people (5 people / group) whose native language is Chinese to listen to and score 125 audios each time, and mixes 100 test phrases synthesized by a corresponding model with 25 original recordings, so that the audience cannot know which category each audio belongs to. The score range is 1 to 5, with 5 representing "completely natural voice". The final MOS is obtained by averaging the two groups. In the above table, the number before the ± sign indicates the average score of the two groups of testers, and the number after the ± sign indicates the variance.

[0127] For the CMOS test, the same audience listened to paired test phrases synthesized by two different systems in random order and rated the comparative perception of the latter with the former on a score scale of [-3,3], where the interval is 1. Using the embodiment of the present disclosure as the benchmark, both the Fastspeech score and the Tacotron score were lower than the embodiment of the present disclosure.

[0128] Thus, the acoustic model according to the embodiment of the present disclosure closes the gap between Fastspeech and Tacotron in naturalness in the two writing styles. The audience prefers the synthesis results of the embodiment of the present disclosure over the synthesis results of the other two systems. As can be seen from these results, the embodiment of the present disclosure obtains a competitive expression in terms of naturalness.

[0129] Table 2 below compares the word error rate (WER) of a reading APP using the method of an embodiment of the present disclosure with a reading APP using Fastspeech or Tacotron to measure the robustness of the embodiment of the present disclosure. [Table 2]

[0130] Robustness is evaluated by measuring the failure rate and / or word error rate (WER). Model errors are mainly recognized by determining whether the synthesized audio is prematurely terminated, whether it overlaps the same snippet, and whether it contains meaningless snippets that severely affect content comprehension. Regarding robustness, due to poor attention alignment, the failure rates of Tacotron and GMM attention in the two databases are 1.5% and 2.8%, respectively. The no-attention model has no severe synthesis failure. General speech recognition performs morphological recognition on the synthesized samples and has no severe failure. Based on the word error rate (WER) results recorded in Table 2, both the embodiments of the present disclosure obtain the lowest word error rate, whether they are trained using a studio-quality corpus or a challenging audiobook corpus.

[0131] Table 3 below shows a comparison of the naturalness of sound between a reading APP using the method of the embodiment of the present disclosure and other types of reading APP. [Table 3]

[0132] Table 4 below shows a comparison of word error rates between a reading APP using the method of the embodiment of the present disclosure and other types of reading APPs. [Table 4]

[0133] As shown in Table 3, replacing the encoder 301 with a conventional RNN-based CBHG (convolutional bank, highway network, and bidirectional gated recurrent unit) encoder reduces the MOS in the test results by 1.7% and 5.2%, respectively. For CMOS, the audience likes the results synthesized by the acoustic model according to the embodiment of the present disclosure more than the results after replacement. As shown in Table 4, even if the CBHG encoder is used, the WER value is higher. Thus, the output of the encoder 301 contains more linguistic information due to the coarse fusion process and the fine fusion process. While the encoder 301 contributes to improving the naturalness, the output of the encoder 301 is more suitable for a non-autoregressive RNN-based decoder.

[0134] As shown in Table 3, when the non-autoregressive RNN decoder 303 is replaced by the autoregressive decoder (ARD), the model after the replacement is obviously less natural and robust. Especially in the more challenging audiobook corpus, there are obvious word skips and mispronunciations in some domain phrases in the test results. Thus, the decoder 303 obviously reduces the error rate caused by the exposure bias and local information preference of the autoregressive decoder.

[0135] According to another aspect of the present disclosure, there is further provided an electronic device for implementing the method according to the embodiment of the present disclosure. Figure 12 is a schematic diagram of an electronic device 2000 according to the embodiment of the present disclosure.

[0136] 12, the electronic device 2000 includes one or more processors 2010, and one or more memories 2020. The memory 2020 stores computer readable code that, when executed by the one or more processors 2010, performs the search request processing method described above.

[0137] The processor in the embodiments of the present disclosure may be an integrated circuit chip having signal processing capabilities. The processor may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The embodiments of the present disclosure may implement or perform each method, step, and logic block diagram disclosed. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor of X86 architecture or ARM architecture, etc.

[0138] In general, various exemplary embodiments of the disclosure may be implemented in hardware or special purpose circuits, software, firmware, logic, or any combination thereof. Some aspects are implemented in hardware, while other aspects are implemented in firmware or software executed by a controller, microprocessor, or other computing device. When aspects of the embodiments of the disclosure are shown or described as block diagrams, flow charts, or other graphical representations, the blocks, devices, systems, techniques, or methods described herein may be implemented in, by way of non-limiting example only, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller, or other computing device, or some combination thereof.

[0139] For example, the method or apparatus according to the embodiment of the present disclosure may be realized by the architecture of a computing device 3000 in FIG. 13. As shown in FIG. 13, the computing device 3000 includes a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, stores various data or files used in the processing and / or communication of the method provided by the present disclosure, and program instructions executed by the CPU. The computing device 3000 further includes a user interface 3080. Of course, the architecture in FIG. 13 is merely exemplary, and when implementing a different device, one or more components in the computing device in FIG. 13 may be omitted according to actual needs.

[0140] According to another aspect of the present disclosure, there is further provided a computer readable storage medium. Figure 14 is a schematic diagram of a storage medium 4000 according to the present disclosure.

[0141] As shown in FIG. 14, the computer storage medium 4020 stores computer readable instructions 4010. When the computer readable instructions 4010 are executed by a processor, it performs the method according to the embodiment of the present disclosure described in the above figures. The computer readable storage medium in the embodiment of the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM) used as an external high speed cache. By way of example and non-limiting illustration, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), dynamic dynamic random access memory (SDRAM), double data rate dynamic random access memory (DDRSDRAM), enhanced dynamic dynamic random access memory (ESDRAM), dynamically connected dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM), where memory in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0142] An embodiment of the present disclosure further provides a computer program product or computer program, the computer program product or computer program including computer instructions, the computer instructions being stored in a computer readable storage medium, a processor of a computing device reading the computer instructions from the computer readable storage medium, and the processor executing the computer instructions, causing the computing device to perform a method according to an embodiment of the present disclosure.

[0143] Here, the flowcharts and block diagrams in the drawings show possible system architectures, functions and operations of the systems, methods and computer program products of various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams represents a module, program segment or part of code, which includes one or more executable instructions for implementing a certain logical function. In addition, in some alternative implementations, the functions described in the blocks may occur in a different order from that shown in the drawings. For example, in practice, two blocks shown in succession may essentially be executed in parallel, or in some cases may be executed in a reverse order, depending on the functions involved. In addition, each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts may be realized by a specialized system with hardware that executes a certain function or operation, or may be realized by a combination of specialized hardware and computer instructions.

[0144] In general, various exemplary embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, firmware, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software executed by a controller, microprocessor, or other computing device. When aspects of the embodiments of the present disclosure are shown or described as block diagrams, flow charts, or other graphical representations, the blocks, devices, systems, techniques, or methods described herein may be implemented in, by way of non-limiting example only, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller, or other computing device, or some combination thereof.

[0145] The above-described exemplary embodiments of the present disclosure are for illustrative purposes only and not for limiting purposes. As can be understood by those skilled in the art, various modifications and combinations may be made to these embodiments or features thereof without departing from the principle and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure. [Explanation of symbols]

[0146] 100 Mystery Scenarios 200 ways 300 equipment 300 Acoustic Models 301 encoder 302 Regulator 303 Decoder 701 Neural Networks 702 Bilateral Gate Cycle Unit 703 Linear Layer 800 ways 900 ways 2000 Electronic equipment 2010 Processors 2020 Memory 3000 Computing Equipment 3010 Bus 3020 CPU 3021 Predictor 3022 Regularizer 3030 Dedicated memory (ROM) 3030 ROM 3040 Random Access Memory (RAM) 3050 communication port 3060 Output Component 3070 Hard Disk 3080 User Interface 4000 storage medium 4010 Computer Readable Instructions 4020 Computer storage media

Claims

1. 1. An electronic device-implemented method for converting text data into acoustic features, the text data including a plurality of contiguous blocks of text data, the method comprising: For each text data block of the plurality of text data blocks, obtaining hidden text features corresponding to the text data block and hidden text features corresponding to at least one text data block subsequent to the text data block by a neural network model, the hidden text features being non-final output feature vectors generated from the neural network model; performing a coarse fusion process on the hidden text features corresponding to the text data block and the hidden text features corresponding to at least one text data block subsequent to the text data block to obtain at least one coarse fusion text feature; performing a fine fusion process on the at least one coarse fusion text feature to obtain a fine fusion text feature corresponding to the text data block; adjusting the length corresponding to the thin fusion text feature; and converting the length-adjusted thin fused text features into acoustic features.

2. a hidden text feature corresponding to the text data block is generated by fusing the first prediction information and the second prediction information of the text data block; 2. The method of claim 1, wherein the first prediction information is predicted based on the block of text data, and the second prediction information is predicted based on at least one block of text data subsequent to the block of text data.

3. 3. The method of claim 2, wherein the first prediction information includes one or more of Pinyin information, phoneme information, semantic information and emotion information extracted based on the block of text data, and the second prediction information includes one or more of emotion information and speech rate information extracted based on at least one text data block subsequent to the block of text data.

4. The coarse fusion process is performed by an encoder, the encoder including N transformers and N adders; performing a coarse fusion process on the hidden text features corresponding to the text data block and the hidden text features corresponding to at least one text data block subsequent to the text data block to obtain at least one coarse fusion text feature; 2. The method of claim 1 , further comprising: generating, by the N transformers and the N adders, N coarse-fused text features based on hidden text features corresponding to the text data block and hidden text features corresponding to at least one text data block subsequent to the text data block, where N is a positive integer.

5. The step of performing a fine fusion process on the at least one coarse fusion text feature comprises:

2. The method of claim 1, further comprising: setting one or more of the at least one coarsely fused text feature as a search feature; setting hidden text features corresponding to the text data block and hidden text features corresponding to at least one text data block subsequent to the text data block as value features and key features, respectively; and predicting finely fused text features corresponding to the text data block based on the search features, value features and key features.

6. the coarse merging process and the fine merging process are performed by an encoder; The encoder includes N transformers, N adders, and one multi-head attention layer; The output of the nth transformer and the output of the n-1th adder are both connected to the input of the nth adder, and the output of the nth adder is connected to the input of the n+1th transformer, and the input of the multi-head attention layer includes the outputs of the N transformers and the output of the Nth adder; Each transformer outputs one hidden text feature, each adder outputs one coarse fused text feature, and the multi-head attention layer of the encoder outputs a fine fused text feature corresponding to the block of text data; 2. The method of claim 1, wherein 1≦n≦N, and N is a positive integer greater than 1.

7. A step of adjusting the length corresponding to the thin fusion text feature by a length adjuster is performed, the step of adjusting the length corresponding to the thin fusion text feature includes: predicting the length of the acoustic feature corresponding to the thin fused text feature by a length predictor in a length adjuster; 2. The method of claim 1, further comprising: adjusting lengths corresponding to the thin-fused text features by a length regularizer in a length adjuster based on lengths of the acoustic features corresponding to the thin-fused text features and the thin-fused text features to generate length-adjusted thin-fused text features.

8. The length predictor includes a plurality of cascaded one-dimensional convolutional neural networks, a bilateral gate cycle unit, and a first linear layer; The one-dimensional convolutional neural network is for extracting a relative position feature in the thin fusion text feature, the relative position feature indicating a relative position information in the text data of the text data block; the bilateral gate cycle unit generates global context features based on the relative position features; The method of claim 7 , wherein the first linear layer generates lengths of acoustic features corresponding to the thin fused text features based on the global context features.

9. The number of the cascaded one-dimensional convolutional neural networks is K (K>0), and the step of predicting the length of the acoustic feature corresponding to the thin fused text feature by a length predictor in the length adjuster includes: extracting a first relative position feature in the thin fusion text features based on the thin fusion text features by a first one-dimensional convolutional neural network; generating K-th relative position features by a second one-dimensional convolutional neural network through a K-th one-dimensional convolutional neural network, where 2≦k≦K for the k-th length prediction layer; generating a k-th relative position feature based on the k-1-th relative position feature by a one-dimensional convolutional neural network in a k-th length prediction layer; generating, by the bilateral gate cycle unit, a global context feature based on the Kth relative position feature; The method of claim 8 , further comprising: generating, by the first linear layer, lengths of acoustic features corresponding to the thin fused text features based on global context features.

10. The length predictor is trained based on a plurality of training sample sets, each training sample in the plurality of training sample sets including training text data and a length of an acoustic feature corresponding to the training text data; Training the length predictor comprises: generating, by a post-training length extractor, a length of a target acoustic feature corresponding to at least one block of text data in the training text data based on the training text data and lengths of acoustic features corresponding to the training text data; generating, by a length predictor, a length of a predicted acoustic feature corresponding to at least one block of text data in the training text data based on the training text data; calculating a value corresponding to a third target function based on a length of a target acoustic feature and a length of a predicted acoustic feature corresponding to at least one block of text data in the training text data; and adjusting parameters of neurons in the length predictor based on a value corresponding to the third target function to converge on a value corresponding to the third target function.

11. The coarse merging process and the fine merging process are performed by an encoder; The length extractor includes a length extraction encoder, an attention network, and a length extraction decoder, the length extraction encoder having a structure similar to that of the encoder, the decoder being a two-layer autoregressive loop neural network, and the length extractor is trained by the plurality of training sample sets before training the length predictor; Training the length extractor comprises: generating, by a length extractor coder in the length extractor, a thin merged text feature corresponding to at least one block of text data in the training text data; predicting, by the attention network and the length extractor decoder, the length of the acoustic feature corresponding to the thin fusion text feature based on the thin fusion text feature; calculating a value corresponding to a fourth target function based on lengths of acoustic features corresponding to the training text data and lengths of acoustic features corresponding to the predicted thin fusion text features; and adjusting parameters of neurons in the length extractor based on a value corresponding to the fourth target function to converge on a value corresponding to the fourth target function.

12. 2. The method of claim 1, further comprising converting the length-adjusted thin-fused text features into acoustic features by a decoder, the decoder being a single-layer non-autoregressive loop neural network, the input of the decoder being the length-adjusted thin-fused text features, and the output of the decoder being acoustic features.

13. combining, by a synthesis layer, a plurality of acoustic features corresponding to a plurality of text data blocks to generate a combined acoustic feature; The method of claim 12 , further comprising: linearly adjusting the combined acoustic features by a second linear layer.

14. performing sequence adjustment on the linearly adjusted combined acoustic features by a post-processing network; The method of claim 13 , further comprising: converting the sequence-adjusted combined acoustic features into audio data and outputting the audio data by a vocoder.

15. The coarse fusion process and the fine fusion process are performed by an encoder, and the encoder, the decoder, the synthesis layer and the second linear layer are trained based on a plurality of training sample sets, each training sample in the plurality of training sample sets includes training text data and a target acoustic feature corresponding to the training text data; The training of the encoder, the decoder, the synthesis layer, and the second linear layer comprises: converting the training text data into the linearly adjusted combined acoustic features by the encoder, the decoder, the synthesis layer and the second linear layer; calculating a value corresponding to a first target function based on the linearly adjusted combined acoustic feature and a target acoustic feature corresponding to the training text data; and adjusting parameters of neurons in the encoder, the decoder, the synthesis layer, and the second linear layer based on a value corresponding to the first target function to converge on a value corresponding to the first target function.

16. The coarse fusion process and the fine fusion process are performed by an encoder, and the encoder, the decoder, the synthesis layer, the second linear layer and the post-processing network are trained based on a plurality of training sample sets, each training sample in the plurality of training sample sets includes training text data and a target acoustic feature corresponding to the training text data; The training of the encoder, the decoder, the synthesis layer, the second linear layer and the post-processing network comprises: converting the training text data into the sequence-aligned combined acoustic features by the encoder, the decoder, the synthesis layer, the second linear layer and the post-processing network; calculating a value corresponding to a second target function based on the sequence-adjusted combined acoustic features and target acoustic features corresponding to the training text data; and adjusting parameters of neurons in the encoder, the decoder, the synthesis layer, the second linear layer and the post-processing network based on a value corresponding to the second target function to converge on a value corresponding to the second target function.

17. An electronic device, one or more processors; one or more memories; An electronic device, the memory storing a computer executable program, the computer executable program being operable, when executed by the processor, to perform the method of any one of claims 1 to 16.

18. A computer program having computer executable instructions stored thereon which, when executed by a processor, implements the method according to any one of claims 1 to 16.

Images