Audio generation method, device, equipment and storage medium based on artificial intelligence

Abstract

The present application provides an audio generation method, device, electronic device and computer-readable storage medium based on artificial intelligence; it relates to artificial intelligence technology; the method includes: encoding a phoneme sequence corresponding to a text to obtain a context representation of the phoneme sequence; determining the alignment position of the first frame implicit state relative to the context representation based on the first frame implicit state corresponding to each phoneme in the phoneme sequence; when the alignment position corresponds to a non-end position in the context representation, decoding is performed based on the context representation and the first frame implicit state to obtain a second frame implicit state; synthesis is performed based on the first frame implicit state and the second frame implicit state to obtain audio data corresponding to the text. Through the present application, the accuracy of audio synthesis can be improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to an audio generation method, device, electronic device and computer-readable storage medium based on artificial intelligence. Background Art

[0002] Artificial Intelligence (AI) is a comprehensive technology in computer science. By studying the design principles and implementation methods of various intelligent machines, machines are given the functions of perception, reasoning and decision-making. Artificial intelligence technology is an interdisciplinary subject that covers a wide range of fields, such as natural language processing technology and machine learning / deep learning. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0003] The audio synthesis method in the related art is relatively rough, and usually the frequency spectrum corresponding to the text data is directly synthesized to obtain the audio data corresponding to the text data. This synthesis method cannot achieve accurate audio synthesis, thereby affecting the normal audio synthesis experience of the user. Summary of the invention

[0004] The embodiments of the present application provide an audio generation method, device, electronic device and computer-readable storage medium based on artificial intelligence, which can improve the accuracy of audio synthesis.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application embodiment provides an audio generation method based on artificial intelligence, including:

[0007] Encoding the phoneme sequence corresponding to the text to obtain a context representation of the phoneme sequence;

[0008] Determining an alignment position of the first frame hidden state relative to the context representation based on the first frame hidden state corresponding to each phoneme in the phoneme sequence;

[0009] When the alignment position corresponds to a non-end position in the context representation, decoding is performed based on the context representation and the first frame implicit state to obtain a second frame implicit state;

[0010] Synthesis processing is performed based on the implicit state of the first frame and the implicit state of the second frame to obtain audio data corresponding to the text.

[0011] In the above technical solution, the encoding process of the phoneme sequence corresponding to the text to obtain the context representation of the phoneme sequence includes:

[0012] Performing forward encoding processing on the phoneme sequence corresponding to the text to obtain a forward latent vector of the phoneme sequence;

[0013] Performing backward encoding processing on the phoneme sequence corresponding to the text to obtain a backward latent vector of the phoneme sequence;

[0014] The forward latent vector and the backward latent vector are fused to obtain a context representation of the phoneme sequence.

[0015] In the above technical solution, the forward encoding process is performed on the phoneme sequence corresponding to the text to obtain the forward latent vector of the phoneme sequence, including:

[0016] By means of an encoder, each phoneme in the phoneme sequence corresponding to the text is encoded in sequence along a first direction to obtain a latent vector of each phoneme in the first direction;

[0017] The backward encoding process is performed on the phoneme sequence corresponding to the text to obtain a backward latent vector of the phoneme sequence, including:

[0018] By means of the encoder, each of the phonemes is encoded in sequence along a second direction to obtain a latent vector of each phoneme in the second direction;

[0019] The fusing the forward latent vector and the backward latent vector to obtain a context representation of the phoneme sequence includes:

[0020] Concatenating the forward latent vector and the backward latent vector to obtain a context representation of the phoneme sequence;

[0021] The second direction is the opposite direction of the first direction.

[0022] The present application provides an audio generation device, including:

[0023] An encoding module, used to encode the phoneme sequence corresponding to the text to obtain a context representation of the phoneme sequence;

[0024] an attention module, configured to determine an alignment position of a first frame hidden state relative to the context representation based on a first frame hidden state corresponding to each phoneme in the phoneme sequence;

[0025] A decoding module, configured to, when the alignment position corresponds to a non-end position in the context representation, perform decoding processing based on the context representation and the first frame implicit state to obtain a second frame implicit state;

[0026] A synthesis module is used to perform synthesis processing based on the implicit state of the first frame and the implicit state of the second frame to obtain audio data corresponding to the text.

[0027] In the above technical solution, the first frame implicit state represents the implicit state of the first frame, the second frame implicit state represents the implicit state of the second frame, and the first frame and the second frame are any two adjacent frames in the spectrum data corresponding to the phoneme;

[0028] When the first frame hidden state is recorded as the t-th frame hidden state, the attention module is further used to perform the following processing on each phoneme in the phoneme sequence:

[0029] Determining, based on the t-th frame hidden state corresponding to the phoneme, an alignment position of the t-th frame hidden state relative to the context representation;

[0030] The decoding module is further configured to perform decoding processing based on the context representation and the tth frame implicit state to obtain the t+1th frame implicit state when the alignment position of the tth frame implicit state relative to the context representation corresponds to a non-end position in the context representation;

[0031] Wherein, t is a natural number that increases from 1 and satisfies 1≤t≤T, T is the total number of frames corresponding to the phoneme sequence when the alignment position corresponds to the end position in the context representation, and the total number of frames represents the number of frames of spectral data corresponding to the implicit state of each phoneme in the phoneme sequence.

[0032] In the above technical solution, the synthesis module is further used to perform splicing processing on the implicit state of the T frame when the alignment position corresponds to the end position in the context representation to obtain the implicit state corresponding to the text;

[0033] Performing smoothing on the implicit state corresponding to the text to obtain spectrum data corresponding to the text;

[0034] Perform Fourier transform on the frequency spectrum data corresponding to the text to obtain audio data corresponding to the text.

[0035] In the above technical solution, the attention module is further used to perform Gaussian prediction processing based on the implicit state of the t-th frame corresponding to the phoneme to obtain the t-th Gaussian parameter corresponding to the implicit state of the t-th frame;

[0036] An alignment position of the hidden state of the t-th frame relative to the context representation is determined based on the t-th Gaussian parameter.

[0037] In the above technical solution, the attention module is further used to perform prediction processing based on the Gaussian function based on the implicit state of the t-th frame corresponding to the phoneme, and obtain the t-th Gaussian variance and the t-th Gaussian mean change corresponding to the implicit state of the t-th frame;

[0038] Get the t-1th Gaussian parameter corresponding to the hidden state of the t-1th frame;

[0039] Adding the t-1th Gaussian mean value and the tth Gaussian mean change amount included in the t-1th Gaussian parameter, and using the summed result as the tth Gaussian mean value corresponding to the tth frame implicit state;

[0040] Taking the set of the t-th Gaussian variance and the t-th Gaussian mean as the t-th Gaussian parameter corresponding to the hidden state of the t-th frame;

[0041] The t-th Gaussian mean is used as the alignment position of the t-th frame hidden state relative to the context representation.

[0042] In the above technical solution, the attention module is also used to obtain the content text length of the context representation of the phoneme sequence;

[0043] When the t-th Gaussian mean is greater than the content text length, determining that the alignment position corresponds to the end position in the context representation;

[0044] When the t-th Gaussian mean is less than or equal to the content text length, it is determined that the alignment position corresponds to a non-end position in the context representation.

[0045] In the above technical solution, the decoding module is also used to obtain the attention weight corresponding to the implicit state of the tth frame;

[0046] Performing weighted processing on the context representation based on the attention weight to obtain a context vector corresponding to the context representation;

[0047] A state prediction process is performed based on the context vector and the implicit state of the t-th frame to obtain an implicit state of the t+1-th frame.

[0048] In the above technical solution, the attention module is further used to obtain the tth Gaussian parameter corresponding to the implicit state of the tth frame, wherein the tth Gaussian parameter includes the tth Gaussian variance and the tth Gaussian mean;

[0049] Gaussian processing is performed on the context representation based on the t-th Gaussian variance and the t-th Gaussian mean to obtain an attention weight corresponding to the implicit state of the t-th frame.

[0050] In the above technical solution, the audio generation method is implemented by calling a neural network model; the device also includes:

[0051] A training module, used for encoding a phoneme sequence sample corresponding to a text sample through the initialized neural network model to obtain a context representation of the phoneme sequence sample;

[0052] Determining a predicted alignment position of the third frame hidden state relative to the context representation based on the third frame hidden state corresponding to each phoneme in the phoneme sequence sample;

[0053] When the predicted alignment position corresponds to a non-end position in the context representation, decoding is performed based on the context representation and the third frame implicit state to obtain a fourth frame implicit state;

[0054] Performing spectrum post-processing based on the third frame implicit state and the fourth frame implicit state to obtain predicted spectrum data corresponding to the text sample;

[0055] Constructing a loss function of the neural network model based on the predicted spectrum data corresponding to the text sample and the labeled spectrum data corresponding to the text sample;

[0056] Updating the parameters of the neural network model, and using the updated parameters of the neural network model when the loss function converges as the parameters of the trained neural network model;

[0057] The third frame implicit state represents the implicit state of the third frame, the fourth frame implicit state represents the implicit state of the fourth frame, and the third frame and the fourth frame are any two adjacent frames in the spectrum data corresponding to each phoneme in the phoneme sequence sample.

[0058] In the above technical solution, the training module is also used to construct a parameter matrix based on the parameters of the neural network model;

[0059] Dividing the parameter matrix into blocks to obtain a plurality of matrix blocks included in the parameter matrix;

[0060] When the structure is sparse, the mean of the parameters in each of the matrix blocks is determined;

[0061] Sorting the matrix blocks in ascending order based on the mean value of the parameters in each of the matrix blocks, resetting the parameters in the plurality of matrix blocks that are sorted first in the ascending order result, and obtaining a reset parameter matrix;

[0062] The reset parameter matrix is ​​used to update the parameters of the neural network model.

[0063] In the above technical solution, the training module is also used to obtain the content text length of the context representation of the phoneme sequence sample;

[0064] When the predicted alignment position corresponds to the end position in the context representation, constructing a position loss function of the neural network model based on the predicted alignment position and the content text length;

[0065] Constructing a spectrum loss function of the neural network model based on the predicted spectrum data corresponding to the text sample and the labeled spectrum data corresponding to the text sample;

[0066] A weighted sum is performed on the spectrum loss function and the position loss function, and the weighted sum result is used as the loss function of the neural network model.

[0067] In the above technical solution, the encoding module is also used to perform forward encoding processing on the phoneme sequence corresponding to the text to obtain a forward latent vector of the phoneme sequence;

[0068] Performing backward encoding processing on the phoneme sequence corresponding to the text to obtain a backward latent vector of the phoneme sequence;

[0069] The forward latent vector and the backward latent vector are fused to obtain a context representation of the phoneme sequence.

[0070] In the above technical solution, the encoding module is further used to encode each phoneme in the phoneme sequence corresponding to the text in the first direction in sequence through an encoder to obtain a latent vector of each phoneme in the first direction;

[0071] By means of the encoder, each of the phonemes is encoded in sequence along a second direction to obtain a latent vector of each phoneme in the second direction;

[0072] Concatenating the forward latent vector and the backward latent vector to obtain a context representation of the phoneme sequence;

[0073] The second direction is the opposite direction of the first direction.

[0074] An embodiment of the present application provides an electronic device for audio generation, the electronic device comprising:

[0075] A memory for storing executable instructions;

[0076] The processor is used to implement the artificial intelligence-based audio generation method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0077] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the artificial intelligence-based audio generation method provided in the embodiment of the present application.

[0078] The embodiments of the present application have the following beneficial effects:

[0079] By determining the alignment position of the implicit state relative to the context representation, subsequent decoding operations are performed based on the accurate alignment position, thereby achieving accurate audio generation based on the accurate implicit state. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 is a schematic diagram of an application scenario of the audio generation system provided in an embodiment of the present application;

[0081] Figure 2 is a schematic diagram of the structure of an electronic device for audio generation provided in an embodiment of the present application;

[0082] Figure 3-5 It is a flowchart of an audio generation method based on artificial intelligence provided in an embodiment of the present application;

[0083] Figure 6 is a coding schematic diagram of a content encoder provided in an embodiment of the present application;

[0084] Figure 7 It is a flowchart of an audio generation method based on artificial intelligence provided in an embodiment of the present application;

[0085] Figure 8 is a schematic diagram of the alignment position provided in the embodiment of the present application corresponding to the end position in the context representation;

[0086] Fig. 9 is a schematic diagram of the alignment position provided in the embodiment of the present application corresponding to the non-end position in the context representation;

[0087] Fig.10 is a schematic diagram of a parameter matrix provided in an embodiment of the present application;

[0088] Fig.11 It is a schematic diagram of the training process of the end-to-end speech synthesis acoustic model provided in the embodiment of the present application;

[0089] Fig.12 It is a schematic diagram of the reasoning process of the end-to-end speech synthesis acoustic model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0091] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0092] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0093] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0094] 1) Convolutional Neural Networks (CNN): A type of feedforward neural network (FNN) that includes convolutional calculations and has a deep structure. It is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities and can perform shift-invariant classification on input images according to their hierarchical structure.

[0095] 2) Recurrent Neural Network (RNN): A type of recurrent neural network that takes sequence data as input, performs recursion in the direction of sequence evolution, and all nodes (recurrent units) are connected in a chain. Recurrent neural networks have memory, parameter sharing, and Turing completeness, so they have certain advantages when learning nonlinear features of sequences.

[0096] 3) Phoneme: The smallest basic unit of speech. Phoneme is the basis for humans to distinguish one word from another. Phonemes form syllables, and syllables form different words and phrases.

[0097] 4) Hidden state: The decoder (e.g., hidden Markov model) outputs a sequence used to characterize the spectral data. By smoothing the hidden state, the corresponding spectral data can be obtained. Since the audio signal is a non-stationary signal in a long period of time (e.g., more than one second), it can be approximated as a stationary signal in a short period of time (e.g., 50 milliseconds). The characteristic of a stationary signal is that the spectral distribution of the signal is stable, and the spectral distribution in different time periods is similar. The hidden Markov model classifies the continuous signal corresponding to a small segment of similar spectrum into a hidden state. The hidden state is the actual implicit state in the Markov model, and it is a sequence used to characterize the spectral data that cannot be obtained through direct observation. The training process of the hidden Markov model is to maximize the likelihood. The data generated by each hidden state is represented by a probability distribution. Only when similar continuous signals are classified as the same state can the likelihood be as large as possible.

[0098] In the embodiment of the present application, the first frame implicit state represents the implicit state of the first frame, and the second frame implicit state represents the implicit state of the second frame, wherein the first frame and the second frame are any two adjacent frames in the spectrum data corresponding to the phoneme.

[0099] 5) Contextual representation: The vector sequence output by the encoder that is used to represent the contextual content of the text.

[0100] 6) End position: the position after the last data (such as phoneme, word, phrase, etc.) in the text. For example, if the phoneme sequence corresponding to a text has 5 phonemes, then position 0 represents the starting position of the phoneme sequence, position 1 represents the position of the first phoneme in the phoneme sequence, ..., position 5 represents the position of the fifth phoneme in the phoneme sequence, and position 6 represents the end position in the phoneme sequence, where positions 0-5 represent non-end positions in the phoneme sequence.

[0101] 7) Mean Absolute Error (MAE): Also known as L1 Loss, the average value of the distance between the model predicted value f(x) and the true value y.

[0102] 8) Block Sparsity: The weights are first divided into blocks during the training process, and then sorted according to the average absolute value of the parameters in each block each time the parameters are updated, and the weights on the blocks with smaller absolute values ​​are reset to 0.

[0103] 9) Synthesis real-time rate: one second of audio and the computer running time required to synthesize the one second of audio. For example, if it takes 100 milliseconds of computer running time to synthesize 1 second of audio, the synthesis real-time rate is 10 times.

[0104] The embodiments of the present application provide an audio generation method, device, electronic device and computer-readable storage medium based on artificial intelligence, which can improve the accuracy of audio synthesis.

[0105] The artificial intelligence-based audio generation method provided in the embodiments of the present application can be implemented by the terminal / server alone; it can also be implemented by the terminal and the server in collaboration. For example, the terminal alone undertakes the artificial intelligence-based audio generation method described below, or the terminal sends a request for audio generation (including the text of the audio to be generated) to the server, and the server executes the artificial intelligence-based audio generation method according to the received request for audio generation. In response to the request for audio generation, when the alignment position corresponds to a non-end position in the context representation, decoding processing is performed based on the context representation and the implicit state of the first frame to obtain the implicit state of the second frame, and synthesis processing is performed based on the implicit state of the first frame and the implicit state of the second frame to obtain audio data corresponding to the text, thereby realizing intelligent and accurate generation of audio.

[0106] The electronic device for audio generation provided in the embodiments of the present application may be various types of terminal devices or servers, wherein the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services; the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the present application.

[0107] Taking the server as an example, it can be a server cluster deployed in the cloud, opening artificial intelligence cloud services (AI as a Service, AIaaS) to users. The AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to an AI theme mall. All users can access one or more artificial intelligence services provided by the AIaaS platform through an application programming interface.

[0108] For example, one of the artificial intelligence cloud services can be an audio generation service, that is, the server in the cloud encapsulates the audio generation program provided by the embodiment of the present application. The user calls the audio generation service in the cloud service through a terminal (running a client, such as an audio client, a car client, etc.), so that the server deployed in the cloud calls the encapsulated audio generation program, and when the alignment position corresponds to the non-end position in the context representation, decoding processing is performed based on the context representation and the implicit state of the first frame to obtain the implicit state of the second frame, and synthesis processing is performed based on the implicit state of the first frame and the implicit state of the second frame to obtain the audio data corresponding to the text.

[0109] As an application example, for an audio client, the user may be a broadcaster on a broadcasting platform, who needs to regularly broadcast precautions, life tips, etc. to residents in the community. For example, the broadcaster inputs a text in the audio client, which needs to be converted into audio to be broadcast to residents in the community. In the process of converting text into audio, the alignment position of the implicit state relative to the context representation of the phoneme sequence corresponding to the text is continuously judged, and subsequent decoding operations are performed based on the accurate alignment position, thereby achieving accurate audio generation based on the accurate implicit state, so as to broadcast the generated audio to residents.

[0110] As another application example, for an in-vehicle client, when a user is driving, it is not convenient to obtain information in the form of text, but the user can obtain information by reading audio to avoid missing important information. For example, when a user is driving, the leader sends a text of an important meeting to the user, and the user needs to read and process the text in a timely manner. After the in-vehicle client receives the text, it needs to convert the text into audio to play the audio to the user. In the process of converting the text to audio, the alignment position of the implicit state relative to the context representation of the phoneme sequence corresponding to the text is continuously judged, and subsequent decoding operations are performed based on the accurate alignment position, thereby achieving accurate audio generation based on the accurate implicit state, and playing the generated audio to the user, so that the user can read the audio in a timely manner.

[0111] See also Figure 1 , Figure 1 It is a schematic diagram of an application scenario of the audio generation system 10 provided in an embodiment of the present application. The terminal 200 is connected to the server 100 via a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0112] Terminal 200 (running a client, such as an audio client, a car client, etc.) can be used to obtain a request for generating audio. For example, when a user inputs the text of the audio to be generated through terminal 200, terminal 200 automatically obtains the text of the audio to be generated and automatically generates a request for generating audio.

[0113] In some embodiments, an audio generation plug-in may be implanted in the client running in the terminal to implement an audio generation method based on artificial intelligence locally on the client. For example, after the terminal 200 obtains a generation request for audio (including the text of the audio to be generated), the audio generation plug-in is called to implement an audio generation method based on artificial intelligence. When the alignment position corresponds to a non-end position in the context representation, a decoding process is performed based on the context representation and the first frame implicit state to obtain the second frame implicit state, and a synthesis process is performed based on the first frame implicit state and the second frame implicit state to obtain audio data corresponding to the text, thereby realizing intelligent and accurate generation of audio. For example, for a recording application, the user cannot perform high-quality personalized sound customization in a non-recording studio scenario, then enter a text to be recorded in the recording client, and the text needs to be converted into personalized audio. By continuously judging the alignment position of the implicit state relative to the context representation of the phoneme sequence corresponding to the text in the process of converting the text into audio, a subsequent decoding operation is performed based on the accurate alignment position, thereby generating accurate personalized audio based on the accurate implicit state, so as to realize personalized sound customization in a non-recording studio scenario.

[0114] In some embodiments, after the terminal 200 obtains the generation request for the audio, it calls the audio generation interface of the server 100 (which can be provided in the form of a cloud service, i.e., an audio generation service). When the alignment position corresponds to a non-end position in the context representation, the server 100 performs decoding processing based on the context representation and the implicit state of the first frame to obtain the implicit state of the second frame, performs synthesis processing based on the implicit state of the first frame and the implicit state of the second frame to obtain audio data corresponding to the text, and sends the audio data to the terminal 200. For example, for a recording application, the user cannot perform high-quality personalized sound customization in a non-recording studio scenario. In this case, the user enters a text to be recorded in the terminal 200, and automatically generates a generation request for the audio, and sends the generation request for the audio to the server 100. The server 100 continuously determines the alignment position of the implicit state relative to the context representation of the phoneme sequence corresponding to the text during the process of converting the text into audio, so as to perform subsequent decoding operations based on the accurate alignment position, thereby generating accurate personalized audio based on the accurate implicit state, and sending the generated personalized audio to the terminal 200 in response to the generation request for the audio, thereby realizing personalized sound customization in a non-recording studio scenario.

[0115] The structure of the electronic device for audio generation provided by the embodiment of the present application is described below. Figure 2 , Figure 2 is a structural diagram of an electronic device 500 for audio generation provided in an embodiment of the present application, and takes the electronic device 500 as a server as an example. Figure 2The electronic device 500 for audio generation shown includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not described in detail. Figure 2 Various buses are labeled as bus system 540 .

[0116] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0117] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiment of the present application is intended to include any suitable type of memory. The memory 550 optionally includes one or more storage devices that are physically far away from the processor 510.

[0118] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0119] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0120] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0121] In some embodiments, the audio generation device provided in the embodiments of the present application can be implemented in software, for example, it can be an audio generation plug-in in the terminal described above, or it can be an audio generation service in the server described above. Of course, it is not limited to this, and the audio generation device provided in the embodiments of the present application can be provided as various software embodiments, including various forms including application programs, software, software modules, scripts or codes.

[0122] Figure 2 An audio generating device 555 stored in a memory 550 is shown, which may be software in the form of a program and a plug-in, such as an audio generating plug-in, and includes a series of modules, including an encoding module 5551, an attention module 5552, a decoding module 5553, a synthesis module 5554, and a training module 5555; wherein the encoding module 5551, the attention module 5552, the decoding module 5553, and the synthesis module 5554 are used to implement the audio generation function provided in an embodiment of the present application, and the training module 5555 is used to train a neural network model, wherein the audio generation method is implemented by calling the neural network model.

[0123] As mentioned above, the audio generation method based on artificial intelligence provided in the embodiments of the present application can be implemented by various types of electronic devices. Figure 3 , Figure 3 is a flowchart of an audio generation method based on artificial intelligence provided by an embodiment of the present application, combined with Figure 3 The steps shown are explained.

[0124] In the following steps, a piece of text corresponds to a phoneme sequence, and a phoneme corresponds to multiple frames of spectrum data sets (ie, audio data). For example, phoneme A corresponds to 50 milliseconds of spectrum data, and one frame of spectrum data is 10 milliseconds, then phoneme A corresponds to 5 frames of spectrum data.

[0125] In step 101, the phoneme sequence corresponding to the text is encoded to obtain a context representation of the phoneme sequence.

[0126] As an example of obtaining text, the user inputs the text of the audio to be generated through the terminal, the terminal automatically obtains the text of the audio to be generated, and automatically generates a generation request for the audio, and sends the generation request for the audio to the server. The server parses the generation request for the audio to obtain the text of the audio to be generated, and pre-processes the text of the audio to be generated to obtain the phoneme sequence corresponding to the text, so as to perform encoding processing based on the phoneme sequence later. For example, the phoneme sequence corresponding to the text "speech synthesis" is "v3in1 h e2 ch eng2". The phoneme sequence is encoded by a content encoder (a model with contextual relevance) to obtain a context representation of the phoneme sequence. The context representation output by the content encoder has the ability to model context.

[0127] In some embodiments, the phoneme sequence corresponding to the text is encoded to obtain a contextual representation of the phoneme sequence, including: forward encoding the phoneme sequence corresponding to the text to obtain a forward latent vector of the phoneme sequence; backward encoding the phoneme sequence corresponding to the text to obtain a backward latent vector of the phoneme sequence; and fusing the forward latent vector and the backward latent vector to obtain a contextual representation of the phoneme sequence.

[0128] For example, a phoneme sequence can be input into a content encoder (such as RNN, bidirectional long short-term memory network (BLSTM or BiLSTM, Bidirectional Long Short-term Memory), etc.), and the phoneme sequence is forward encoded and backward encoded by the content encoder, thereby obtaining a forward latent vector and a backward latent vector of the corresponding phoneme sequence, and the forward latent vector and the backward latent vector are fused to obtain a context representation containing context information, wherein the forward latent vector contains all forward information, and the backward latent vector contains all backward information. Therefore, the encoded information after fusing the forward latent vector and the backward latent vector contains all the information of the phoneme sequence.

[0129] In some embodiments, a phoneme sequence corresponding to a text is forward encoded to obtain a forward latent vector of the phoneme sequence, including: encoding each phoneme in the phoneme sequence corresponding to the text in sequence according to a first direction by an encoder to obtain a latent vector of each phoneme in the first direction; correspondingly, a phoneme sequence corresponding to the text is backward encoded to obtain a backward latent vector of the phoneme sequence, including: encoding each phoneme in sequence according to a second direction by an encoder to obtain a latent vector of each phoneme in the second direction; correspondingly, the forward latent vector and the backward latent vector are fused to obtain a contextual representation of the phoneme sequence, including: concatenating the forward latent vector and the backward latent vector to obtain a contextual representation of the phoneme sequence.

[0130] like Figure 6As shown, the second direction is the opposite direction of the first direction. When the first direction is the direction from the first phoneme to the last phoneme in the phoneme sequence, the second direction is the direction from the last phoneme to the first phoneme in the phoneme sequence; when the first direction is the direction from the last phoneme to the first phoneme in the phoneme sequence, the second direction is the direction from the first phoneme to the last phoneme in the phoneme sequence. The content encoder encodes each phoneme in the phoneme sequence in the first direction and the second direction in turn to obtain the latent vector of each phoneme in the first direction and the latent vector of each phoneme in the second direction, and concatenates the forward latent vector and the backward latent vector to obtain a context representation containing context information, wherein the latent vector in the first direction contains all the information in the first direction, and the latent vector in the second direction contains all the information in the second direction. Therefore, the encoded information after concatenating the latent vector in the first direction and the latent vector in the second direction contains all the information in the phoneme sequence.

[0131] For example, 0<j≤M, and j and M are positive integers, and M is the number of phonemes in the phoneme sequence. When there are M phonemes in the phoneme sequence, the M phonemes are encoded in the first direction, and M latent vectors in the first direction are obtained in sequence. For example, after the phoneme sequence is encoded in the first direction, the latent vector in the first direction is {h 1l ,h 2l ,...h jl ...,h Ml}, where h il represents the jth latent vector of the jth phoneme in the first direction. The M phonemes are encoded in the second direction, and the M latent vectors in the second direction are obtained in sequence. For example, after the phonemes are encoded in the second direction, the latent vector in the second direction is {h 1r ,h 2r ,...h jr ...,h Mr}, where h ir represents the jth hidden vector of the j phoneme in the second direction. The hidden vector in the first direction is {h 1l ,h 2l ,...h jl ...,h Ml} and the hidden vector in the second direction is {h 1r ,h 2r ,...h jr ...,h Mr} to obtain the context representation containing context information {[h 1l ,h 1r ],[h 2l ,h 2r ],...[h jl ,h jr]...,[h Ml ,h Mr ]}, for example, the jth hidden vector h in the first direction is jl , the jth vector in the second direction is the jth hidden vector h jr Perform concatenation to obtain the i-th coded information {h jl ,h jr In order to save the calculation process, since the last latent vector in the first direction contains most of the information in the first direction and the last latent vector in the second direction contains most of the information in the second direction, the last latent vector in the first direction and the last latent vector in the second direction can be directly fused to obtain a context representation containing context information.

[0132] In step 102, based on the first frame hidden state corresponding to each phoneme in the phoneme sequence, an alignment position of the first frame hidden state relative to the context representation is determined.

[0133] In step 103, when the alignment position corresponds to a non-end position in the context representation, decoding processing is performed based on the context representation and the first frame implicit state to obtain the second frame implicit state.

[0134] Each phoneme corresponds to multiple frames of implicit states. The first frame implicit state represents the implicit state of the first frame, the second frame implicit state represents the implicit state of the second frame, and the first frame and the second frame are any two adjacent frames in the spectrum data corresponding to the phoneme.

[0135] See also Figure 4 , Figure 4 is an optional flow chart of an audio generation method based on artificial intelligence provided in an embodiment of the present application, Figure 4 Show Figure 3 Step 102 in can be Figure 4 Step 102A shown in FIG. 1 is implemented as follows: In step 102A, when the first frame implicit state is recorded as the t-th frame implicit state, the following processing is performed for each phoneme in the phoneme sequence: based on the t-th frame implicit state corresponding to the phoneme, the alignment position of the t-th frame implicit state relative to the context representation is determined; correspondingly, step 103 can be performed by Figure 4 The illustrated step 103A is implemented as follows: in step 103A, when the implicit state of the t-th frame corresponds to a non-end position in the context representation relative to the alignment position of the context representation, decoding processing is performed based on the context representation and the implicit state of the t-th frame to obtain the implicit state of the t+1-th frame; wherein t is a natural number that increases incrementally from 1 and satisfies 1≤t≤T, and T is the total number of frames corresponding to the phoneme sequence when the alignment position corresponds to the end position in the context representation, and the total number of frames represents the number of frames of spectral data corresponding to the implicit state of each phoneme in the phoneme sequence.

[0136] like Figure 7 As shown, the following iterative processing is performed for each phoneme in the phoneme sequence: the hidden state of the t-th frame output by the autoregressive decoder is input into the Gaussian attention mechanism, and the Gaussian attention mechanism determines the alignment position of the hidden state of the t-th frame relative to the context representation based on the hidden state of the t-th frame. When the alignment position of the hidden state of the t-th frame relative to the context representation corresponds to a non-end position in the context representation, the autoregressive decoder continues the decoding process, and performs decoding processing based on the context representation and the hidden state of the t-th frame to obtain the hidden state of the t+1-th frame, and the iterative processing is stopped until the alignment position of the hidden state relative to the context representation corresponds to the end position in the context representation.

[0137] See also Figure 5 , Figure 5 is an optional flow chart of an audio generation method based on artificial intelligence provided in an embodiment of the present application, Figure 5 Show Figure 4 Step 102A in Figure 5 Steps 1021A to 1022A are shown to implement: in step 1021A, Gaussian prediction processing is performed based on the implicit state of the t-th frame corresponding to the phoneme to obtain the t-th Gaussian parameter corresponding to the implicit state of the t-th frame; in step 1022A, the alignment position of the implicit state of the t-th frame relative to the context representation is determined based on the t-th Gaussian parameter.

[0138] Continuing with the above example, the Gaussian attention mechanism includes a fully connected layer, which performs Gaussian prediction processing based on the hidden state of the t-th frame corresponding to the phoneme to obtain the t-th Gaussian parameter corresponding to the hidden state of the t-th frame, thereby determining the alignment position of the hidden state of the t-th frame relative to the context representation based on the t-th Gaussian parameter.

[0139] For example, a Gaussian function-based prediction process is performed on the implicit state of the tth frame corresponding to the phoneme to obtain the tth Gaussian variance and the tth Gaussian mean change corresponding to the implicit state of the tth frame; the t-1th Gaussian parameter corresponding to the implicit state of the t-1th frame is obtained; the t-1th Gaussian mean and the tth Gaussian mean change included in the t-1th Gaussian parameter are added together, and the sum is used as the tth Gaussian mean corresponding to the implicit state of the tth frame; the set of the tth Gaussian variance and the tth Gaussian mean is used as the tth Gaussian parameter corresponding to the implicit state of the tth frame; the tth Gaussian mean is used as the alignment position of the implicit state of the tth frame relative to the context representation.

[0140] In some embodiments, the content text length of the context representation of the phoneme sequence is obtained; when the tth Gaussian mean is greater than the content text length, it is determined that the alignment position corresponds to the end position in the context representation; when the tth Gaussian mean is less than or equal to the content text length, it is determined that the alignment position corresponds to the non-end position in the context representation.

[0141] like Figure 8 As shown, for example, the content text length of the context representation is 6. When the t-th Gaussian mean is greater than the content text length, the alignment position corresponds to the end position in the context representation, that is, the alignment position points to the end position of the context representation.

[0142] like Fig. 9 As shown, for example, the content text length of the context representation is 6. When the tth Gaussian mean is less than or equal to the content text length, the alignment position corresponds to the non-end position in the context representation, that is, the alignment position points to the position including the content in the context representation, for example, the alignment position points to the position of the second content in the context representation.

[0143] In some embodiments, decoding processing is performed based on the context representation and the implicit state of the tth frame to obtain the implicit state of the t+1th frame, including: obtaining the attention weight corresponding to the implicit state of the tth frame; weighted processing is performed on the context representation based on the attention weight to obtain a context vector corresponding to the context representation; state prediction processing is performed based on the context vector and the implicit state of the tth frame to obtain the implicit state of the t+1th frame.

[0144] For example, when the alignment position corresponds to a non-end position in the context representation, it means that decoding processing needs to continue. First, the attention weight corresponding to the implicit state of the t-th frame is determined through the Gaussian attention mechanism, and the context representation is weighted based on the attention weight to obtain the context vector corresponding to the context representation. The context vector is sent to the autoregressive decoder. The autoregressive decoder performs state prediction processing based on the context vector and the implicit state of the t-th frame to obtain the implicit state of the t+1-th frame, thereby realizing autoregression of the implicit state with front-end correlation.

[0145] In some embodiments, obtaining the attention weight corresponding to the implicit state of the tth frame includes: obtaining the tth Gaussian parameter corresponding to the implicit state of the tth frame, wherein the tth Gaussian parameter includes the tth Gaussian variance and the tth Gaussian mean; performing Gaussian processing on the context representation based on the tth Gaussian variance and the tth Gaussian mean to obtain the attention weight corresponding to the implicit state of the tth frame.

[0146] For example, the attention weight is calculated as Among them, α t,jrepresents the attention weight of the jth element of the input content encoder phoneme sequence during the t-th iteration calculation (t-th frame hidden state), μ t represents the mean and σ of the Gaussian function at the tth step t 2 represents the variance of the Gaussian function at step t. Other variant weight calculation formulas are also applicable to the embodiments of the present application.

[0147] In step 104, a synthesis process is performed based on the implicit state of the first frame and the implicit state of the second frame to obtain audio data corresponding to the text.

[0148] For example, the implicit state of the first frame represents the implicit state of the first frame, and the implicit state of the second frame represents the implicit state of the second frame. The first frame and the second frame are any two adjacent frames in the spectral data corresponding to the phoneme. When the alignment position corresponds to the end position in the context representation, the implicit states of T frames are spliced ​​to obtain the implicit state corresponding to the text, the implicit state corresponding to the text is smoothed to obtain the spectral data corresponding to the text, the spectral data corresponding to the text is Fourier transformed to obtain the audio data corresponding to the text, and when to stop decoding is determined based on the alignment position, thereby solving the problem of premature decoding stop and improving the naturalness and stability of speech synthesis.

[0149] In some embodiments, it is necessary to train a neural network model so that the trained neural network model can realize audio generation, and the audio generation method is realized by calling the neural network model; the training process of the neural network model includes: encoding the phoneme sequence sample corresponding to the text sample through the initialized neural network model to obtain the context representation of the phoneme sequence sample; based on the third frame implicit state corresponding to each phoneme in the phoneme sequence sample, determining the predicted alignment position of the third frame implicit state relative to the context representation; when the predicted alignment position corresponds to a non-end position in the context representation, decoding is performed based on the context representation and the third frame implicit state. Obtain the hidden state of the fourth frame; perform spectrum post-processing based on the hidden state of the third frame and the hidden state of the fourth frame to obtain predicted spectrum data corresponding to the text sample; construct a loss function of the neural network model based on the predicted spectrum data corresponding to the text sample and the labeled spectrum data corresponding to the text sample; update the parameters of the neural network model, and use the updated parameters of the neural network model when the loss function converges as the parameters of the trained neural network model; wherein the hidden state of the third frame represents the hidden state of the third frame, the hidden state of the fourth frame represents the hidden state of the fourth frame, and the third frame and the fourth frame are any two adjacent frames in the spectrum data corresponding to each phoneme in the phoneme sequence sample.

[0150] For example, after determining the value of the loss function of the neural network model based on the predicted spectrum data corresponding to the text sample and the labeled spectrum data corresponding to the text sample, it can be determined whether the value of the loss function of the neural network model exceeds a preset threshold. When the value of the loss function of the neural network model exceeds the preset threshold, the error signal of the neural network model is determined based on the loss function of the neural network model, the error information is back-propagated in the neural network model, and the model parameters of each layer are updated during the propagation process.

[0151] Here, back propagation is explained. The training sample data is input into the input layer of the neural network model, passes through the hidden layer, and finally reaches the output layer and outputs the result. This is the forward propagation process of the neural network model. Since there is an error between the output result of the neural network model and the actual result, the error between the output result and the actual value is calculated, and the error is backpropagated from the output layer to the hidden layer until it propagates to the input layer. During the back propagation process, the value of the model parameter is adjusted according to the error; the above process is continuously iterated until convergence.

[0152] In some embodiments, before updating the parameters of the neural network model, it also includes: constructing a parameter matrix based on the parameters of the neural network model; dividing the parameter matrix into blocks to obtain a plurality of matrix blocks included in the parameter matrix; when the structure reaches a sparse timing, determining the mean of the parameters in each matrix block; sorting the matrix blocks in ascending order based on the mean of the parameters in each matrix block, resetting the parameters in the plurality of matrix blocks sorted first in the ascending sorting results, and obtaining a reset parameter matrix; wherein the reset parameter matrix is ​​used to update the parameters of the neural network model.

[0153] like Fig.10 As shown, in order to improve the audio synthesis speed, in the process of training the neural network model, the parameters in the neural network model can be trained in blocks. First, a parameter matrix is ​​constructed based on all the parameters of the neural network model, and then the parameter matrix is ​​divided into blocks to obtain matrix block 1, matrix block 2, ..., matrix block 16. When a preset number of training times or a preset training time is reached, the mean value of the parameters in each matrix block is determined, and the matrix blocks are sorted in ascending order based on the mean value of the parameters in each matrix block, and the parameters in the plurality of matrix blocks sorted in the ascending order are reset to 0. For example, The parameters in the first 8 matrix blocks are charged to 0, and matrix block 3, matrix block 4, matrix block 7 and matrix block 8, matrix block 9, matrix block 10, matrix block 13 and matrix block 14 are the first 8 matrix blocks in the ascending sorting result. Then the parameters in the dotted box 1001 (including matrix block 3, matrix block 4, matrix block 7 and matrix block 8) and the dotted box 1002 (including matrix block 9, matrix block 10, matrix block 13 and matrix block 14) are reset to 0, thereby obtaining a reset parameter matrix, and the multiplication operation of the parameter matrix can be accelerated, thereby improving the training speed.

[0154] In some embodiments, before constructing the loss function of the neural network model, it also includes: obtaining the content text length of the context representation of the phoneme sequence sample; when the predicted alignment position corresponds to the end position in the context representation, constructing the position loss function of the neural network model based on the predicted alignment position and the content text length; constructing the spectral loss function of the neural network model based on the predicted spectral data corresponding to the text sample and the annotated spectral data corresponding to the text sample; performing weighted summation on the spectral loss function and the position loss function, and using the weighted summation result as the loss function of the neural network model.

[0155] For example, in order to solve problems such as early stopping of decoding, missing words, and repeated reading, a position loss function of the neural network model is constructed, so that the trained neural network model can learn the ability to accurately predict the alignment position, thereby improving the naturalness and stability of speech generation.

[0156] The following is an explanation of an exemplary application of the embodiments of the present application in an actual speech synthesis application scenario.

[0157] The embodiments of the present application can be applied to various speech synthesis application scenarios (for example, smart devices with speech synthesis capabilities such as smart speakers, speakers with screens, smart watches, smart phones, smart homes, smart maps, smart cars, etc., online education, smart robots, artificial intelligence customer service, speech synthesis cloud services, etc.). For example, for in-vehicle applications, when the user is driving, it is not convenient to understand the information in the form of text, but the information can be understood by reading the voice to avoid missing important information. After the in-vehicle client receives the text, it needs to convert the text into voice to play the voice to the user, so that the user can read the voice corresponding to the text in time.

[0158] In the related art, the acoustic model uses a content-based, position-based attention mechanism or a hybrid of the two, combined with a stop token mechanism to predict the stop position of the generated audio. The related technical solutions have the following problems: 1) Alignment errors occur, resulting in unbearable problems such as missing words or repeated reading of words, making it difficult for the speech synthesis system to be put into practical application; 2) The synthesis of long and complex sentences will stop prematurely, resulting in incomplete audio synthesis; 3) The speed of training and inference is very slow, making it difficult to deploy speech synthesis (TTS, Text To Speech) on edge devices such as mobile phones.

[0159] In order to solve the above problems, the embodiment of the present application uses a single Gaussian attention mechanism (Single GaussianAttention), a monotonic, normalized, stable and more expressive attention mechanism, which solves the instability problem of the attention mechanism used in the related technology, and removes the Stop Token mechanism, and proposes to use attention stop prediction (Attentive Stop Loss) (used to judge the stop value in the autoregressive decoding process, for example, set to 0.5 when the probability exceeds the threshold) to ensure the result, and directly judge the stop based on alignment, which solves the problem of early stopping and improves the naturalness and stability of speech synthesis; on the other hand, the embodiment of the present application uses pruning technology to block sparsify the autoregressive decoder (AutoregressiveDecoder), further improving the speed of training and synthesis, and can achieve 35 times the real-time synthesis rate on a single-core central processing unit (CPU, Central Processing Unit), making it possible to deploy TTS on edge devices.

[0160] The embodiments of the present application can be applied to all products with speech synthesis capabilities, including but not limited to smart speakers, speakers with screens, smart watches, smart phones, smart homes, smart cars and other smart devices, smart robots, AI customer service, TTS cloud services, etc., and their usage scenarios can enhance the stability of synthesis and improve the speed of synthesis through the algorithm proposed in the embodiments of the present application.

[0161] like Fig.11 As shown, the end-to-end speech synthesis acoustic model (neural network model) of the embodiment of the present application includes a content encoder, a Gaussian attention mechanism, an autoregressive decoder, and a spectrum post-processing network. The following specifically introduces each module of the end-to-end speech synthesis acoustic model:

[0162] 1) Content encoder: Converts the input phoneme sequence into context-related implicit features (context representation). The content encoder is composed of a model with contextual relevance. The features generated by the content encoder have the ability to model context. Among them, linguistic features represent the text content to be synthesized. The basic unit of the text is a character or phoneme. In Chinese speech synthesis, the text is composed of initials, finals, and silent syllables, among which the finals are tonal. For example, the tonal phoneme sequence of the text "speech synthesis" is "v3 in1 h e2 ch eng2".

[0163] 2) Gaussian attention mechanism: Combine the current state of the decoder to generate the corresponding content context information (context vector) for the autoregressive decoder to better predict the next frame of spectrum. Speech synthesis is a task of establishing a monotonic mapping from a text sequence to a spectrum sequence. Therefore, when generating each frame of Mel spectrum, only a small part of the phoneme content needs to be focused on, and this part of the phoneme content is generated through the attention mechanism. Among them, the speaker identity information (SpeakerIdentity) is represented by a set of vectors to represent the unique identifier of a speaker.

[0164] 3) Autoregressive decoder: The spectrum of the current frame is generated by the content context information generated by the current Gaussian attention mechanism and the spectrum predicted by the previous frame. Since it needs to rely on the output of the previous frame, it is called an autoregressive decoder. Among them, replacing the autoregressive decoder with a parallel fully connected form can further improve the training speed.

[0165] 4) Mel spectrum post-processing network: Smooth the spectrum predicted by the autoregressive decoder to obtain a higher quality spectrum.

[0166] Combine the following Fig.11 and Fig.12 The optimization of the stability and speed of speech synthesis in the embodiment of the present application is specifically described as follows:

[0167] A) Fig.11 As shown, the embodiment of the present application adopts a single Gaussian attention mechanism, which is a monotonic, normalized, stable, and more expressive attention mechanism. The single Gaussian attention mechanism calculates the attention weight in the manner of formula (1) and formula (2).

[0168]

[0169] μ i = i-1 +Δ i (2)

[0170] Among them, α i,j represents the attention weight of the jth element of the input content encoder phoneme sequence during the i-th iteration, exp represents the exponential function, μ i represents the mean and σ of the Gaussian function in the i-th step calculation i 2 represents the variance of the Gaussian function at the i-th step, Δ i It represents the predicted mean change in the iterative calculation of the i-th step. The mean change, variance, etc. are obtained through a fully connected network based on the implicit state of the autoregressive decoder.

[0171] Each iteration predicts the mean change and variance of the Gaussian at the current moment, where the cumulative sum of the mean change represents the position of the attention window at the current moment, that is, the position of the input linguistic features aligned with it, and the variance represents the width of the attention window. The phoneme sequence is used as the input of the content encoder, and the context vector required by the autoregressive decoder is obtained through the Gaussian attention mechanism. The autoregressive decoder generates the Mel spectrum in an autoregressive manner. The stop sign of the autoregressive decoding is determined by whether the mean of the Gaussian attention distribution touches the end of the phoneme sequence. The embodiment of the present application ensures the monotonicity of the alignment process by ensuring that the mean change is non-negative, and because the Gaussian function itself is normalized, the stability of the attention mechanism is ensured.

[0172] The context vector required by the autoregressive decoder at each moment is obtained by weighting the weights generated by the Gaussian attention mechanism and the output of the content encoder. The distribution of the weights is determined by the mean of the Gaussian attention. The speech synthesis task is a strictly monotonic task, that is, the output Mel spectrum must be monotonically generated from left to right based on the input text. Therefore, if the mean of the Gaussian attention is at the end of the input phoneme sequence, it means that the Mel spectrum generation is close to the end. The width of the attention window represents the range of the content encoder output content required for each decoding. The width is affected by the language structure. For example, for the silent prediction of a pause, the width is relatively small; when encountering a word or phrase, the width is relatively large, because the pronunciation of a word in a word or phrase will be affected by the previous and next words.

[0173] B) The embodiment of the present application removes the separate Stop Token architecture, uses Gaussian Attention to stop directly based on alignment judgment, and proposes Attentive Stop Loss to ensure the alignment result and solve the problem of premature stopping of complex or long sentences. Assuming that the mean at the last moment of training should be iterated to the next position of the input text length, based on this assumption, an L1Loss (i.e. L2Loss) between the mean of the Gaussian distribution and the length of the input text sequence is constructed. stop ), as shown in formula (3), Fig.12 As shown, during the reasoning process, the solution of the embodiment of the present application determines whether to stop based on whether the mean of the Gaussian Attention at the current moment is greater than the length of the input text plus one:

[0174] L stop =|μ I -J-1| (3)

[0175] Among them, μ I is the total number of iterations, and J is the length of the phoneme sequence.

[0176] If the Stop Token architecture is adopted, it may stop prematurely because the Stop Token architecture does not take into account the integrity of the phonemes. A significant problem caused by this Stop Token architecture is that it is necessary to ensure that the silence at the beginning and end of the recorded audio and the pauses in the middle need to be of similar length, so that the Stop Token architecture prediction will be more accurate. Once the recorder pauses for a long time, the trained Stop Token prediction will be inaccurate. Therefore, the Stop Token architecture has relatively high requirements for data quality and will bring higher audit costs. The Attention Stop Loss proposed in the embodiment of the present application can reduce the requirements for data quality, thereby reducing costs.

[0177] C) The embodiment of the present application performs block sparsification on the autoregressive decoder, thereby improving the calculation speed of the autoregressive decoder. For example, the sparsification scheme adopted in the present application is: starting from the 1000th training step, structured sparsification is performed every 400 steps until the sparsity of 50% is reached after 120K training steps, wherein the L1 Loss between the Mel spectrum predicted by the model and the true Mel spectrum is used as the optimization target, and the parameters of the entire model are optimized by the stochastic gradient descent algorithm. The embodiment of the present application divides the weight matrix into multiple blocks (matrix blocks), and then sorts them from small to large according to the average value of the model parameters in each block, and sets the model parameters of the first 50% (set according to actual conditions) of the blocks to 0 to speed up the decoding process.

[0178] When a matrix is ​​block sparse, that is, the matrix is ​​divided into N blocks, and the elements of some blocks are 0, then the multiplication operation of the matrix can be accelerated. In training, the elements in some blocks are set to 0. This is determined by the amplitude of the elements. That is, if the average amplitude of the elements in a block is very small or close to 0 (that is, less than a certain threshold), then the elements in the block are approximately 0, thereby achieving the purpose of sparseness. In practice, the amplitudes of the elements in multiple blocks of a matrix can be sorted by the average value, and the first 50% of the blocks with smaller average amplitudes will be sparse, that is, the elements are uniformly set to zero.

[0179] In practical applications, the text is first converted into a phoneme sequence. The phoneme sequence is passed through the content encoder to obtain an abstract representation containing context information (i.e., context representation). When predicting the Mel spectrum, a segment of all-zero vectors is first input into the autoregressive decoder as the initial context vector. Then, the implicit state output by the autoregressive decoder each time is used as the input of the Gaussian attention mechanism. The weight of the content encoder output at each moment can be calculated. Combining the weight and the abstract representation of the content encoder, the context vector required by the autoregressive decoder at each moment can be calculated. In this way, when the mean of the Gaussian attention is at the end of the abstract representation (phoneme sequence) of the content encoder, the decoding can be stopped. The Mel spectrum (implicit state) predicted by the autoregressive decoder is spliced ​​together and sent to the Mel post-processing network. The purpose is to make the Mel spectrum smoother and make its generation process not only dependent on past information, but also on future information. After obtaining the final Mel spectrum, the final audio waveform is obtained by signal processing or a neural network synthesizer to achieve the function of speech synthesis.

[0180] In summary, the embodiments of the present application have the following beneficial effects: 1) By combining the monotonic and stable Gaussian Attention mechanism and Attentive Stop Loss, the stability of speech synthesis is effectively improved, avoiding unbearable phenomena such as repeated reading and missing words; 2) The autoregressive decoder is block-sparse, which greatly improves the synthesis speed of the acoustic model and reduces the requirements for hardware equipment.

[0181] Since the embodiment of the present application proposes a more robust attention mechanism acoustic model, it has the advantages of fast speed and high stability. The acoustic model can be applied to embedded devices such as smart homes and smart cars. Since the computing power of these embedded devices is relatively low, end-to-end speech synthesis becomes easier to implement on the device side; due to its high robustness, the solution can be applied to personalized voice customization scenarios with low data quality in non-recording studio scenarios, such as voice customization of mobile map users, large-scale voice cloning of online teachers in online education, etc. Because the recording users in these scenarios are not professional voice actors, the recordings may have long pauses. For such data, the embodiment of the present application can effectively ensure the stability of the acoustic model.

[0182] So far, the audio generation method based on artificial intelligence provided by the embodiment of the present application has been described in combination with the exemplary application and implementation of the server provided by the embodiment of the present application. The embodiment of the present application also provides an audio generation device. In practical applications, each functional module in the audio generation device can be implemented by hardware resources of an electronic device (such as a terminal device, a server or a server cluster), such as computing resources such as a processor, communication resources (such as for supporting various communication methods such as optical cables and cellular), and memory. Figure 2 An audio generating device 555 stored in a memory 550 is shown, which may be software in the form of programs and plug-ins, for example, software modules designed in programming languages ​​such as C / C++ and Java, application software designed in programming languages ​​such as C / C++ and Java, or dedicated software modules in large software systems, application programming interfaces, plug-ins, cloud services, and other implementation methods. Different implementation methods are illustrated below with examples.

[0183] Example 1: The audio generating device is a mobile application and module

[0184] The audio generating device 555 in the embodiment of the present application can be provided as a software module designed using a programming language such as software C / C++, Java, etc., and embedded in various mobile terminal applications based on systems such as Android or iOS (stored in a storage medium on the mobile terminal as executable instructions and executed by a processor on the mobile terminal), thereby directly using the mobile terminal's own computing resources to complete related information recommendation tasks, and transmitting the processing results to a remote server regularly or irregularly through various network communication methods, or saving them locally on the mobile terminal.

[0185] Example 2: The audio generation device is a server application and platform

[0186] The audio generating device 555 in the embodiment of the present application can be provided as an application software designed using programming languages ​​such as C / C++, Java, etc., or a dedicated software module in a large software system, running on the server side (stored in the storage medium on the server side in the form of executable instructions and run by the processor on the server side), and the server uses its own computing resources to complete related information recommendation tasks.

[0187] The embodiments of the present application can also be provided on a distributed, parallel computing platform composed of multiple servers, equipped with a customized, easy-to-interact network (Web) interface or other user interfaces (UI, User Interface), to form an information recommendation platform (for recommendation lists) for use by individuals, groups or units.

[0188] Example 3: The audio generation device is a server-side application program interface (API) and a plug-in

[0189] The audio generating device 555 in the embodiment of the present application can be provided as a server-side API or plug-in for users to call to execute the artificial intelligence-based audio generation method of the embodiment of the present application, and embedded in various applications.

[0190] Example 4: The audio generation device is a mobile device client API and plug-in

[0191] The audio generating device 555 in the embodiment of the present application can be provided as an API or plug-in on the mobile device for the user to call to execute the artificial intelligence-based audio generating method in the embodiment of the present application.

[0192] Example 5: The audio generation device is a cloud-based open service

[0193] The audio generating device 555 in the embodiment of the present application can provide an information recommendation cloud service developed for users, so that individuals, groups or units can obtain audio.

[0194] The audio generation device 555 includes a series of modules, including an encoding module 5551, an attention module 5552, a decoding module 5553, a synthesis module 5554, and a training module 5555. The following further describes how the various modules in the audio generation device 555 provided in the embodiment of the present application cooperate to implement the audio generation solution.

[0195] The encoding module 5551 is used to encode the phoneme sequence corresponding to the text to obtain the context representation of the phoneme sequence; the attention module 5552 is used to determine the alignment position of the first frame implicit state relative to the context representation based on the first frame implicit state corresponding to each phoneme in the phoneme sequence; the decoding module 5553 is used to perform decoding processing based on the context representation and the first frame implicit state to obtain the second frame implicit state when the alignment position corresponds to a non-end position in the context representation; the synthesis module 5554 is used to perform synthesis processing based on the first frame implicit state and the second frame implicit state to obtain the audio data corresponding to the text.

[0196] In some embodiments, the first frame implicit state represents the implicit state of the first frame, the second frame implicit state represents the implicit state of the second frame, and the first frame and the second frame are any two adjacent frames in the spectral data corresponding to the phoneme; when the first frame implicit state is recorded as the t-th frame implicit state, the attention module 5552 is also used to perform the following processing for each phoneme in the phoneme sequence: based on the t-th frame implicit state corresponding to the phoneme, determine the alignment position of the t-th frame implicit state relative to the context representation; correspondingly, the decoding module 5553 is also used to perform decoding processing based on the context representation and the t-th frame implicit state when the alignment position of the t-th frame implicit state relative to the context representation corresponds to a non-end position in the context representation, so as to obtain the t+1-th frame implicit state; wherein t is a natural number that increases from 1 and satisfies 1≤t≤T, T is the total number of frames corresponding to the phoneme sequence when the alignment position corresponds to the end position in the context representation, and the total number of frames represents the number of frames of spectral data corresponding to the implicit state of each phoneme in the phoneme sequence.

[0197] In some embodiments, the synthesis module 5554 is also used to, when the alignment position corresponds to the end position in the context representation, perform splicing processing on the implicit state of the T frame to obtain the implicit state corresponding to the text; perform smoothing processing on the implicit state corresponding to the text to obtain the spectrum data corresponding to the text; perform Fourier transform on the spectrum data corresponding to the text to obtain the audio data corresponding to the text.

[0198] In some embodiments, the attention module 5552 is also used to perform Gaussian prediction processing based on the implicit state of the tth frame corresponding to the phoneme to obtain the tth Gaussian parameter corresponding to the implicit state of the tth frame; and determine the alignment position of the implicit state of the tth frame relative to the context representation based on the tth Gaussian parameter.

[0199] In some embodiments, the attention module 5552 is also used to perform Gaussian function-based prediction processing based on the implicit state of the tth frame corresponding to the phoneme, and obtain the tth Gaussian variance and the tth Gaussian mean change corresponding to the implicit state of the tth frame; obtain the t-1th Gaussian parameter corresponding to the implicit state of the t-1th frame; add the t-1th Gaussian mean and the tth Gaussian mean change included in the t-1th Gaussian parameter, and use the sum obtained as the tth Gaussian mean corresponding to the implicit state of the tth frame; use the set of the tth Gaussian variance and the tth Gaussian mean as the tth Gaussian parameter corresponding to the implicit state of the tth frame; use the tth Gaussian mean as the alignment position of the implicit state of the tth frame relative to the context representation.

[0200] In some embodiments, the attention module 5552 is also used to obtain the content text length of the context representation of the phoneme sequence; when the tth Gaussian mean is greater than the content text length, it is determined that the alignment position corresponds to the end position in the context representation; when the tth Gaussian mean is less than or equal to the content text length, it is determined that the alignment position corresponds to the non-end position in the context representation.

[0201] In some embodiments, the decoding module 5553 is also used to obtain the attention weight corresponding to the implicit state of the tth frame; perform weighted processing on the context representation based on the attention weight to obtain a context vector corresponding to the context representation; perform state prediction processing based on the context vector and the implicit state of the tth frame to obtain the implicit state of the t+1th frame.

[0202] In some embodiments, the attention module 5552 is also used to obtain the tth Gaussian parameter corresponding to the implicit state of the tth frame, wherein the tth Gaussian parameter includes the tth Gaussian variance and the tth Gaussian mean; based on the tth Gaussian variance and the tth Gaussian mean, Gaussian processing is performed on the context representation to obtain the attention weight corresponding to the implicit state of the tth frame.

[0203] In some embodiments, the audio generation method is implemented by calling a neural network model; the audio generation device 555 also includes: a training module 5555, which is used to encode the phoneme sequence sample corresponding to the text sample through the initialized neural network model to obtain the context representation of the phoneme sequence sample; based on the third frame implicit state corresponding to each phoneme in the phoneme sequence sample, determine the predicted alignment position of the third frame implicit state relative to the context representation; when the predicted alignment position corresponds to a non-end position in the context representation, decode based on the context representation and the third frame implicit state to obtain the fourth frame implicit state; based on the third frame implicit state The three frames of hidden states and the fourth frame of hidden state are subjected to spectral post-processing to obtain predicted spectral data corresponding to the text sample; based on the predicted spectral data corresponding to the text sample and the annotated spectral data corresponding to the text sample, a loss function of the neural network model is constructed; the parameters of the neural network model are updated, and the updated parameters of the neural network model when the loss function converges are used as the parameters of the trained neural network model; wherein the third frame of hidden state represents the hidden state of the third frame, the fourth frame of hidden state represents the hidden state of the fourth frame, and the third frame and the fourth frame are any two adjacent frames in the spectral data corresponding to each phoneme in the phoneme sequence sample.

[0204] In some embodiments, the training module 5555 is also used to construct a parameter matrix based on the parameters of the neural network model; divide the parameter matrix into blocks to obtain a plurality of matrix blocks included in the parameter matrix; when the structure reaches a sparse timing, determine the mean of the parameters in each of the matrix blocks; sort the matrix blocks in ascending order based on the mean of the parameters in each of the matrix blocks, reset the parameters in the plurality of matrix blocks sorted first in the ascending sorting results, and obtain a reset parameter matrix; wherein the reset parameter matrix is ​​used to update the parameters of the neural network model.

[0205] In some embodiments, the training module 5555 is also used to obtain the content text length of the context representation of the phoneme sequence sample; when the predicted alignment position corresponds to the end position in the context representation, constructing the position loss function of the neural network model based on the predicted alignment position and the content text length; constructing the spectral loss function of the neural network model based on the predicted spectral data corresponding to the text sample and the annotated spectral data corresponding to the text sample; performing weighted summation on the spectral loss function and the position loss function, and using the weighted summation result as the loss function of the neural network model.

[0206] In some embodiments, the encoding module 5551 is also used to perform forward encoding processing on the phoneme sequence corresponding to the text to obtain a forward latent vector of the phoneme sequence; perform backward encoding processing on the phoneme sequence corresponding to the text to obtain a backward latent vector of the phoneme sequence; and fuse the forward latent vector and the backward latent vector to obtain a contextual representation of the phoneme sequence.

[0207] In some embodiments, the encoding module 5551 is also used to encode each phoneme in the phoneme sequence corresponding to the text in a first direction through an encoder to obtain a latent vector of each phoneme in the first direction; encode each phoneme in a second direction through the encoder to obtain a latent vector of each phoneme in the second direction; concatenate the forward latent vector and the backward latent vector to obtain a contextual representation of the phoneme sequence; wherein the second direction is the opposite direction of the first direction.

[0208] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned artificial intelligence-based audio generation method of the embodiment of the present application.

[0209] The embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the audio generation method based on artificial intelligence provided by the embodiment of the present application, for example, Figure 3-5 An artificial intelligence-based audio generation method is shown.

[0210] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0211] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0212] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0213] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0214] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An audio generation method based on artificial intelligence, characterized in that: The method comprises: Encoding the phoneme sequence corresponding to the text to obtain a context representation of the phoneme sequence; For each phoneme in the phoneme sequence, the following processing is performed: based on the t-th frame hidden state corresponding to the phoneme, determining the alignment position of the t-th frame hidden state relative to the context representation; When the alignment position of the implicit state of the t-th frame with respect to the context representation corresponds to a non-end position in the context representation, decoding processing is performed based on the context representation and the implicit state of the t-th frame to obtain the implicit state of the t+1-th frame, wherein t is a natural number that increases from 1 and satisfies 1≤t≤T, and T is the total number of frames corresponding to the phoneme sequence when the alignment position corresponds to the end position in the context representation, and the total number of frames represents the number of frames of spectral data corresponding to the implicit state of each phoneme in the phoneme sequence; When the alignment position corresponds to the end position in the context representation, the T-frame implicit state is spliced ​​to obtain the audio data corresponding to the text.

2. The method according to claim 1, characterized in that The step of performing splicing processing on the implicit state of the T frame to obtain the audio data corresponding to the text includes: Performing splicing processing on the implicit states of the T frames to obtain the implicit states corresponding to the text; Performing smoothing on the implicit state corresponding to the text to obtain spectrum data corresponding to the text; Perform Fourier transform on the frequency spectrum data corresponding to the text to obtain audio data corresponding to the text.

3. The method according to claim 1, characterized in that The determining, based on the implicit state of the t-th frame corresponding to the phoneme, an alignment position of the implicit state of the t-th frame relative to the context representation comprises: Performing Gaussian prediction processing based on the implicit state of the t-th frame corresponding to the phoneme to obtain the t-th Gaussian parameter corresponding to the implicit state of the t-th frame; An alignment position of the hidden state of the t-th frame relative to the context representation is determined based on the t-th Gaussian parameter.

4. The method according to claim 3, characterized in that The performing Gaussian prediction processing based on the implicit state of the t-th frame corresponding to the phoneme to obtain the t-th Gaussian parameter corresponding to the implicit state of the t-th frame includes: Performing a prediction process based on a Gaussian function based on the implicit state of the t-th frame corresponding to the phoneme to obtain a t-th Gaussian variance and a t-th Gaussian mean change corresponding to the implicit state of the t-th frame; Get the t-1th Gaussian parameter corresponding to the hidden state of the t-1th frame; Adding the t-1th Gaussian mean value and the tth Gaussian mean change amount included in the t-1th Gaussian parameter, and using the summed result as the tth Gaussian mean value corresponding to the tth frame implicit state; Taking the set of the t-th Gaussian variance and the t-th Gaussian mean as the t-th Gaussian parameter corresponding to the hidden state of the t-th frame; The determining, based on the t-th Gaussian parameter, an alignment position of the t-th frame implicit state relative to the context representation comprises: The t-th Gaussian mean is used as the alignment position of the t-th frame hidden state relative to the context representation.

5. The method according to claim 4, characterized in that The method further comprises: Obtaining the content text length of the context representation of the phoneme sequence; When the t-th Gaussian mean is greater than the content text length, determining that the alignment position corresponds to the end position in the context representation; When the t-th Gaussian mean is less than or equal to the content text length, it is determined that the alignment position corresponds to a non-end position in the context representation.

6. The method according to claim 1, characterized in that The decoding process is performed based on the context representation and the implicit state of the t-th frame to obtain the implicit state of the t+1-th frame, including: Obtaining the attention weight corresponding to the implicit state of the t-th frame; Performing weighted processing on the context representation based on the attention weight to obtain a context vector corresponding to the context representation; A state prediction process is performed based on the context vector and the implicit state of the t-th frame to obtain an implicit state of the t+1-th frame.

7. The method according to claim 6, characterized in that The obtaining the attention weight corresponding to the implicit state of the t-th frame includes: Obtaining the tth Gaussian parameter corresponding to the implicit state of the tth frame, wherein the tth Gaussian parameter includes the tth Gaussian variance and the tth Gaussian mean; Gaussian processing is performed on the context representation based on the t-th Gaussian variance and the t-th Gaussian mean to obtain an attention weight corresponding to the implicit state of the t-th frame.

8. The method according to claim 1, characterized in that The audio generation method is implemented by calling a neural network model; The training process of the neural network model includes: Encoding the phoneme sequence sample corresponding to the text sample through the initialized neural network model to obtain the context representation of the phoneme sequence sample; Determining a predicted alignment position of the third frame hidden state relative to the context representation based on the third frame hidden state corresponding to each phoneme in the phoneme sequence sample; When the predicted alignment position corresponds to a non-end position in the context representation, decoding is performed based on the context representation and the third frame implicit state to obtain a fourth frame implicit state; Performing spectrum post-processing based on the third frame implicit state and the fourth frame implicit state to obtain predicted spectrum data corresponding to the text sample; Constructing a loss function of the neural network model based on the predicted spectrum data corresponding to the text sample and the labeled spectrum data corresponding to the text sample; Updating the parameters of the neural network model, and using the updated parameters of the neural network model when the loss function converges as the parameters of the trained neural network model; The third frame implicit state represents the implicit state of the third frame, the fourth frame implicit state represents the implicit state of the fourth frame, and the third frame and the fourth frame are any two adjacent frames in the spectrum data corresponding to each phoneme in the phoneme sequence sample.

9. The method according to claim 8, characterized in that Before updating the parameters of the neural network model, the method further includes: Constructing a parameter matrix based on the parameters of the neural network model; Dividing the parameter matrix into blocks to obtain a plurality of matrix blocks included in the parameter matrix; When the structure is sparse, the mean of the parameters in each of the matrix blocks is determined; Sorting the matrix blocks in ascending order based on the mean value of the parameters in each of the matrix blocks, resetting the parameters in the plurality of matrix blocks that are sorted first in the ascending order result, and obtaining a reset parameter matrix; The reset parameter matrix is ​​used to update the parameters of the neural network model.

10. The method according to claim 8, characterized in that Before constructing the loss function of the neural network model, the method further includes: Obtaining the content text length of the context representation of the phoneme sequence sample; When the predicted alignment position corresponds to the end position in the context representation, constructing a position loss function of the neural network model based on the predicted alignment position and the content text length; The loss function of the neural network model is constructed based on the predicted spectrum data corresponding to the text sample and the labeled spectrum data corresponding to the text sample, including: Constructing a spectrum loss function of the neural network model based on the predicted spectrum data corresponding to the text sample and the labeled spectrum data corresponding to the text sample; A weighted sum is performed on the spectrum loss function and the position loss function, and the weighted sum result is used as the loss function of the neural network model.

11. The method according to claim 1, characterized in that: The encoding process of the phoneme sequence corresponding to the text to obtain the context representation of the phoneme sequence includes: Performing forward encoding processing on the phoneme sequence corresponding to the text to obtain a forward latent vector of the phoneme sequence; Performing backward encoding processing on the phoneme sequence corresponding to the text to obtain a backward latent vector of the phoneme sequence; The forward latent vector and the backward latent vector are fused to obtain a context representation of the phoneme sequence.

12. An audio generating device, characterized in that: The device comprises: An encoding module, used to encode the phoneme sequence corresponding to the text to obtain a context representation of the phoneme sequence; an attention module, configured to perform the following processing for each phoneme in the phoneme sequence: determining an alignment position of the t-th frame hidden state relative to the context representation based on the t-th frame hidden state corresponding to the phoneme; A decoding module, configured to perform decoding processing based on the context representation and the implicit state of the tth frame to obtain the implicit state of the t+1th frame when the alignment position of the implicit state of the tth frame relative to the context representation corresponds to a non-end position in the context representation, wherein t is a natural number that increases from 1 and satisfies 1≤t≤T, and T is the total number of frames corresponding to the phoneme sequence when the alignment position corresponds to the end position in the context representation, and the total number of frames represents the number of frames of spectral data corresponding to the implicit state of each phoneme in the phoneme sequence; A synthesis module is used to perform splicing processing on the implicit state of the T frame when the alignment position corresponds to the end position in the context representation to obtain audio data corresponding to the text.

13. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor, used to implement the artificial intelligence-based audio generation method described in any one of claims 1 to 11 when executing the executable instructions stored in the memory.

14. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to implement the artificial intelligence-based audio generation method described in any one of claims 1 to 11 when executed by a processor.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the artificial intelligence-based audio generation method according to any one of claims 1 to 11 is implemented.

Images