Method and apparatus for sequence speech synthesis based on monotonicity constraint function

Abstract

The embodiment of the application belongs to the field of artificial intelligence, is applied to the field of sequence speech synthesis, and relates to a method for sequence speech synthesis based on monotonicity constraint function, which comprises the following steps: obtaining a to-be-synthesized sequence text; obtaining a coding sequence based on an encoder; obtaining a feature sequence through a decoder; obtaining attention weight values corresponding to each feature value through an attention mechanism with monotonicity constraint; performing weighted average on each feature value and the attention weight value corresponding to the feature value to obtain a context vector; and realizing sequence speech synthesis through each feature value and the context vector corresponding to the feature value. The application also provides a device for sequence speech synthesis based on monotonicity constraint function, a computer device and a storage medium. Compared with a traditional sequence speech synthesis method, the training and synthesis process of the application is simple, the alignment speed is improved, the alignment problem is reduced, and the synthesis stability is improved under the condition of ensuring the monotonicity of the attention mechanism.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and sequence speech synthesis, and particularly relates to a method and device for sequence speech synthesis based on monotonicity constraint function. BACKGROUND

[0002] The sequence-to-sequence model uses an attention mechanism to align the input and output sequences, which is the main paradigm of current end-to-end speech synthesis. End-to-end speech synthesis methods have proven that the naturalness of synthesized speech in certain specific fields can be comparable to human real speech. Despite these successes, the alignment based on the attention mechanism has robustness problems, which can lead to incomplete synthesized speech, possibly missing or repeating some words, and difficulty in generalizing to long texts.

[0003] The traditional attention mechanism is mainly based on content information, without using the monotonicity and locality of speech synthesis alignment. The improved hybrid attention mechanism combines content and position information, improving the generalization ability to long texts. However, it still has occasional alignment problems. Some attention mechanisms that use monotonicity improve the stability of synthesized speech, increase the alignment speed, and reduce alignment problems, but have problems such as complex training process. SUMMARY

[0004] The purpose of the embodiments of the present application is to propose a method and device for sequence speech synthesis based on monotonicity constraint function, computer equipment and storage medium, so as to reduce alignment problems and improve the stability and alignment speed of sequence speech synthesis in the case of simple training process.

[0005] To solve the above technical problems, the embodiments of the present application provide a method for sequence speech synthesis based on monotonicity constraint function, which adopts the following technical solutions:

[0006] A method for sequence speech synthesis based on monotonicity constraint function, comprising the following steps:

[0007] Obtaining a sequence text to be synthesized;

[0008] Encoding the sequence text based on a preset encoder to obtain an encoded sequence mapped in a hidden state;

[0009] Extracting feature values from the encoded sequence through a preset decoder to obtain a feature sequence;

[0010] Passing the feature sequence into an attention mechanism with a monotonicity constraint function to obtain attention weight values corresponding to each feature value in the feature sequence;

[0011] weighting average the feature values in the feature sequence with the attention weight values corresponding to the feature values to obtain a context vector;

[0012] The current feature value in the feature sequence and the context vector corresponding to the current feature value are taken as sequence speech synthesis parameters to synthesize a sequence speech segment corresponding to the current feature value, the sequence speech segment corresponding to the current feature value is spliced with the sequence speech synthesis result obtained by the previous feature value to obtain a sequence speech synthesis result corresponding to the current feature value, and sequence speech synthesis at the current feature value is completed.

[0013] Further, before the feature sequence is input into the attention mechanism with the monotonicity constraint function, the method further comprises:

[0014] The softplus function and the sigmoid function are taken as the monotonicity constraint function of the attention mechanism.

[0015] Further, before the step of inputting the feature sequence into the attention mechanism with the monotonicity constraint function, the method further comprises:

[0016] Two layers of feedforward neural networks MLP are set in the preset decoder in advance,

[0017] The variable value between two adjacent feature values in the feature sequence is obtained through one of the layers of feedforward neural networks MLP;

[0018] The standard deviation of all feature values before the current feature value in the feature sequence is obtained through another layer of feedforward neural networks MLP;

[0019] The variable value and the standard deviation are respectively positively processed based on the softplus function.

[0020] Further, the encoding and decoding process is completed by two layers of recurrent neural networks RNN, one of the layers of recurrent neural networks RNN is taken as the encoder to perform the encoding process, and the other layer of recurrent neural networks RNN is taken as the decoder to perform the decoding process, and the two layers of feedforward neural networks MLP are set in the preset decoder in advance, specifically, the two layers of feedforward neural networks MLP are set before the last hidden layer of the recurrent neural network RNN performing the decoding process.

[0021] Further, the feature sequence is input into the attention mechanism with the monotonicity constraint function to obtain the attention weight value corresponding to each feature value in the feature sequence, specifically comprising:

[0022] The positively processed variable value and the standard deviation are taken as parameters and input into a preset first algorithm formula and a preset second algorithm formula in sequence;

[0023] The operation results of the first algorithm formula and the second algorithm formula are transmitted into the sigmoid function as relative parameters respectively, and difference operation is performed to obtain a difference operation result as the attention weight value.

[0024] Further, the first algorithm formula and the second algorithm formula are specifically:

[0025] The first algorithm formula is: The second algorithm formula is: Wherein, u i = u i-1 - Δ i , i is the subscript value of the current feature value in the feature sequence, the initial value of i is 1, u i The initial value, that is, u1=0, j is the subscript value of the current element in the sequence text, the initial value of j is 1, σ i is the standard deviation, and Δ i is the variable value.

[0026] To solve the above technical problems, the embodiment of the application further provides a device for sequence speech synthesis based on a monotonicity constraint function, which adopts the following technical scheme:

[0027] A device for sequence speech synthesis based on a monotonicity constraint function, comprising:

[0028] A sequence text acquisition module is configured to acquire a sequence text to be synthesized into speech.

[0029] An encoded sequence acquisition module is configured to encode the sequence text based on a preset encoder to acquire an encoded sequence mapped in a hidden state.

[0030] A feature sequence acquisition module is configured to extract feature values from the encoded sequence by a preset decoder to acquire a feature sequence.

[0031] An attention weight value acquisition module is configured to transmit the feature sequence into an attention mechanism with a monotonicity constraint function to acquire an attention weight value corresponding to each feature value in the feature sequence.

[0032] A context vector acquisition module is configured to perform weighted average on each feature value in the feature sequence and the attention weight value corresponding thereto to acquire a context vector.

[0033] The sequence speech synthesis module is configured to take the current feature value in the feature sequence and the context vector corresponding to the current feature value as sequence speech synthesis parameters, synthesize a sequence speech segment corresponding to the current feature value, splice the sequence speech segment corresponding to the current feature value and a sequence speech synthesis result obtained by a previous feature value, and obtain a sequence speech synthesis result corresponding to the current feature value, thereby completing sequence speech synthesis at the current feature value.

[0034] Further, the attention weight value acquisition module comprises a softplus function processing submodule and a sigmoid function processing submodule.

[0035] The softplus function processing submodule is configured to pre-set two layers of feedforward neural networks in the preset decoder, acquire a variable value between two adjacent feature values in the feature sequence through one of the two layers of feedforward neural networks, acquire a standard deviation of all feature values before the current feature value in the feature sequence through the other layer of feedforward neural networks, and perform positive value processing on the variable value and the standard deviation based on a softplus function.

[0036] The sigmoid function processing submodule is configured to take the variable value and the standard deviation after the positive value processing as parameters, sequentially input the parameters into a preset first algorithm formula and a preset second algorithm formula, take operation results of the first algorithm formula and the second algorithm formula as relative parameters, respectively input the relative parameters into the sigmoid function, and perform difference value operation to obtain a difference value operation result as the attention weight value.

[0037] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0038] The method for sequence speech synthesis based on a monotonicity constraint function provided in the embodiments of the present application comprises the following steps: obtaining a sequence text to be synthesized; obtaining an encoded sequence based on an encoder; obtaining a feature sequence through a decoder; obtaining an attention weight value corresponding to each feature value through an attention mechanism with a monotonicity constraint function; obtaining a context vector by weightedly averaging each feature value and the attention weight value corresponding to the feature value; and taking each feature value and the context vector corresponding to the feature value as sequence speech synthesis parameters to perform sequence speech synthesis. Compared with a traditional sequence speech synthesis method, the training and synthesis process of the present application is simple, the alignment speed is improved, the alignment problem is reduced, and the synthesis stability is improved under the condition of ensuring the monotonicity of the attention mechanism. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the solutions in the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced as follows. Obviously, the drawings in the following description only represent some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0040] Figure 1 is an exemplary system architecture diagram to which the present application can be applied;

[0041] Figure 2 a flow chart of one embodiment of the method for sequence speech synthesis based on monotonicity constraint function according to the present application;

[0042] Figure 3 an execution architecture diagram of one embodiment of the method for sequence speech synthesis based on monotonicity constraint function according to the present application;

[0043] Figure 4 is a structural schematic diagram of one embodiment of the device for sequence speech synthesis based on monotonicity constraint function according to the present application;

[0044] Figure 5 is Figure 4 is a structural schematic diagram of one specific embodiment of the attention weight value obtaining module;

[0045] Figure 6 is a structural schematic diagram of one embodiment of the computer device according to the present application. DETAILED DESCRIPTION

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs; the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application; the description and the claims of the present application and the above drawings, the terms "comprising" and "having" and any variations thereof are intended to cover not exclusively containing. The description and claims of the present application or the above drawings, the terms "first", "second" and the like are used to distinguish different objects, not to describe a particular order.

[0047] Reference herein to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification indicates that, unless otherwise indicated, the described features, structures, or characteristics can be combined in any suitable manner. It is explicitly and implicitly understood that the embodiments described herein can be combined with other embodiments.

[0048] In order to better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described below with reference to the drawings.

[0049] As shown in Figure 1 , the system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0050] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0051] The terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smartphones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0052] The server 105 can be a server providing various services, such as a background server supporting the pages displayed on the terminal devices 101, 102, 103.

[0053] It should be noted that the method for performing sequence speech synthesis based on monotonicity constraint function provided by the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the device for performing sequence speech synthesis based on monotonicity constraint function is generally arranged in a server / terminal device.

[0054] It should be understood that Figure 1 the number of terminal devices, networks, and servers in

[0055] Continuing to refer to Figure 2The diagram illustrates a flowchart of an embodiment of the method for sequence speech synthesis based on a monotonicity constraint function according to this application. The method for sequence speech synthesis based on a monotonicity constraint function includes the following steps:

[0056] Step 201: Obtain the text sequence to be used for speech synthesis. Where L is a positive integer representing the length of the sequence text, j is the index of the current element in the sequence text, and the initial value of j is 1, j≤L.

[0057] In this embodiment, the sequence text Essentially, it is {x1, x2, x3, ..., x} j-1 ,x j A set of text elements in the form of} to be synthesized can be understood as the sequence of text consisting of L text elements to be synthesized.

[0058] In this embodiment, the method for sequence speech synthesis based on monotonic constraint functions runs on electronic devices (e.g., Figure 1 The server / terminal device shown can acquire the sequence text to be synthesized via wired or wireless connection, receive requests for sequence speech synthesis, and use the sequence text as the data to be synthesized for sequence speech synthesis. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future known wireless connection methods.

[0059] Step 202: Encode the sequence text based on a preset encoder to obtain its encoded sequence mapped to the hidden state. Where L is a positive integer representing the length of the encoded sequence, j is the index of the current element in the encoded sequence, and the initial value of j is 1, j≤L.

[0060] In this embodiment, the preset encoder uses a first recurrent neural network (RNN) to process the sequence text. Encode the sequence to obtain its mapping under the hidden layer of the first recurrent neural network. Essentially, the sequence text is obtained in the hidden layer of the first recurrent neural network. The encoded value with the same number of elements, i.e. This can be represented as {h1,h2,h3,…,h j-1 ,h jA set of codes in the form of} can be understood as a sequence of codes consisting of L codes, each of which is encoded by the text element to be synthesized for speech at the corresponding position.

[0061] Step 203: Extract feature values ​​from the encoded sequence using a preset decoder to obtain a feature sequence. Where T is a positive integer representing the number of frames of the target speech sequence to be synthesized, i is the index of the current feature value in the feature sequence, and the initial value of i is 1, i≤T.

[0062] In this embodiment, the preset decoder uses a second recurrent neural network (RNN) to process the encoded sequence. Decode the sequence to obtain its mapping under the first hidden layer of the second recurrent neural network. In essence, T decoded values ​​are obtained in the first hidden layer of the second recurrent neural network, that is... This can be represented as {s1,s2,s3,…,s} i-1 ,s i A decoded set in the form of} can be understood as the encoded sequence consisting of T decoded values, each of which is a feature value.

[0063] Step 204: Input the feature sequence into an attention mechanism with a monotonicity constraint function to obtain the attention weight value corresponding to each feature value in the feature sequence.

[0064] In this embodiment, before passing the feature sequence into the attention mechanism with a monotonicity constraint function, the method further includes: using the softplus function and the sigmoid function as the monotonicity constraint functions of the attention mechanism.

[0065] In this embodiment, before the step of feeding the feature sequence into the attention mechanism with a monotonicity constraint function, the method further includes: pre-setting two layers of feedforward neural networks (Multilayer Perceptron, MLP) in the preset decoder; obtaining the variable value between two adjacent feature values ​​in the feature sequence through one layer of the feedforward neural network; obtaining the standard deviation of all previous feature values ​​in the feature sequence, including the current feature value, through the other layer of the feedforward neural network; and performing positive value processing on the variable value and the standard deviation based on the softplus function. Only by ensuring the variable value and the standard deviation can the monotonicity of the attention be guaranteed, that is, the correspondence between the j-th frame of the synthesized speech sequence and the i-th element in the text sequence is monotonic.

[0066] In the embodiment, two layers of feedforward neural networks are arranged in advance in the preset decoder, specifically, the first feedforward neural network and the second feedforward neural network are arranged before the last hidden layer of the second recurrent neural network, that is, two layers of the feedforward neural networks are arranged before the last hidden layer of the recurrent neural network performing the decoding process. Since the recurrent neural network (RNN) decoder obtains the attention weight value a ij mapped in the last hidden layer of the RNN recurrent neural network, in order to ensure that the attention weight value a ij is obtained in the case of attention monotony, two layers of feedforward neural networks are introduced before the last hidden layer of the recurrent neural network (RNN) performing the decoding process to process the variable value and the standard deviation.

[0067] In the embodiment, the softplus function is used to perform positive value processing on the variable value and the standard deviation, specifically, the variable value is taken as a parameter, and the softplus function Y(x)=log(1+e x ) is used to obtain a positive value processing result corresponding to the variable value. Similarly, the standard deviation is taken as a parameter, and the softplus function Y(x)=log(1+e x ) is used to obtain a positive value processing result corresponding to the standard deviation.

[0068] In the embodiment, the feature sequence is input into the attention mechanism with a monotonicity constraint function to obtain an attention weight value corresponding to each feature value in the feature sequence, specifically, the variable value and the standard deviation after positive value processing are taken as parameters and sequentially input into a preset first algorithm formula and a preset second algorithm formula; the operation results of the first algorithm formula and the second algorithm formula are taken as relative parameters and input into the sigmoid function: and difference operation is performed to obtain an attention weight value corresponding to each feature value in the feature sequence.

[0069] Specifically, since the recurrent neural network decoder obtains the attention weight value a ij mapped in the last hidden layer of the recurrent neural network, and the attention weight value a ij is obtained through the decoding sequence mapped in the first hidden layer of the second recurrent neural network, therefore, two layers of fully connected feedforward neural networks are introduced before the last hidden layer of the second recurrent neural network to process the decoding sequence , and the processing manner is that the first feedforward neural network is used to obtain a variable value Δ i; through the second feedforward neural network, a standard deviation σ of all previous characteristic values including the current characteristic value in the characteristic sequence is acquired i Since the decoding process is continuous, assuming that the decoder is decoding the encoding sequence and 3 decoding sequences have been acquired, the standard deviation of the 3 decoding sequences is acquired, and by analogy, assuming that 10 decoding sequences have been acquired, the standard deviation of the 10 decoding sequences is acquired, and by analogy, assuming that i decoding sequences have been acquired, the standard deviation of the i decoding sequences is acquired; after the variable value Δ i and the standard deviation σ i are acquired, in order to ensure that both are positive values, a softplus function Y(x) = log(1 + e x ) is introduced to perform positive value processing on both, that is, Correspondingly, the softplus function can be encapsulated, and the variable value Δ i and the standard deviation σ i are encapsulated in an execution block as two variables of a method block Y(Δ i , σ i ) as input parameters to perform positive value conversion; then, the variable value Δ i and the standard deviation σ i after the positive value processing are sequentially transmitted as parameters into a preset first algorithm formula and a preset second algorithm formula, wherein the preset first algorithm formula is and the preset second algorithm formula is wherein u i = u i-1 - Δ i , i is a subscript value of the current characteristic value in the characteristic sequence, the initial value of i is 1, u i is the initial value, that is, u1 = 0, j is a subscript value of the current element in the sequence text, the initial value of j is 1, σ i is the standard deviation, and Δ i is the variable value; the result acquired through the preset first algorithm formula and the preset second algorithm formula is transmitted as a relative parameter into the sigmoid function: and difference value operation is performed to acquire a difference value operation result as the attention weight value a ij of each characteristic value in the characteristic sequence, that is, wherein 1 ≤ j ≤ L, and L is the length of the sequence text, so that the sigmoid function is used to ensure the monotonicity of attention, that is, the corresponding relationship between the jth frame of speech and the ith character of the text is monotonic.

[0070] Step 205, weighting average each feature value in the feature sequence with its corresponding attention weight value to obtain a context vector.

[0071] In this embodiment, the feature values in the feature sequence are weighted and averaged with the corresponding attention weight values to obtain a context vector, specifically including: through a preset weighted average algorithm formula: obtain the context vector c corresponding to each feature value i .

[0072] Step 206, taking the current feature value in the feature sequence and the context vector corresponding to the current feature value as a sequence speech synthesis parameter, synthesizing a sequence speech segment corresponding to the current feature value, splicing the sequence speech segment corresponding to the current feature value with the sequence speech synthesis result obtained by the previous feature value to obtain the sequence speech synthesis result corresponding to the current feature value, and completing the sequence speech synthesis at the current feature value.

[0073] In this embodiment, the current feature value in the feature sequence and the context vector corresponding to the current feature value are taken as a sequence speech synthesis parameter to synthesize a sequence speech segment corresponding to the current feature value, and the sequence speech segment corresponding to the current feature value is spliced with the sequence speech synthesis result obtained by the previous feature value to obtain the sequence speech synthesis result corresponding to the current feature value, and the sequence speech synthesis at the current feature value is completed, and the specific implementation manner is: assuming that the sequence speech synthesis function is f(s i ,c i ), directly taking the feature value s i and the context vector c i corresponding to the feature value in the feature sequence as a sequence speech synthesis parameter to perform speech synthesis on the i-th element of the sequence text. Since the sequence speech synthesis is dynamic, that is, after the first feature value obtains the synthesized segment, it is stored first, then the second feature value obtains the synthesized segment, which is spliced with the previously stored to serve as the second synthesis result, and in turn, the current feature value obtains the synthesized segment, which is spliced with the synthesis result of the previous step to complete the current sequence speech synthesis.

[0074] In this embodiment, the encoding and decoding process is completed by two layers of recurrent neural networks, one layer of recurrent neural networks is used as the encoder to perform the encoding process, and the other layer of recurrent neural networks is used as the decoder to perform the decoding process, and two layers of feedforward neural networks are set in the preset decoder in advance, specifically: the two layers of feedforward neural networks are set before the last hidden layer of the recurrent neural network performing the decoding process.

[0075] With reference to the above Figure 3, shows an execution architecture diagram of one embodiment of a method for sequence speech synthesis based on monotonicity constraint function according to the present application. The method for sequence speech synthesis based on monotonicity constraint function specifically comprises: encoding the sequence text based on a recurrent neural network encoder to obtain an encoded sequence mapped in a hidden state; extracting feature values of the encoded sequence through a recurrent neural network decoder to obtain a feature sequence; setting two layers of feedforward neural networks in front of the last hidden layer of the recurrent neural network performing the decoding process; obtaining a variable value between two adjacent feature values in the feature sequence through one of the layers of feedforward neural networks; obtaining a standard deviation of all feature values before the current feature value in the feature sequence through the other layer of feedforward neural networks; performing positive value processing on the variable value and the standard deviation based on a softplus function, taking the variable value and the standard deviation after the positive value processing as parameters, and sequentially inputting the parameters into a preset first algorithm formula and a preset second algorithm formula; taking operation results of the first algorithm formula and the second algorithm formula as relative parameters, respectively inputting the relative parameters into a sigmoid function, and performing difference value operation to obtain a difference value operation result as the attention weight value; performing weighted average on each feature value in the feature sequence and the attention weight value corresponding to the feature value to obtain a context vector; and taking each feature value in the feature sequence and the context vector corresponding to the feature value as a sequence speech synthesis parameter to perform sequence speech synthesis.

[0076] The present application performs positive value processing on the feature value through a softplus function, and then introduces a sigmoid function as a constraint function into an attention mechanism. The sequence text to be synthesized is obtained based on an encoder to obtain an encoded sequence, and a decoder is used to obtain a feature sequence. The attention weight value corresponding to each feature value is obtained through the attention mechanism with monotonicity constraint, under the condition that the text corresponding relationship is monotonic. The context vector is obtained by performing weighted average on each feature value and the attention weight value corresponding to the feature value. The sequence speech synthesis parameter is obtained by taking each feature value and the context vector corresponding to the feature value. Compared with the traditional sequence speech synthesis method, the alignment problem can be reduced and the stability and alignment speed of sequence speech synthesis can be improved in the case of simple training process.

[0077] The embodiment of the present application can obtain and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0078] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric identification technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc. The present application can be applied to the field of sequence speech synthesis, so as to ensure that the text corresponding relationship is monotonic, which is convenient for reducing the alignment problem and improving the stability and alignment speed of sequence speech synthesis in the case of simple training process.

[0079] The present application belongs to the field of sequence speech synthesis, and the monotonicity of attention can be ensured by the present solution, so as to ensure that the text corresponding relationship is monotonic.

[0080] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by computer readable instructions instructing related hardware, and the computer readable instructions can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of each method. Among them, the storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0081] It should be understood that, although each step in the flowchart of the accompanying drawings is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise stated herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other orders. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or sub-steps or stages of other steps.

[0082] Further referring to Figure 4 , as an implementation of the method shown in Figure 2 , the present application provides an embodiment of a device for sequence speech synthesis based on monotonicity constraint function. The device embodiment corresponds to the method embodiment shown in Figure 2 , and the device can be specifically applied to various electronic devices.

[0083] As Figure 4As shown, the device 400 for sequence speech synthesis based on monotonicity constraint function in the embodiment comprises a sequence text acquisition module 401, an encoded sequence acquisition module 402, a feature sequence acquisition module 403, an attention weight value acquisition module 404, a context vector acquisition module 405, and a sequence speech synthesis module 406. Among them:

[0084] The sequence text acquisition module 401 is configured to acquire a sequence text to be synthesized. Wherein, L is a positive integer, representing the length of the sequence text, j is the subscript value of the current element in the sequence text, the initial value of j is 1, and j≤L;

[0085] The encoded sequence acquisition module 402 is configured to encode the sequence text based on a preset encoder to acquire an encoded sequence mapped in a hidden state. Wherein, L is a positive integer, representing the length of the encoded sequence, j is the subscript value of the current element in the encoded sequence, the initial value of j is 1, and j≤L;

[0086] The feature sequence acquisition module 403 is configured to extract feature values from the encoded sequence through a preset decoder to acquire a feature sequence. Wherein, T is a positive integer, representing the frame number of the target sequence speech to be synthesized, i is the subscript value of the current feature value in the feature sequence, the initial value of i is 1, and i≤T;

[0087] The attention weight value acquisition module 404 is configured to input the feature sequence into an attention mechanism with a monotonicity constraint function to acquire attention weight values corresponding to each feature value in the feature sequence.

[0088] The context vector acquisition module 405 is configured to weight average each feature value in the feature sequence and its corresponding attention weight value to acquire a context vector.

[0089] The sequence speech synthesis module 406 is configured to take the current feature value in the feature sequence and the context vector corresponding to the current feature value as sequence speech synthesis parameters, synthesize a sequence speech segment corresponding to the current feature value, splice the sequence speech segment corresponding to the current feature value with the sequence speech synthesis result obtained by the previous feature value to acquire the sequence speech synthesis result corresponding to the current feature value, and complete the sequence speech synthesis at the current feature value.

[0090] Referring to Figure 5 is a structural schematic diagram of one specific implementation of the attention weight value acquisition module, which comprises a softplus function processing submodule 4041 and a sigmoid function processing submodule 4042,

[0091] The softplus function processing submodule 4041 is used to pre-set two layers of feedforward neural networks in the preset decoder; obtain the variable value between two adjacent feature values ​​in the feature sequence through one layer of feedforward neural network; obtain the standard deviation of all previous feature values ​​in the feature sequence, including the current feature value, through the other layer of feedforward neural network; and perform positive value processing on the variable value and the standard deviation based on the softplus function.

[0092] The sigmoid function processing submodule 4042 is used to take the positiveized variable value and the standard deviation as parameters, and pass them in sequentially to a preset first algorithm formula and a preset second algorithm formula; take the operation results of the first algorithm formula and the second algorithm formula as relative parameters, pass them in to the sigmoid function respectively, and perform difference operation to obtain the difference operation result as the attention weight value.

[0093] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0094] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected via a system bus. It should be noted that only the computer device 6 with components 61-63 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0095] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0096] The memory 61 includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or a memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 6. Of course, the memory 61 can also include both an internal storage unit and an external storage device of the computer device 6. In this embodiment, the memory 61 is generally used to store an operating system and various application software installed on the computer device 6, such as computer readable instructions of the method for performing sequential speech synthesis based on monotonicity constraint functions, etc. In addition, the memory 61 can also be used to temporarily store various data that have been output or will be output.

[0097] The processor 62 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run computer readable instructions or process data stored in the memory 61, such as computer readable instructions of the method for performing sequential speech synthesis based on monotonicity constraint functions.

[0098] The network interface 63 can include a wireless network interface or a wired network interface, which is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0099] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the method described in each embodiment of the present application.

[0100] Obviously, the above-described embodiments are only some of the embodiments of the present application, not all the embodiments, and the preferred embodiments of the present application are given in the drawings, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing specific embodiments, or make equivalent replacements to some of the technical features. Any equivalent structure made by using the content of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the scope of the patent protection of the present application.

Claims

1. A method for sequence speech synthesis based on a monotonic constraint function, characterized in that, Includes the following steps: Obtain the text sequence to be used for speech synthesis; The sequence text is encoded based on a preset encoder to obtain its encoded sequence mapped to the hidden state; Feature sequences are obtained by extracting feature values ​​from the encoded sequence using a preset decoder. The method further includes: inputting the feature sequence into an attention mechanism with a monotonicity constraint function to obtain the attention weight values ​​corresponding to each feature value in the feature sequence; wherein, before executing the step of inputting the feature sequence into the attention mechanism with a monotonicity constraint function, the method further includes: Two layers of feedforward neural network MLP are pre-configured within the preset decoder; The variable values ​​between two adjacent feature values ​​in the feature sequence are obtained through one layer of the feedforward neural network MLP; The standard deviation of all previous feature values, including the current feature value, is obtained by using another feedforward neural network MLP. The variable values ​​and the standard deviation are positiveized using the softplus function; The step of feeding the feature sequence into an attention mechanism with monotonic constraints to obtain the attention weight value corresponding to each feature value in the feature sequence specifically includes: The variable value after positive value processing and the standard deviation are used as parameters and sequentially passed into the preset first algorithm formula and the preset second algorithm formula, wherein the first algorithm formula and the second algorithm formula are specifically as follows: The first algorithm formula is: The second algorithm formula is: In the formula, , The index value of the current feature value in the feature sequence. The initial value is 1, that is for initial value, , The index value of the current element in the sequence text. The initial value is 1. The standard deviation is... The value of the variable; The results of the first and second algorithm formulas are used as relative parameters and passed to the sigmoid function respectively. The difference is then calculated and the result is used as the attention weight value. The context vector is obtained by taking a weighted average of each feature value in the feature sequence and its corresponding attention weight value. The current feature value and the context vector corresponding to the current feature value in the feature sequence are used as parameters for sequence speech synthesis. The sequence speech segment corresponding to the current feature value is synthesized. The sequence speech segment corresponding to the current feature value is concatenated with the sequence speech synthesis result obtained from the previous feature value to obtain the sequence speech synthesis result corresponding to the current feature value, thus completing the sequence speech synthesis at the current feature value.

2. The method for sequence speech synthesis based on monotonic constraint functions according to claim 1, characterized in that, The encoding and decoding process is completed by two layers of recurrent neural networks (RNNs). One layer of RNNs is used as the encoder to perform the encoding process, and the other layer of RNNs is used as the decoder to perform the decoding process. Specifically, the two layers of feedforward neural networks (MLPs) are set in the preset decoder before the last hidden layer of the RNN that performs the decoding process.

3. A device for sequence speech synthesis based on a monotonicity constraint function, characterized in that, The apparatus for sequence speech synthesis based on a monotonicity constraint function is used to implement the steps of the method for sequence speech synthesis based on a monotonicity constraint function as described in any one of claims 1 to 2, wherein the apparatus for sequence speech synthesis based on a monotonicity constraint function comprises: The sequence text acquisition module is used to acquire the sequence text to be used for speech synthesis; The encoding sequence acquisition module is used to encode the sequence text based on a preset encoder to obtain its encoding sequence mapped in the hidden state; The feature sequence acquisition module is used to extract feature values ​​from the encoded sequence using a preset decoder to acquire a feature sequence; The attention weight value acquisition module is used to input the feature sequence into an attention mechanism with monotonicity constraints and obtain the attention weight value corresponding to each feature value in the feature sequence. The attention weight value acquisition module includes a softplus function processing submodule and a sigmoid function processing submodule. The softplus function processing submodule is used to pre-set two layers of feedforward neural networks in the preset decoder. Through one layer of feedforward neural network, the variable value between two adjacent feature values ​​in the feature sequence is obtained. Through the other layer of feedforward neural network, the standard deviation of all previous feature values ​​in the feature sequence, including the current feature value, is obtained. The variable value and the standard deviation are positiveized based on the softplus function. The sigmoid function processing submodule is used to take the positiveized variable value and the standard deviation as parameters, and sequentially input them into a preset first algorithm formula and a preset second algorithm formula. The results of the calculations of the first and second algorithm formulas are used as relative parameters and input into the sigmoid function respectively. A difference operation is performed, and the difference result is used as the attention weight value. Specifically, the first and second algorithm formulas are: The first algorithm formula is: The second algorithm formula is: In the formula, , The index value of the current feature value in the feature sequence. The initial value is 1, that is for initial value, , The index value of the current element in the sequence text. The initial value is 1. The standard deviation is... The value of the variable; The context vector acquisition module is used to perform a weighted average of each feature value in the feature sequence and its corresponding attention weight value to obtain the context vector. The sequence speech synthesis module is used to take the current feature value and the context vector corresponding to the current feature value in the feature sequence as sequence speech synthesis parameters, synthesize the sequence speech segment corresponding to the current feature value, concatenate the sequence speech segment corresponding to the current feature value with the sequence speech synthesis result obtained from the previous feature value, obtain the sequence speech synthesis result corresponding to the current feature value, and complete the sequence speech synthesis at the current feature value.

4. A computer device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor, when executing the computer-readable instructions, implements the steps of the method for sequence speech synthesis based on a monotonicity constraint function as described in any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the method for sequence speech synthesis based on a monotonicity constraint function as described in any one of claims 1 to 2.

Images