Speech synthesis method, device, electronic device and storage medium

Abstract

The present invention relates to speech synthesis technology, and discloses a speech synthesis method, comprising: obtaining a character vector, performing attention calculation on the character vector using a multi-head attention network to obtain an attention vector; performing residual connection on the attention vector and the character vector to obtain a character attention vector; performing feature extraction on the character attention vector using a character feature extraction network to obtain a character feature sequence; inputting the character vector into a pre-constructed pronunciation pause prediction model to obtain a pronunciation pause sequence; performing residual connection on the character feature sequence and the pronunciation pause sequence to obtain a speech sequence, and performing speech synthesis on the speech sequence using a pre-constructed vocoder synthesizer to obtain the synthesized speech of the character text. The present invention also discloses a speech synthesis device, an electronic device, and a storage medium. The present invention can solve the problem that the synthesized speech is not smooth and natural enough.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis, and in particular to a speech synthesis method, device, electronic equipment and computer-readable storage medium. Background Art

[0002] With the rapid development of deep learning, speech synthesis methods based on deep learning networks have sprung up. Currently, commonly used speech synthesis methods include LSTM synthesis method, BERT synthesis method, etc. Although these methods can achieve speech synthesis, due to the lack of improvement in the naturalness and fluency of speech, the synthesized speech is not smooth and natural enough. Summary of the invention

[0003] The present invention provides a speech synthesis method, device, electronic device and computer-readable storage medium, the main purpose of which is to solve the problem that the synthesized speech is not smooth and natural enough.

[0004] To achieve the above object, the present invention provides a speech synthesis method, comprising:

[0005] Receiving a character text, performing pinyin replacement on the character text to obtain a character pinyin, and using a pre-constructed alphabet table to calculate a character position of the character pinyin in the alphabet table;

[0006] Performing encoding operation on the character position and the character pinyin to obtain a character vector;

[0007] Inputting the character vector into a pre-trained attention feature model, wherein the attention feature model includes a multi-head attention network and a character feature extraction network;

[0008] Using the multi-head attention network to perform attention calculation on the character vector to obtain an attention vector;

[0009] Performing a residual connection on the attention vector and the character vector to obtain a character attention vector;

[0010] Using the character feature extraction network, perform feature extraction on the character attention vector to obtain a character feature sequence;

[0011] Inputting the character vector into a pre-built pronunciation pause prediction model to obtain a pronunciation pause sequence;

[0012] The character feature sequence and the pronunciation pause sequence are subjected to residual connection to obtain a speech sequence, and a pre-built vocoder is used to perform speech synthesis on the speech sequence to obtain synthesized speech of the character text.

[0013] Optionally, the using the character feature extraction network to perform feature extraction on the character attention vector to obtain a character feature sequence includes:

[0014] Normalizing the character attention vector to obtain a character normalized vector;

[0015] Performing a convolution operation on the normalized vector to obtain a character convolution vector;

[0016] Perform a residual connection on the character convolution vector and the character attention vector to obtain the character feature sequence.

[0017] Optionally, performing a convolution operation on the normalized vector to obtain a character convolution vector includes:

[0018] Construct a convolution kernel according to the preset convolution kernel dimension;

[0019] The convolution kernel is used to perform a convolution operation on the normalized vector to obtain the character convolution vector.

[0020] Optionally, the step of inputting the character vector into a pre-built pronunciation pause prediction model to obtain a pronunciation pause sequence comprises:

[0021] Convert the pinyin of the character into a word vector to obtain a pinyin vector;

[0022] Inputting the pinyin vector and the character vector into the pronunciation pause prediction model, and performing Fourier transform on the pinyin vector and the character vector using the pronunciation pause prediction model to obtain a Fourier transform sequence;

[0023] Pronunciation pause prediction is performed on the Fourier transform sequence to obtain the pronunciation pause sequence.

[0024] Optionally, the pre-trained attention feature model includes:

[0025] Step A: constructing an attention feature model to be trained including the multi-head attention network and the character feature extraction network;

[0026] Step B: receiving a training text set and a training label set, inputting the training text set into the attention feature model to be trained for feature extraction, and obtaining a feature sequence training set;

[0027] Step C: Construct multi-layer linear activation layers;

[0028] Step D: using the multi-layer linear activation layer, performing an activation operation on the feature sequence training set to obtain a prediction sequence set;

[0029] Step E: Calculate the error value between the prediction sequence set and the training label set, and determine the magnitude relationship between the error value and a preset error threshold;

[0030] Step F: If the error value is greater than the error threshold, adjust the internal parameters of the attention feature model to be trained, and return to step B;

[0031] Step G: If the error value is less than or equal to the error threshold, the attention feature model is obtained.

[0032] Optionally, the step of inputting the training text set into the attention feature model to be trained for feature extraction to obtain a feature sequence training set comprises:

[0033] Performing pinyin replacement on the training text set to obtain a pinyin training set;

[0034] Calculate the character positions of the pinyin training set in the alphabet to obtain a position training set;

[0035] Performing encoding operation on the pinyin training set and the position training set to obtain a vector training set;

[0036] Using the multi-head attention network to perform attention calculation on the vector training set to obtain an attention vector set;

[0037] Performing residual connection on the attention vector set and the vector training set to obtain an attention vector training set;

[0038] The character feature extraction network is used to perform feature extraction on the attention vector training set to obtain the feature sequence training set.

[0039] Optionally, the using the multi-layer linear activation layer to perform an activation operation on the feature sequence training set to obtain a prediction sequence set includes:

[0040] Normalizing the feature sequence training set to obtain a feature sequence normalized set;

[0041] The Gaussian distribution of the normalized set of feature sequences is calculated, and the predicted sequence set is calculated based on the Gaussian distribution.

[0042] In order to solve the above problem, the present invention further provides a speech synthesis device, the device comprising:

[0043] A character vector construction module is used to receive a character text, perform a pinyin replacement on the character text to obtain a character pinyin, calculate a character position of the character pinyin in the alphabet using a pre-constructed alphabet, perform an encoding operation on the character position and the character pinyin to obtain a character vector;

[0044] A character feature sequence extraction module is used to input the character vector into a pre-trained attention feature model, wherein the attention feature model includes a multi-head attention network and a character feature extraction network, use the multi-head attention network to perform attention calculation on the character vector to obtain an attention vector, perform residual connection on the attention vector and the character vector to obtain a character attention vector, and use the character feature extraction network to perform feature extraction on the character attention vector to obtain a character feature sequence;

[0045] A pronunciation pause sequence extraction module, used for inputting the character vector into a pre-built pronunciation pause prediction model to obtain a pronunciation pause sequence;

[0046] The speech synthesis module is used to perform residual connection on the character feature sequence and the pronunciation pause sequence to obtain a speech sequence, and use a pre-built vocoder synthesizer to perform speech synthesis on the speech sequence to obtain the synthesized speech of the character text.

[0047] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0048] a memory storing at least one instruction; and

[0049] A processor executes instructions stored in the memory to implement any one of the above-mentioned speech synthesis methods.

[0050] In order to solve the above problems, the present invention also provides a computer-readable storage medium, including a data storage area and a program storage area, the data storage area stores created data, and the program storage area stores a computer program; wherein, when the computer program is executed by a processor, any one of the speech synthesis methods described above is implemented.

[0051] The embodiment of the present invention divides speech synthesis into two parts. First, a pre-trained attention feature model is used to perform feature extraction on the character text to obtain a character feature sequence. Secondly, a pronunciation pause prediction model is used to predict the pronunciation pause sequence of the character text. Finally, the character feature sequence and the pronunciation pause sequence are residually connected to obtain a speech sequence. A pre-built voice code synthesizer is used to perform speech synthesis on the speech sequence to obtain the synthesized speech of the character text. Compared with the background technology that simply uses LSTM, BERT and other models for synthesis, the present invention not only predicts the character feature sequence, but also adds a prediction process for the pronunciation pause sequence. Therefore, the synthesized speech is closer to natural human voice in terms of frequency amplitude, etc. Therefore, the speech synthesis method, device and computer-readable storage medium proposed in the present invention can solve the problem that the synthesized speech is not smooth and natural enough. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A flowchart of a speech synthesis method provided by an embodiment of the present invention;

[0053] Figure 2 A detailed flowchart of S6 in the speech synthesis method provided by one embodiment of the present invention;

[0054] Figure 3 A schematic diagram of a module of a speech synthesis device provided by an embodiment of the present invention;

[0055] Figure 4 A schematic diagram of the internal structure of an electronic device for implementing a speech synthesis method provided by an embodiment of the present invention;

[0056] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0058] The embodiment of the present application provides a speech synthesis method, and the execution subject of the speech synthesis method includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the speech synthesis method can be executed by software or hardware installed in a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0059] Reference Figure 1 FIG. 1 is a flow chart of a speech synthesis method provided by an embodiment of the present invention. In this embodiment, the speech synthesis method includes:

[0060] S1. Receive a character text, perform pinyin replacement on the character text to obtain the character pinyin, and use a pre-constructed alphabet to calculate the character position of the character pinyin in the alphabet.

[0061] In a preferred embodiment of the present invention, a character text input by a user can be accepted, such as a character text A input by a user: "Hello, there will be heavy rain and strong winds when traveling today, please pay attention to safety". Then, the character text A is replaced with pinyin to obtain a character pinyin B: "nihao, jintianchuxingbanyoubaoyukuangfeng, qingzhuyianquan". In the embodiment of the present invention, the character text is replaced with pinyin to obtain the character pinyin, including: using pinyin4j in JAVA language to construct a pinyin replacement program; using the pinyin replacement program to replace the character text with pinyin to obtain the character pinyin.

[0062] Among them, pinyin4j is located in net.sourceforge.pinyin4j in JAVA language, so use importnet.sourceforge.pinyin4j to import pinyin4j to obtain the pinyin replacement program.

[0063] In an embodiment of the present invention, the alphabet is constructed using pinyin. For example, in the alphabet, a corresponds to 1, b corresponds to 2, and c corresponds to 3. Then the above-mentioned character pinyin B: "nihao, jintianchuxingbanyoubaoyukuangfeng, qingzhuyianquan" is constructed using the alphabet to obtain character positions including numbers.

[0064] S2. Perform encoding operation on the character position and the character pinyin to obtain a character vector.

[0065] In detail, the embodiment of the present invention adopts a one-hot encoding method to perform encoding operations on the character position and the character pinyin to obtain a character vector.

[0066] S3. Input the character vector into a pre-trained attention feature model, wherein the attention feature model includes a multi-head attention network and a character feature extraction network.

[0067] In the embodiment of the present invention, before executing S3, the attention feature model needs to be trained. Specifically, the training of the attention feature model includes:

[0068] Step A: Construct an attention feature model to be trained including the multi-head attention network and the character feature extraction network.

[0069] In detail, step A includes: constructing the multi-head attention network according to the multi-head attention mechanism; constructing the character feature extraction network according to the convolutional neural network; combining the multi-head attention network and the character feature extraction network to obtain the attention feature model to be trained.

[0070] Among them, constructing the multi-head attention network according to the multi-head attention mechanism includes: receiving a trained Transform model, extracting an encoder from the Transform model, and using the multi-head attention mechanism in the encoder to construct the multi-head attention network.

[0071] In an embodiment of the present invention, a user can complete the training of a Transform model in advance. The Transform model is a deep learning model that can realize classification or fitting, including an encoder and a decoder, wherein the encoder includes a multi-head attention mechanism. In an embodiment of the present invention, the network layer where the multi-head attention mechanism is located is extracted to construct the multi-head attention network.

[0072] Furthermore, in an embodiment of the present invention, the attention feature model to be trained is obtained by combining the multi-head attention network in front and the character feature extraction network in the back.

[0073] Step B: receiving a training text set and a training label set, inputting the training text set into the attention feature model to be trained for feature extraction, and obtaining a feature sequence training set.

[0074] In the embodiment of the present invention, the training text set is a text set collected and sorted by the user in advance, and the training label set is a speech set corresponding to the training text set. For example, the training text set contains training text X 1 : "Bad environment, not suitable for outings", then in the training label set there is a corresponding voice Y 1 =(y 1 ,y 2 , .., y n ), where y n Indicates voice Y 1 speech sequence.

[0075] Furthermore, after obtaining the training text set, the attention feature model to be trained is used to perform feature extraction. Specifically, the training text set is input into the attention feature model to be trained for feature extraction to obtain a feature sequence training set, including: performing pinyin replacement on the training text set to obtain a pinyin training set, calculating the character position of the pinyin training set in the alphabet to obtain a position training set, performing encoding operations on the pinyin training set and the position training set to obtain a vector training set, performing attention calculation on the vector training set using the multi-head attention network to obtain an attention vector set; performing residual connection on the attention vector set and the vector training set to obtain an attention vector training set; and performing feature extraction on the attention vector training set using the character feature extraction network to obtain the feature sequence training set.

[0076] In detail, the process of performing pinyin replacement, character position calculation and encoding operations on the training text set to obtain the vector training set is similar to the above S1 and S2, and will not be repeated here.

[0077] In an embodiment of the present invention, according to the principle of the multi-head attention mechanism of the encoder in the above-mentioned Transform model, attention calculation is performed on the vector training set to obtain the attention vector set.

[0078] Furthermore, the present invention uses the following formula to perform residual connection on the attention vector set and the vector training set:

[0079] Result attention =s+p

[0080] Among them, result attention Represents the attention vector training set, s represents the attention vector set, and p represents the vector training set.

[0081] In the embodiment of the present invention, the convolution operation in the character feature extraction network is used to perform feature extraction on each attention vector in the attention vector training set in turn, thereby obtaining the feature sequence training set. The convolution operation is an operation of performing convolution calculation based on a convolution kernel. The present invention sets the convolution kernel size to 3*3, thereby obtaining the feature sequence training set.

[0082] Step C: Construct multiple linear activation layers.

[0083] After obtaining the attention feature model to be trained and using the attention feature model to be trained to perform feature extraction to obtain a feature sequence training set, the present invention constructs a linear activation layer to help the attention feature model to be trained to perform model training, wherein the linear activation layer includes normalization and activation functions, and the activation function can use a Gaussian distribution function.

[0084] Step D: Utilizing the multi-layer linear activation layer, an activation operation is performed on the feature sequence training set to obtain a prediction sequence set.

[0085] In detail, the method uses the multi-layer linear activation layer to perform an activation operation on the feature sequence training set to obtain a prediction sequence set, including: performing normalization on the feature sequence training set to obtain a feature sequence normalized set, using the Gaussian distribution function to calculate the Gaussian distribution of the feature sequence normalized set, and obtaining the prediction sequence set according to the Gaussian distribution.

[0086] In detail, the normalization is an operation of mapping the numerical values ​​in the feature sequence training set to a specified range. For example, mapping the numerical values ​​in the feature sequence training set to the range of [0, 1] can effectively reduce the numerical values ​​and reduce the calculation pressure.

[0087] Furthermore, the use of the Gaussian distribution function to calculate the Gaussian distribution of the normalized set of feature sequences includes: using the Gaussian distribution function to calculate the mean and variance of the normalized set of feature sequences, and using the mean and variance of the normalized set of feature sequences to solve the Gaussian distribution of the normalized set of feature sequences.

[0088] Since Gaussian distribution is a probability distribution that shows data within a specified range, in the embodiment of the present invention, the maximum probability distribution of the feature sequence training set is found from the Gaussian distribution, that is, the prediction sequence set is obtained.

[0089] Step E: Calculate the error value between the prediction sequence set and the training label set, and determine the magnitude relationship between the error value and a preset error threshold.

[0090] In the embodiment of the present invention, the square error formula is used to calculate the error value between the prediction sequence set and the training label set.

[0091] Step F: If the error value is greater than the error threshold, adjust the internal parameters of the attention feature model to be trained and return to step B.

[0092] Step G: If the error value is less than or equal to the error threshold, the attention feature model of the multi-head attention network and the character feature extraction network is obtained.

[0093] In detail, when the error value is less than or equal to the error threshold, it means that the attention feature model to be trained has a stronger character feature extraction capability, and the training is completed to obtain the attention feature model.

[0094] In an embodiment of the present invention, when executing steps A to G to obtain the trained attention feature model, the character vector can be further input into the pre-trained attention feature model.

[0095] S4. Use the multi-head attention network to perform attention calculation on the character vector to obtain an attention vector.

[0096] In the embodiment of the present invention, S4 is similar to the training stage in S3, and both utilize the multi-head attention mechanism principle of the encoder in the Transform model to perform attention calculation and obtain the attention vector.

[0097] S5. Perform residual connection on the attention vector and the character vector to obtain a character attention vector.

[0098] In the embodiment of the present invention, the following formula is used to perform residual connection on the attention vector and the character vector to obtain the character attention vector:

[0099] character attention =m+u

[0100] Among them, character attention represents the character attention vector, m represents the attention vector, and u represents the character vector.

[0101] S6. Utilize the character feature extraction network to perform feature extraction on the character attention vector to obtain a character feature sequence.

[0102] In the embodiment of the present invention, refer to Figure 2 As shown, the S6 includes:

[0103] S61, normalizing the character attention vector to obtain a character normalized vector;

[0104] S62, performing a convolution operation on the normalized vector to obtain a character convolution vector;

[0105] S63. Perform a residual connection on the character convolution vector and the character attention vector to obtain the character feature sequence.

[0106] The normalization is as described above, and is an operation of mapping the numerical value in the character attention vector to a specified range. In an embodiment of the present invention, the numerical value in the character attention vector is mapped to a range of [0, 1].

[0107] In detail, performing a convolution operation on the normalized vector to obtain a character convolution vector includes: constructing a convolution kernel according to a preset convolution kernel dimension; and performing a convolution operation on the normalized vector using the convolution kernel to obtain the character convolution vector.

[0108] Furthermore, the residual connection is the same as above, and the character convolution vector and the character attention vector are added correspondingly to obtain the character feature sequence.

[0109] S7. Input the character vector into a pre-built pronunciation pause prediction model to obtain a pronunciation pause sequence.

[0110] In detail, the pronunciation pause prediction model is constructed based on multiple fast Fourier transform modules. In the embodiment of the present invention, 10 fast Fourier transform modules are used to construct the pronunciation pause prediction model.

[0111] In detail, S7 includes: converting the character pinyin into a word vector to obtain a pinyin vector; inputting the pinyin vector and the character vector into the pronunciation pause prediction model, and using the pronunciation pause prediction model to perform Fourier transform on the pinyin vector and the character vector to obtain a Fourier transform sequence; performing pronunciation pause prediction on the Fourier transform sequence to obtain the pronunciation pause sequence.

[0112] The fast Fourier transform is a fast algorithm of discrete Fourier transform (DFT), which can predict the Fourier transform sequence corresponding to the character vector and the pinyin vector, wherein the Fourier transform sequence includes speech frequency, amplitude and phase, and the pronunciation pause sequence can be obtained through the Fourier transform sequence.

[0113] S8. Perform residual connection on the character feature sequence and the pronunciation pause sequence to obtain a speech sequence, and use a pre-built vocoder synthesizer to perform speech synthesis on the speech sequence to obtain synthesized speech of the character text.

[0114] In the embodiment of the present invention, the vocoder is a decoder capable of realizing speech synthesis, including a channel vocoder, a formant vocoder, a pattern vocoder, a linear prediction vocoder, a correlation vocoder, an orthogonal function vocoder, etc. In the embodiment of the present invention, the synthesized speech of the character text can be obtained by inputting the speech sequence into the vocoder.

[0115] The embodiment of the present invention divides speech synthesis into two parts. First, a pre-trained attention feature model is used to perform feature extraction on the character text to obtain a character feature sequence. Secondly, a pronunciation pause prediction model is used to predict the pronunciation pause sequence of the character text. Finally, the character feature sequence and the pronunciation pause sequence are residually connected to obtain a speech sequence. A pre-built voice code synthesizer is used to perform speech synthesis on the speech sequence to obtain the synthesized speech of the character text. Compared with the background technology that simply uses LSTM, BERT and other models for synthesis, the present invention not only predicts the character feature sequence, but also adds a prediction process for the pronunciation pause sequence. Therefore, the synthesized speech is closer to natural human voice in terms of frequency amplitude, etc. Therefore, the speech synthesis method, device and computer-readable storage medium proposed in the present invention can solve the problem that the synthesized speech is not smooth and natural enough.

[0116] like Figure 3 FIG. 1 is a schematic diagram of a module of a speech synthesis device according to the present invention.

[0117] The speech synthesis device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the speech synthesis device may include a character vector construction module 101, a character feature sequence extraction module 102, a pronunciation pause sequence extraction module 103 and a speech synthesis module 104. The module described in the present invention may also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0118] In this embodiment, the functions of each module / unit are as follows:

[0119] The character vector construction module 101 is used to receive a character text, perform pinyin replacement on the character text to obtain a character pinyin, calculate the character position of the character pinyin in the alphabet using a pre-constructed alphabet, perform encoding operation on the character position and the character pinyin to obtain a character vector;

[0120] The character feature sequence extraction module 102 is used to input the character vector into a pre-trained attention feature model, wherein the attention feature model includes a multi-head attention network and a character feature extraction network, use the multi-head attention network to perform attention calculation on the character vector to obtain an attention vector, perform residual connection on the attention vector and the character vector to obtain a character attention vector, and use the character feature extraction network to perform feature extraction on the character attention vector to obtain a character feature sequence;

[0121] The pronunciation pause sequence extraction module 103 is used to input the character vector into a pre-built pronunciation pause prediction model to obtain a pronunciation pause sequence;

[0122] The speech synthesis module 104 is used to perform residual connection on the character feature sequence and the pronunciation pause sequence to obtain a speech sequence, and use a pre-built vocoder synthesizer to perform speech synthesis on the speech sequence to obtain the synthesized speech of the character text.

[0123] Each module in the speech synthesis device 100 provided in the embodiment of the present invention can be used based on the same means as the above-mentioned speech synthesis method. The specific implementation steps will not be repeated here. The technical effects produced by the functions of each module / unit are the same as the technical effects of the above-mentioned speech synthesis method, that is, the problem that the synthesized speech is not smooth and natural enough is solved.

[0124] like Figure 4 FIG. 1 is a schematic diagram of the structure of an electronic device for implementing the speech synthesis method of the present invention.

[0125] The electronic device 1 may include a processor 10 , a memory 11 , and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 10 , such as a speech synthesis program 12 .

[0126] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. The memory 11 may be an internal storage unit of the electronic device 1 in some embodiments, such as a mobile hard disk of the electronic device 1. The memory 11 may also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 11 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 may not only be used to store application software and various types of data installed in the electronic device 1, such as the code of the speech synthesis program 12, etc., but may also be used to temporarily store data that has been output or is to be output.

[0127] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules (such as executing speech synthesis programs, etc.) stored in the memory 11, and calls data stored in the memory 11 to execute various functions of the electronic device 1 and process data.

[0128] The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0129] Figure 4 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 4 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0130] For example, although not shown, the electronic device 1 may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0131] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0132] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0133] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0134] The speech synthesis program 12 stored in the memory 11 of the electronic device 1 is a combination of multiple instructions, and when running in the processor 10, it can achieve:

[0135] Receiving a character text, performing pinyin replacement on the character text to obtain a character pinyin, and using a pre-constructed alphabet table to calculate a character position of the character pinyin in the alphabet table;

[0136] Performing encoding operation on the character position and the character pinyin to obtain a character vector;

[0137] Inputting the character vector into a pre-trained attention feature model, wherein the attention feature model includes a multi-head attention network and a character feature extraction network;

[0138] Using the multi-head attention network to perform attention calculation on the character vector to obtain an attention vector;

[0139] Performing a residual connection on the attention vector and the character vector to obtain a character attention vector;

[0140] Using the character feature extraction network, perform feature extraction on the character attention vector to obtain a character feature sequence;

[0141] Inputting the character vector into a pre-built pronunciation pause prediction model to obtain a pronunciation pause sequence;

[0142] The character feature sequence and the pronunciation pause sequence are subjected to residual connection to obtain a speech sequence, and a pre-built vocoder is used to perform speech synthesis on the speech sequence to obtain synthesized speech of the character text.

[0143] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0144] Furthermore, the computer-usable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0145] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:

[0146] Receiving a character text, performing pinyin replacement on the character text to obtain a character pinyin, and using a pre-constructed alphabet table to calculate a character position of the character pinyin in the alphabet table;

[0147] Performing encoding operation on the character position and the character pinyin to obtain a character vector;

[0148] Inputting the character vector into a pre-trained attention feature model, wherein the attention feature model includes a multi-head attention network and a character feature extraction network;

[0149] Using the multi-head attention network to perform attention calculation on the character vector to obtain an attention vector;

[0150] Performing a residual connection on the attention vector and the character vector to obtain a character attention vector;

[0151] Using the character feature extraction network, perform feature extraction on the character attention vector to obtain a character feature sequence;

[0152] Inputting the character vector into a pre-built pronunciation pause prediction model to obtain a pronunciation pause sequence;

[0153] The character feature sequence and the pronunciation pause sequence are subjected to residual connection to obtain a speech sequence, and a pre-built vocoder is used to perform speech synthesis on the speech sequence to obtain synthesized speech of the character text.

[0154] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0155] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0157] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0158] Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any associated diagram mark in the claims should not be regarded as limiting the claims involved.

[0159] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0160] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Receiving a character text, performing pinyin replacement on the character text to obtain a character pinyin, and using a pre-constructed alphabet table to calculate a character position of the character pinyin in the alphabet table; Performing encoding operation on the character position and the character pinyin to obtain a character vector; Inputting the character vector into a pre-trained attention feature model, wherein the attention feature model includes a multi-head attention network and a character feature extraction network; Using the multi-head attention network to perform attention calculation on the character vector to obtain an attention vector; Performing a residual connection on the attention vector and the character vector to obtain a character attention vector; Using the character feature extraction network, perform feature extraction on the character attention vector to obtain a character feature sequence of the character text; Inputting the character vector into a pre-built pronunciation pause prediction model to obtain a pronunciation pause sequence of the character text; Performing residual connection on the character feature sequence and the pronunciation pause sequence to obtain a speech sequence, and performing speech synthesis on the speech sequence using a pre-built vocoder synthesizer to obtain synthesized speech of the character text; The step of inputting the character vector into a pre-built pronunciation pause prediction model to obtain a pronunciation pause sequence of the character text includes: Convert the pinyin of the character into a word vector to obtain a pinyin vector; Input the pinyin vector and the character vector into the pronunciation pause prediction model, the pronunciation pause prediction model is obtained by training based on the Fourier transform algorithm, and use the pronunciation pause prediction model to perform Fourier transform on the pinyin vector and the character vector to obtain the Fourier transform sequence corresponding to the pinyin vector and the character vector, wherein the Fourier transform sequence includes speech frequency, amplitude and phase; Pronunciation pause prediction is performed on the Fourier transform sequence to obtain a pronunciation pause sequence of the character text.

2. The speech synthesis method according to claim 1, characterized in that: The step of using the character feature extraction network to perform feature extraction on the character attention vector to obtain a character feature sequence includes: Normalizing the character attention vector to obtain a character normalized vector; Performing a convolution operation on the normalized vector to obtain a character convolution vector; Perform a residual connection on the character convolution vector and the character attention vector to obtain the character feature sequence.

3. The speech synthesis method according to claim 2, characterized in that: The performing a convolution operation on the normalized vector to obtain a character convolution vector includes: Construct a convolution kernel according to the preset convolution kernel dimension; The convolution kernel is used to perform a convolution operation on the normalized vector to obtain the character convolution vector.

4. The speech synthesis method according to claim 1, wherein: The pre-trained attention feature model includes: Step A: constructing an attention feature model to be trained including the multi-head attention network and the character feature extraction network; Step B: receiving a training text set and a training label set, inputting the training text set into the attention feature model to be trained for feature extraction, and obtaining a feature sequence training set; Step C: Construct multi-layer linear activation layers; Step D: using the multi-layer linear activation layer, performing an activation operation on the feature sequence training set to obtain a prediction sequence set; Step E: Calculate the error value between the prediction sequence set and the training label set, and determine the magnitude relationship between the error value and a preset error threshold; Step F: If the error value is greater than the error threshold, adjust the internal parameters of the attention feature model to be trained, and return to step B; Step G: If the error value is less than or equal to the error threshold, the attention feature model is obtained.

5. The speech synthesis method according to claim 4, characterized in that: The step of inputting the training text set into the attention feature model to be trained for feature extraction to obtain a feature sequence training set includes: Performing pinyin replacement on the training text set to obtain a pinyin training set; Calculate the character positions of the pinyin training set in the alphabet to obtain a position training set; Performing encoding operation on the pinyin training set and the position training set to obtain a vector training set; Using the multi-head attention network to perform attention calculation on the vector training set to obtain an attention vector set; Performing residual connection on the attention vector set and the vector training set to obtain an attention vector training set; The character feature extraction network is used to perform feature extraction on the attention vector training set to obtain the feature sequence training set.

6. The speech synthesis method according to claim 4 or 5, characterized in that: The method of using the multi-layer linear activation layer to perform an activation operation on the feature sequence training set to obtain a prediction sequence set includes: Normalizing the feature sequence training set to obtain a feature sequence normalized set; The Gaussian distribution of the normalized set of feature sequences is calculated, and the predicted sequence set is calculated based on the Gaussian distribution.

7. A speech synthesis device, used to implement the speech synthesis method according to any one of claims 1 to 6, characterized in that: The device comprises: A character vector construction module is used to receive a character text, perform a pinyin replacement on the character text to obtain a character pinyin, calculate a character position of the character pinyin in the alphabet using a pre-constructed alphabet, perform an encoding operation on the character position and the character pinyin to obtain a character vector; A character feature sequence extraction module is used to input the character vector into a pre-trained attention feature model, wherein the attention feature model includes a multi-head attention network and a character feature extraction network, use the multi-head attention network to perform attention calculation on the character vector to obtain an attention vector, perform residual connection on the attention vector and the character vector to obtain a character attention vector, and use the character feature extraction network to perform feature extraction on the character attention vector to obtain a character feature sequence; A pronunciation pause sequence extraction module, used for inputting the character vector into a pre-built pronunciation pause prediction model to obtain a pronunciation pause sequence; The speech synthesis module is used to perform residual connection on the character feature sequence and the pronunciation pause sequence to obtain a speech sequence, and use a pre-built vocoder synthesizer to perform speech synthesis on the speech sequence to obtain the synthesized speech of the character text.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the speech synthesis method according to any one of claims 1 to 6.

9. A computer-readable storage medium, comprising a data storage area and a program storage area, characterized in that: The data storage area stores created data, and the program storage area stores a computer program; wherein the computer program, when executed by a processor, implements the speech synthesis method according to any one of claims 1 to 6.

Images