Speech synthesis method, device, electronic device and readable storage medium

Through the methods of speech feature extraction and text vector splicing, the problem of poor flexibility in speech synthesis in the prior art is solved, and multi-style speech synthesis is realized, and the voice of different accents and languages ​​can be generated more flexibly.

CN112397047BActive Publication Date: 2024-11-19PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011442571.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-11
Publication Date
2024-11-19
Estimated Expiration
2040-12-11

AI Technical Summary

Technical Problem

The existing speech synthesis technology cannot meet the needs of multi-style speech synthesis, and has poor flexibility, so it is impossible to synthesize Chinese text into pronunciations of different accents or languages.

Method used

By obtaining sample audio for sound feature extraction and vectorization, standard speech vectors are obtained; phoneme conversion and vector conversion are performed for synthetic text, vector splicing is combined with speech vectors and text matrix, spectrum feature extraction and speech synthesis are performed, and preset sounds are used to use preset sounds The coder generates synthetic audio.

Benefits of technology

It improves the flexibility of pronunciation, can synthesize pronunciations of different styles and languages ​​more flexibly, eliminates the differences in pronunciation of different texts, and realizes the flexible combination of pronunciation and text characteristics.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112397047B_ABST
    Figure CN112397047B_ABST
Patent Text Reader

Abstract

The present invention relates to speech synthesis technology, and discloses a speech synthesis method, comprising: obtaining sample audio, performing sound feature extraction conversion and vectorization processing on the sample audio to obtain a standard speech vector; when receiving a text to be synthesized, performing phoneme conversion on the text to be synthesized to obtain a text phoneme sequence; performing vector conversion on the text phoneme sequence to obtain a text matrix; performing vector splicing on the standard speech vector and the text matrix to obtain a target matrix; performing spectrum feature extraction on the target matrix to obtain spectrum feature information; performing speech synthesis on the spectrum feature information using a preset vocoder to obtain synthesized audio. The present invention also relates to a blockchain technology, and the spectrum feature information can be stored in a blockchain. The present invention also proposes a speech synthesis device, an electronic device, and a readable storage medium. The present invention can improve the flexibility of speech synthesis.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis, and in particular to a speech synthesis method, device, electronic equipment and readable storage medium. Background Art

[0002] With the development of artificial intelligence, speech synthesis, as an important part of artificial intelligence, can convert any text information into standard and fluent speech in real time, which is equivalent to installing an artificial mouth on the machine. Therefore, speech synthesis technology has received more and more attention.

[0003] However, currently, speech synthesis methods can only synthesize text into speech of a certain style or language, such as only synthesizing Chinese text into Mandarin with a Beijing accent, but not a Sichuan accent or a Japanese accent; they cannot meet people's demand for multiple styles of speech synthesis, and the flexibility of speech synthesis is poor. Summary of the invention

[0004] The present invention provides a speech synthesis method, device, electronic equipment and computer-readable storage medium, the main purpose of which is to improve the flexibility of speech synthesis.

[0005] To achieve the above object, the present invention provides a speech synthesis method, comprising:

[0006] Acquire sample audio, perform sound feature extraction, conversion and vectorization processing on the sample audio to obtain a standard speech vector;

[0007] When receiving a text to be synthesized, performing phoneme conversion on the text to be synthesized to obtain a text phoneme sequence;

[0008] Performing vector conversion on the text phoneme sequence to obtain a text matrix;

[0009] Performing vector concatenation of the standard speech vector and the text matrix to obtain a target matrix;

[0010] Extracting spectrum features from the target matrix to obtain spectrum feature information;

[0011] The frequency spectrum feature information is subjected to speech synthesis using a preset vocoder to obtain synthesized audio.

[0012] Optionally, the performing sound feature extraction, conversion and vectorization processing on the sample audio to obtain a standard speech vector includes:

[0013] Performing sound feature extraction and conversion on the sample audio to obtain a target spectrogram;

[0014] The pre-built picture classification model is used to extract features from the target spectrogram to obtain the standard speech vector.

[0015] Optionally, performing sound feature extraction and conversion on the sample audio to obtain a target spectrogram includes:

[0016] Resampling the sample audio to obtain a digital voice signal;

[0017] Pre-emphasize the digital voice signal to obtain a standard digital voice signal;

[0018] The standard digital speech signal is subjected to feature conversion to obtain the target spectrogram.

[0019] Optionally, the using a pre-built picture classification model to extract features from the target spectrogram to obtain the standard speech vector includes:

[0020] Obtaining outputs of all nodes of the fully connected layer included in the image classification model to obtain a target spectrogram feature value set;

[0021] According to the order of all nodes of the fully connected layer, the feature values ​​in the target spectrogram feature value set are combined vertically to obtain a standard speech vector.

[0022] Optionally, performing feature conversion on the standard digital speech signal to obtain the target spectrogram includes:

[0023] The standard digital speech signal is mapped in the frequency domain using a preset sound processing algorithm to obtain the target spectrogram.

[0024] Optionally, the step of performing vector concatenation of the standard speech vector and the text matrix to obtain a target matrix includes:

[0025] Calculating the phoneme frame length of each phoneme in the text phoneme sequence using a preset algorithm model to obtain a phoneme frame length sequence;

[0026] Converting the phoneme frame length sequence into a phoneme frame length vector;

[0027] The phoneme frame length vector and the text matrix are horizontally spliced ​​to obtain a standard text matrix;

[0028] The standard speech vector is vertically concatenated with each column of the standard text matrix to obtain the target matrix.

[0029] Optionally, performing phoneme conversion on the text to be synthesized to obtain a text phoneme sequence includes:

[0030] Deleting punctuation marks from the text to be synthesized to obtain a standard text;

[0031] The phoneme corresponding to each character in the standard text is marked using a preset phonetic symbol rule to obtain the text phoneme sequence.

[0032] In order to solve the above problem, the present invention further provides a speech synthesis device, the device comprising:

[0033] An audio processing module is used to obtain sample audio, perform sound feature extraction, conversion and vectorization processing on the sample audio to obtain a standard speech vector;

[0034] The text processing module is used for, when receiving the text to be synthesized, performing phoneme conversion on the text to be synthesized to obtain a text phoneme sequence; performing vector conversion on the text phoneme sequence to obtain a text matrix; performing vector concatenation on the standard speech vector and the text matrix to obtain a target matrix;

[0035] The speech synthesis module is used to extract the spectrum feature of the target matrix to obtain spectrum feature information; and use a preset vocoder to perform speech synthesis on the spectrum feature information to obtain synthesized audio.

[0036] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0037] a memory storing at least one computer program; and

[0038] The processor executes the computer program stored in the memory to implement the above-mentioned speech synthesis method.

[0039] In order to solve the above problem, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned speech synthesis method.

[0040] The embodiment of the present invention performs sound feature extraction, conversion and vectorization processing on the sample audio to obtain a standard speech vector; when receiving a text to be synthesized, the text to be synthesized is converted into a text phoneme sequence to eliminate the differences in pronunciation of different types of text, making speech synthesis more flexible; the text phoneme sequence is vectorized to obtain a text matrix; the standard speech vector and the text matrix are vectorized to obtain a target matrix, thereby achieving a flexible combination of the features of the speech and the features of the text to be synthesized, and ensuring that subsequent speech can be flexibly synthesized; the target matrix is ​​subjected to spectral feature extraction to obtain spectral feature information; the spectral feature information is subjected to speech synthesis using a preset vocoder to obtain synthesized audio. Therefore, the speech synthesis method, device, electronic device and computer-readable storage medium proposed in the embodiment of the present invention improve the flexibility of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of a speech synthesis method provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of a process for obtaining a target spectrogram in a speech synthesis method provided in an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of a flow chart of obtaining a standard speech vector in a speech synthesis method provided in an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of a module of a speech synthesis device provided by an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of the internal structure of an electronic device for implementing a speech synthesis method provided by an embodiment of the present invention;

[0046] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0048] The embodiment of the present invention provides a speech synthesis method. The execution subject of the speech synthesis method includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the speech synthesis method can be executed by software or hardware installed in a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0049] Reference Figure 1 FIG. 1 is a flow chart of a speech synthesis method provided by an embodiment of the present invention. In the embodiment of the present invention, the speech synthesis method includes:

[0050] S1. Obtain sample audio, perform sound feature extraction, conversion and vectorization processing on the sample audio to obtain a standard speech vector;

[0051] In the embodiment of the present invention, the sample audio is the speech data of the target speaker to be synthesized subsequently. For example, if the subsequent text is synthesized into the speech of speaker A, the sample audio is the speech data of speaker A.

[0052] Furthermore, in order to make the subsequent speech synthesis of the text more accurate, the present invention performs feature extraction processing on the sample audio to obtain the standard speech vector.

[0053] Since the voice data volume is large and difficult to process, the sample audio is subjected to sound feature extraction and conversion to obtain a spectrogram.

[0054] In detail, in the embodiments of the present invention, refer to Figure 2 As shown, the sound feature extraction and conversion of the sample audio to obtain a target spectrogram includes:

[0055] S11, resampling the sample audio to obtain a digital voice signal;

[0056] In the embodiment of the present invention, in order to facilitate data processing of the sample audio, the sample audio is resampled to obtain the digital voice signal. Preferably, the embodiment of the present invention uses a digital-to-analog converter to resample the sample audio.

[0057] S12, pre-emphasize the digital voice signal to obtain a standard digital voice signal;

[0058] In detail, the embodiment of the present invention uses the following formula to perform the pre-emphasis operation:

[0059] y(t)=x(t)-μx(t-1)

[0060] Wherein, x(t) is the digital voice signal, t is time, y(t) is the standard digital voice signal, μ is the preset adjustment value of the pre-emphasis operation, and preferably, the value range of μ is [0.9, 1.0].

[0061] S13, performing feature conversion on the standard digital speech signal to obtain the target spectrogram;

[0062] In the embodiment of the present invention, the standard digital voice signal can only reflect the changes of audio in the time domain, but cannot reflect the audio characteristics of the standard voice signal. In order to reflect the audio characteristics of the standard voice signal and make the audio characteristics more intuitive and clear, the standard digital voice signal is subjected to feature conversion.

[0063] In detail, in the embodiment of the present invention, the feature conversion of the standard digital voice signal includes: using a preset sound processing algorithm to map the standard digital voice signal in the frequency domain to obtain the target spectrogram. Preferably, the sound processing algorithm in the embodiment of the present invention is a Mel filter algorithm.

[0064] Furthermore, in order to further simplify the use of data and improve the processing efficiency of data, the embodiment of the present invention performs vectorization processing on the target spectrogram, including: using a pre-constructed image classification model to extract features of the target spectrogram to obtain the standard speech vector. Preferably, in the embodiment of the present invention, the pre-constructed image classification model is a residual network model trained using a historical spectrogram set, wherein the historical spectrogram set is a set of multiple spectrograms of the same type as the target spectrogram but with different contents.

[0065] In detail, in the embodiments of the present invention, refer to Figure 3 As shown, the use of a pre-built picture classification model to extract features from the target spectrogram to obtain the standard speech vector includes:

[0066] S21, obtaining the outputs of all nodes of the fully connected layer included in the image classification model to obtain a target spectrogram feature value set;

[0067] For example, the fully connected layer included in the image classification model has a total of 1000 nodes. The target spectrogram T is input into the image classification model, and 1000 node output values ​​are obtained to obtain a target spectrogram feature value set of the target spectrogram T, wherein the output of each node is a feature value of the target spectrogram T, so the target spectrogram feature value set of the target spectrogram T has a total of 1000 feature values.

[0068] S22, according to the order of all nodes of the fully connected layer, vertically combining the feature values ​​in the target spectrogram feature value set to obtain a standard speech vector;

[0069] For example, the fully connected layer has 3 nodes, which are the first node, the second node, and the third node in order. The target spectrogram feature value set of the target spectrogram A has 3 eigenvalues, 3, 5, and 1, where eigenvalue 1 is the output of the first node, eigenvalue 3 is the output of the second node, and eigenvalue 5 is the output of the third node. The three eigenvalues ​​in the target spectrogram feature value set of the target spectrogram A are combined vertically in the order of the nodes to obtain the standard speech vector of the target spectrogram A.

[0070] S2. When receiving a text to be synthesized, performing phoneme conversion on the text to be synthesized to obtain a text phoneme sequence;

[0071] In an embodiment of the present invention, the text to be synthesized is a text that needs to be synthesized into speech, and the phonemes of the pronunciation of texts with different speech can be represented by common phonetic symbol rules. Furthermore, in order to eliminate the differences between texts in different languages, the embodiment of the present invention performs phoneme conversion on the text to be synthesized to obtain a text phoneme sequence.

[0072] Specifically, in the embodiments of the present invention, the phoneme conversion of the text to be synthesized to obtain the text phoneme sequence includes: deleting punctuation marks from the text to be synthesized to obtain a standard text; using a preset phonetic symbol rule to mark the phoneme corresponding to each character in the standard text to obtain the text phoneme sequence. For example, if the preset phonetic symbol rule is the International Phonetic Alphabet rule, the phoneme corresponding to the character "啊" is marked as a, and the obtained text phoneme sequence is [a].

[0073] S3. Perform vector conversion on the text phoneme sequence to obtain a text matrix;

[0074] In the embodiments of the present invention, each phoneme in the text phoneme sequence is converted into a column vector by using the one-hot encoding algorithm to obtain the text matrix.

[0075] S4. Perform vector splicing on the standard speech vector and the text matrix to obtain a target matrix;

[0076] Specifically, in the embodiments of the present invention, in order to perform better speech synthesis subsequently, it is also necessary to determine the speech alignment of each phoneme in the text phoneme sequence, that is, to determine the pronunciation duration of each phoneme in the text phoneme sequence, namely the phoneme frame length. Therefore, in the embodiments of the present invention, a preset algorithm model is used to calculate the phoneme frame length of each phoneme in the text phoneme sequence to obtain a phoneme frame length sequence. The preset algorithm model in the embodiments of the present invention may be a DNN-HMM network model.

[0077] Further, in the embodiments of the present invention, the phoneme frame length sequence is converted into a phoneme frame length vector, that is, the phoneme frame length sequence is converted into a corresponding row vector to obtain the phoneme frame length vector. The phoneme frame length vector and the text matrix are horizontally spliced to obtain the standard text matrix. For example, the phoneme frame length vector is a 1×4 row vector, and the text matrix is a 5×4 matrix. The phoneme frame length vector is used as the fifth row of the text matrix to obtain the 6×4 standard text matrix.

[0078] Specifically, in the embodiments of the present invention, the standard speech vector is vertically spliced with each column of the standard text matrix to obtain the target matrix. For example, the standard text matrix is The standard speech vector is Vertically splicing the standard speech vector with each column of the standard text matrix to obtain the target matrix as

[0079] S5. Extract spectral feature information from the target matrix;

[0080] In order to further perform speech synthesis, the embodiment of the present invention also needs to determine the frequency spectrum feature of the target matrix, wherein the frequency spectrum feature may be a Mel spectrum.

[0081] In detail, in the embodiment of the present invention, the trained acoustic model is used to extract the spectrum feature of the target matrix to obtain the spectrum feature extraction. Preferably, the acoustic model can be a transformer model.

[0082] Furthermore, in the embodiment of the present invention, before using the trained acoustic model to extract the spectral features of the target matrix, the method further includes: obtaining a historical text matrix set; marking the spectral feature information of each historical text matrix in the historical text matrix set to obtain a training set; and training the acoustic model using the training set until the acoustic model converges to obtain the trained acoustic model. The historical text matrix set is a collection of multiple historical text matrices, and the historical text matrix is ​​a target matrix corresponding to a text different from the text to be synthesized.

[0083] In another embodiment of the present invention, in order to ensure the privacy of data, the spectrum characteristic information can be stored in a blockchain node.

[0084] S6. Perform speech synthesis on the frequency spectrum feature information using a preset vocoder to obtain synthesized audio.

[0085] In detail, in the embodiment of the present invention, the frequency spectrum feature information is input into a preset vocoder to obtain the synthesized audio.

[0086] Preferably, the vocoder is a WORLD vocoder.

[0087] like Figure 4 The figure shows a functional module diagram of the speech synthesis device of the present invention.

[0088] The speech synthesis device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the speech synthesis device can include an audio processing module 101, a text processing module 102, and a speech synthesis module 103. The modules described in the present invention can also be referred to as units, which refer to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, and are stored in a memory of the electronic device.

[0089] In this embodiment, the functions of each module / unit are as follows:

[0090] The audio processing module 101 is used to obtain sample audio, perform sound feature extraction, conversion and vectorization processing on the sample audio, and obtain a standard speech vector.

[0091] In the embodiment of the present invention, the sample audio is the speech data of the target speaker to be synthesized subsequently. For example, if the subsequent text is synthesized into the speech of speaker A, the sample audio is the speech data of speaker A.

[0092] Furthermore, in order to make the speech synthesis of subsequent text more accurate, the audio processing module 101 performs feature extraction processing on the sample audio to obtain the standard speech vector.

[0093] Since the speech data volume is large and difficult to process, the audio processing module 101 performs sound feature extraction and conversion on the sample audio to obtain a target spectrogram.

[0094] In detail, in the embodiment of the present invention, the audio processing module 101 uses the following means to extract and convert the sound features of the sample audio to obtain a target spectrogram, including:

[0095] Resampling the sample audio to obtain a digital voice signal;

[0096] In the embodiment of the present invention, in order to facilitate data processing of the sample audio, the sample audio is resampled to obtain the digital voice signal. Preferably, the embodiment of the present invention uses a digital-to-analog converter to resample the sample audio.

[0097] Pre-emphasize the digital voice signal to obtain a standard digital voice signal;

[0098] In detail, the embodiment of the present invention uses the following formula to perform the pre-emphasis operation:

[0099] y(t)=x(t)-μx(t-1)

[0100] Wherein, x(t) is the digital voice signal, t is time, y(t) is the standard digital voice signal, μ is the preset adjustment value of the pre-emphasis operation, and preferably, the value range of μ is [0.9, 1.0].

[0101] Performing feature conversion on the standard digital speech signal to obtain the target spectrogram;

[0102] In the embodiment of the present invention, the standard digital voice signal can only reflect the changes of audio in the time domain, but cannot reflect the audio characteristics of the standard voice signal. In order to reflect the audio characteristics of the standard voice signal and make the audio characteristics more intuitive and clear, the standard digital voice signal is subjected to feature conversion.

[0103] In detail, in the embodiment of the present invention, the audio processing module 101 performs feature conversion on the standard digital voice signal, including: using a preset sound processing algorithm to map the standard digital voice signal in the frequency domain to obtain the target spectrogram. Preferably, the sound processing algorithm in the embodiment of the present invention is a Mel filter algorithm.

[0104] Furthermore, in order to further simplify the use of data and improve the processing efficiency of data, the audio processing module 101 of the embodiment of the present invention performs vectorization processing on the target spectrogram, including: using a pre-constructed image classification model to extract features of the target spectrogram to obtain the standard speech vector. Preferably, in the embodiment of the present invention, the pre-constructed image classification model is a residual network model trained using a historical spectrogram set, wherein the historical spectrogram set is a set of multiple spectrograms of the same type as the target spectrogram but with different contents.

[0105] In detail, in the embodiment of the present invention, the audio processing module 101 uses the following means to extract features from the target spectrogram to obtain the standard speech vector, including:

[0106] Obtaining outputs of all nodes of the fully connected layer included in the image classification model to obtain a target spectrogram feature value set;

[0107] For example, the fully connected layer included in the image classification model has a total of 1000 nodes. The target spectrogram T is input into the image classification model, and 1000 node output values ​​are obtained to obtain a target spectrogram feature value set of the target spectrogram T, wherein the output of each node is a feature value of the target spectrogram T, so the target spectrogram feature value set of the target spectrogram T has a total of 1000 feature values.

[0108] According to the order of all nodes of the fully connected layer, the feature values ​​in the target spectrogram feature value set are combined vertically to obtain a standard speech vector;

[0109] For example, the fully connected layer has 3 nodes, which are the first node, the second node, and the third node in order. The target spectrogram feature value set of the target spectrogram A has 3 eigenvalues, 3, 5, and 1, where eigenvalue 1 is the output of the first node, eigenvalue 3 is the output of the second node, and eigenvalue 5 is the output of the third node. The three eigenvalues ​​in the target spectrogram feature value set of the target spectrogram A are combined vertically in the order of the nodes to obtain the standard speech vector of the target spectrogram A.

[0110] The text processing module 102 is configured to, when receiving the text to be synthesized, perform phoneme conversion on the text to be synthesized to obtain a text phoneme sequence; perform vector conversion on the text phoneme sequence to obtain a text matrix; and splice the standard speech vector and the text matrix to obtain a target matrix.

[0111] In an embodiment of the present invention, the text to be synthesized is the text for which speech needs to be synthesized. The phonemes of the pronunciations of texts in different languages can be represented by common phonetic symbol rules. Further, in order to eliminate the differences between texts in different languages in an embodiment of the present invention, phoneme conversion is performed on the text to be synthesized to obtain a text phoneme sequence.

[0112] Specifically, in an embodiment of the present invention, the text processing module 102 performs phoneme conversion on the text to be synthesized to obtain the text phoneme sequence, including: deleting punctuation marks from the text to be synthesized to obtain a standard text; marking the phoneme corresponding to each character in the standard text using a preset phonetic symbol rule to obtain the text phoneme sequence. For example, if the preset phonetic symbol rule is the International Phonetic Alphabet rule, the phoneme corresponding to the character "啊" is marked as a, and the obtained text phoneme sequence is [a].

[0113] In an embodiment of the present invention, the text processing module 102 converts each phoneme in the text phoneme sequence into a column vector using the onehot encoding algorithm to obtain the text matrix.

[0114] Specifically, in an embodiment of the present invention, in order to perform better speech synthesis subsequently, it is also necessary to determine the speech alignment of each phoneme in the text phoneme sequence, that is, to determine the pronunciation duration of each phoneme in the text phoneme sequence, namely the phoneme frame length. Therefore, in an embodiment of the present invention, the text processing module 102 calculates the phoneme frame length of each phoneme in the text phoneme sequence using a preset algorithm model to obtain a phoneme frame length sequence. The preset algorithm model in an embodiment of the present invention may be a DNN-HMM network model.

[0115] Further, in an embodiment of the present invention, the text processing module 102 converts the phoneme frame length sequence into a phoneme frame length vector, that is, converts the phoneme frame length sequence into a corresponding row vector to obtain the phoneme frame length vector, and horizontally splices the phoneme frame length vector and the text matrix to obtain the standard text matrix. For example, the phoneme frame length vector is a 1*4 row vector, the text matrix is a 5*4 matrix, and the phoneme frame length vector is used as the fifth row of the text matrix to obtain the 6*4 standard text matrix.

[0116] In detail, in the embodiment of the present invention, the text processing module 102 vertically splices the standard speech vector with each column of the standard text matrix to obtain the target matrix. For example, the standard text matrix is The standard speech vector is The standard speech vector is vertically concatenated with each column of the standard text matrix to obtain the target matrix:

[0117] The speech synthesis module 103 is used to extract spectrum features from the target matrix to obtain spectrum feature information; and perform speech synthesis on the spectrum feature information using a preset vocoder to obtain synthesized audio.

[0118] In order to further perform speech synthesis, the embodiment of the present invention also needs to determine the frequency spectrum feature of the target matrix, wherein the frequency spectrum feature may be a Mel spectrum.

[0119] In detail, in the embodiment of the present invention, the trained acoustic model is used to extract the spectrum feature of the target matrix to obtain the spectrum feature extraction. Preferably, the acoustic model can be a transformer model.

[0120] Furthermore, before the speech synthesis module 103 in the embodiment of the present invention uses the trained acoustic model to extract the spectral features of the target matrix, it also includes: obtaining a historical text matrix set; marking the spectral feature information of each historical text matrix in the historical text matrix set to obtain a training set; and training the acoustic model using the training set until the acoustic model converges to obtain the trained acoustic model. The historical text matrix set is a collection of multiple historical text matrices, and the historical text matrix is ​​a target matrix corresponding to a text different from the text to be synthesized.

[0121] In another embodiment of the present invention, in order to ensure the privacy of data, the spectrum characteristic information can be stored in a blockchain node.

[0122] In detail, the speech synthesis module 103 in the embodiment of the present invention inputs the frequency spectrum feature information into a preset vocoder to obtain the synthesized audio.

[0123] Preferably, the vocoder is a WORLD vocoder.

[0124] like Figure 5 FIG. 1 is a schematic diagram of the structure of an electronic device for implementing the speech synthesis method of the present invention.

[0125] The electronic device 1 may include a processor 10 , a memory 11 , and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 10 , such as a speech synthesis program 12 .

[0126] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. The memory 11 may be an internal storage unit of the electronic device 1 in some embodiments, such as a mobile hard disk of the electronic device 1. The memory 11 may also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 11 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 may be used not only to store application software and various types of data installed in the electronic device 1, such as the code of a speech synthesis program, etc., but also to temporarily store data that has been output or is to be output.

[0127] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules (such as speech synthesis programs, etc.) stored in the memory 11, and calls data stored in the memory 11, so as to execute various functions of the electronic device 1 and process data.

[0128] The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0129] Figure 5Only an electronic device with components is shown, and those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0130] For example, although not shown, the electronic device 1 may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0131] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0132] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0133] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0134] The speech synthesis program 12 stored in the memory 11 in the electronic device 1 is a combination of multiple computer programs. When running in the processor 10, it can achieve:

[0135] Acquire sample audio, perform sound feature extraction, conversion and vectorization processing on the sample audio to obtain a standard speech vector;

[0136] When receiving a text to be synthesized, performing phoneme conversion on the text to be synthesized to obtain a text phoneme sequence;

[0137] Performing vector conversion on the text phoneme sequence to obtain a text matrix;

[0138] Performing vector concatenation of the standard speech vector and the text matrix to obtain a target matrix;

[0139] Extracting spectrum features from the target matrix to obtain spectrum feature information;

[0140] The frequency spectrum feature information is subjected to speech synthesis using a preset vocoder to obtain synthesized audio.

[0141] Specifically, the specific implementation method of the processor 10 for the above computer program can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0142] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium can be non-volatile or volatile. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0143] An embodiment of the present invention may further provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program may implement:

[0144] Acquire sample audio, perform sound feature extraction, conversion and vectorization processing on the sample audio to obtain a standard speech vector;

[0145] When receiving a text to be synthesized, performing phoneme conversion on the text to be synthesized to obtain a text phoneme sequence;

[0146] Performing vector conversion on the text phoneme sequence to obtain a text matrix;

[0147] Performing vector concatenation of the standard speech vector and the text matrix to obtain a target matrix;

[0148] Extracting spectrum features from the target matrix to obtain spectrum feature information;

[0149] The frequency spectrum feature information is subjected to speech synthesis using a preset vocoder to obtain synthesized audio.

[0150] Furthermore, the computer-usable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0151] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0152] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0154] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0155] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0156] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0157] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A speech synthesis method, It is characterized in that The method comprises: Acquire sample audio, perform sound feature extraction, conversion and vectorization processing on the sample audio to obtain a standard speech vector; When receiving a text to be synthesized, performing phoneme conversion on the text to be synthesized to obtain a text phoneme sequence; Performing vector conversion on the text phoneme sequence to obtain a text matrix; Performing vector concatenation of the standard speech vector and the text matrix to obtain a target matrix; Extracting spectrum features from the target matrix to obtain spectrum feature information; Using a preset vocoder to perform speech synthesis on the frequency spectrum feature information to obtain synthesized audio; Among them, the vector splicing of the standard speech vector and the text matrix to obtain the target matrix includes: using a preset algorithm model to calculate the phoneme frame length of each phoneme in the text phoneme sequence to obtain a phoneme frame length sequence; converting the phoneme frame length sequence into a phoneme frame length vector; horizontally splicing the phoneme frame length vector and the text matrix to obtain a standard text matrix; vertically splicing the standard speech vector with each column of the standard text matrix to obtain the target matrix.

2. The speech synthesis method according to claim 1, It is characterized in that The step of performing sound feature extraction, conversion and vectorization processing on the sample audio to obtain a standard speech vector includes: Performing sound feature extraction and conversion on the sample audio to obtain a target spectrogram; The pre-built picture classification model is used to perform feature extraction on the target spectrogram to obtain the standard speech vector.

3. The speech synthesis method according to claim 2, It is characterized in that The step of extracting and converting the sample audio to obtain a target spectrogram includes: Resampling the sample audio to obtain a digital voice signal; Pre-emphasize the digital voice signal to obtain a standard digital voice signal; The standard digital speech signal is subjected to feature conversion to obtain the target spectrogram.

4. The speech synthesis method according to claim 2, It is characterized in that The method of extracting features of the target spectrogram using a pre-built picture classification model to obtain the standard speech vector includes: Obtaining the outputs of all nodes of the fully connected layer included in the image classification model to obtain a target spectrogram feature value set; According to the order of all nodes of the fully connected layer, the feature values ​​in the target spectrogram feature value set are combined vertically to obtain a standard speech vector.

5. The speech synthesis method according to claim 3, It is characterized in that The step of performing feature conversion on the standard digital speech signal to obtain the target spectrogram includes: The standard digital speech signal is mapped in the frequency domain using a preset sound processing algorithm to obtain the target spectrogram.

6. The speech synthesis method according to any one of claims 1 to 5, It is characterized in that The step of converting the text to be synthesized into a text phoneme sequence includes: Deleting punctuation marks from the text to be synthesized to obtain a standard text; The phoneme corresponding to each character in the standard text is marked using a preset phonetic symbol rule to obtain the text phoneme sequence.

7. A speech synthesis device, used to implement the speech synthesis method according to any one of claims 1 to 6, It is characterized in that include: An audio processing module is used to obtain sample audio, perform sound feature extraction, conversion and vectorization processing on the sample audio to obtain a standard speech vector; The text processing module is used for, when receiving the text to be synthesized, performing phoneme conversion on the text to be synthesized to obtain a text phoneme sequence; performing vector conversion on the text phoneme sequence to obtain a text matrix; performing vector concatenation on the standard speech vector and the text matrix to obtain a target matrix; The speech synthesis module is used to extract the spectrum feature of the target matrix to obtain spectrum feature information; and use a preset vocoder to perform speech synthesis on the spectrum feature information to obtain synthesized audio.

8. An electronic device, It is characterized in that The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the speech synthesis method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 6 is implemented.