Speech synthesis method, device, equipment and storage medium
By combining emotion recognition and semantic analysis models with speech blockchain technology, the problem of mechanical and rigid speech synthesis is solved, generating emotionally rich speech and improving the flexibility and naturalness of speech synthesis.
Patent Information
- Application Number
- CN202111272328.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-29
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-10-29
AI Technical Summary
Existing speech synthesis methods produce mechanical and rigid speech that lacks emotional expression, resulting in the need for significant human and material resources to dub audio data that is rich in emotion in daily life.
By identifying the emotion type and role of the text to be synthesized through a pre-trained emotion recognition model and semantic analysis model, and querying the corresponding pronunciation parameters by a pre-built speech blockchain node, the text is synthesized using an audio synthesizer to generate emotionally rich speech.
It improves the flexibility of speech synthesis, generates speech with emotional features, reduces the need for human and material resources, and enhances the naturalness of speech synthesis.
Smart Images

Figure CN113990286B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech synthesis technology, and in particular to a speech synthesis method, device, electronic equipment and computer-readable storage medium. Background Art
[0002] The current speech synthesis method is to query the speech pronunciation database, confirm the pronunciation of each word or character, and finally splice the various pronunciations together for output to obtain synthesized speech. However, the synthesized speech is mechanical and rigid, not flexible enough, which greatly affects people's favorability towards speech synthesis products. As a result, various aspects of daily life, such as dubbing of novels and scripts for film and television, still require a lot of manpower, material resources and time for dubbing to obtain emotional audio data. Summary of the invention
[0003] The present invention provides a speech synthesis method, device and computer-readable storage medium, the main purpose of which is to solve the problem that the synthesized speech is mechanical and dull.
[0004] To achieve the above object, the present invention provides a speech synthesis method, comprising:
[0005] Obtaining pre-constructed text to be synthesized, and converting the text to be synthesized into basic audio data;
[0006] Using a pre-trained emotion recognition model to perform emotion recognition on the text to be synthesized, to obtain the emotion type of the text to be synthesized;
[0007] Using a pre-trained semantic analysis model to identify the role to which the text to be synthesized belongs;
[0008] Query the pronunciation parameters corresponding to the role and the emotion type from the pre-built voice blockchain node;
[0009] The pronunciation parameters and the basic speech data are input into an audio synthesizer for synthesis to obtain emotional audio data.
[0010] Optionally, converting the text to be synthesized into basic audio data includes:
[0011] Using Yan's phonetic notation method to phonetically mark the text to be synthesized to obtain a phoneme sequence;
[0012] Extracting audio segments of each syllable in the phoneme sequence from a pre-built basic pronunciation database;
[0013] The audio segments of the syllables are concatenated according to the order of the phoneme sequence to obtain basic audio data.
[0014] Optionally, the identifying the role to which the text to be synthesized belongs by using a pre-trained semantic analysis model includes:
[0015] When the text to be synthesized includes narration text and dialogue text, the semantics of the narration text is identified using a pre-trained semantic analysis model, and the scores of the narration text belonging to each character are determined according to the semantics of the narration text;
[0016] Determine whether there is a score greater than a preset qualified threshold among the scores;
[0017] When there is a belonging score greater than or equal to the preset qualified threshold, the role corresponding to the belonging score greater than or equal to the qualified threshold is used as the belonging role of the text to be synthesized;
[0018] When there is no belonging score greater than or equal to the preset qualified threshold, the semantic analysis model is used to identify the dialogue text to obtain a score queue of each role of the dialogue text;
[0019] The scores of the narration texts belonging to the respective roles and the scores in the score queue are superimposed on the scores of the corresponding roles, and the role corresponding to the highest score is extracted from the superposition result as the role to which the text to be synthesized belongs.
[0020] Optionally, the using a pre-trained emotion recognition model to perform emotion recognition on the text to be synthesized to obtain the emotion type of the text to be synthesized includes:
[0021] quantizing the text to be synthesized to obtain a text quantization matrix;
[0022] Using the convolutional layer of the emotion recognition model to extract features from the text quantization matrix, to obtain a feature matrix set;
[0023] Using the pooling layer and the flatten layer of the emotion recognition model to reduce the dimension of the feature matrix set to obtain a feature sequence;
[0024] The feature sequence is imported into the decision tree classification network of the emotion recognition model for classification, so as to obtain the emotion type of the text to be synthesized.
[0025] Optionally, quantizing the text to be synthesized to obtain a text quantization matrix includes:
[0026] Performing a word segmentation operation on the text to be synthesized to obtain a word set;
[0027] Quantizing the word set by using the Word2Vec model and the positional encoding preconfigured in the emotion recognition model to obtain a word vector set;
[0028] The word vector set is split according to a preset combination strategy, and the split results are spliced to obtain a text quantization matrix.
[0029] Optionally, before performing emotion recognition on the text to be synthesized using a pre-trained emotion recognition model to obtain the emotion type of the text to be synthesized, the method further includes:
[0030] Step I: Connect the Bert neural network with the decision tree classification network to obtain the emotion recognition model to be trained;
[0031] Step II: using the emotion recognition model to be trained to perform emotion recognition on the pre-constructed sentence sample set to obtain a recognition result;
[0032] Step III, using a preset loss function to calculate the loss value of the recognition result and the real quality inspection result corresponding to the sentence sample set;
[0033] Step IV: when the loss value is greater than the preset qualified parameter, the model parameters of the emotion recognition model to be trained are updated according to the Adaboost algorithm, and the process returns to the above step II;
[0034] Step V: When the loss value is less than or equal to the qualified parameter, a trained emotion recognition model is obtained.
[0035] Optionally, before querying the pronunciation parameters corresponding to the role and the emotion type from the pre-built voice blockchain node, the method further includes:
[0036] Obtaining a pre-constructed multi-dimensional pronunciation parameter set, and calculating the pronunciation parameter set using a message digest algorithm to obtain a character label;
[0037] The pronunciation parameter set is encrypted and chained to a pre-built voice blockchain node using the character tag.
[0038] In order to solve the above problem, the present invention further provides a speech synthesis device, the device comprising:
[0039] An audio acquisition module, used to acquire pre-constructed text to be synthesized and convert the text to be synthesized into basic audio data;
[0040] An emotion recognition module, used to perform emotion recognition on the text to be synthesized using a pre-trained emotion recognition model to obtain the emotion type of the text to be synthesized;
[0041] A role identification module, used to identify the role to which the text to be synthesized belongs using a pre-trained semantic analysis model;
[0042] A pronunciation parameter query module, used to query the pronunciation parameters corresponding to the role and the emotion type from the pre-built voice blockchain node;
[0043] The audio style setting module is used to input the pronunciation parameters and the basic voice data into an audio synthesizer for synthesis to obtain emotional audio data.
[0044] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:
[0045] at least one processor; and,
[0046] a memory communicatively connected to the at least one processor; wherein,
[0047] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the speech synthesis method described above.
[0048] In order to solve the above problem, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned speech synthesis method.
[0049] The embodiment of the present invention first uses the basic audio data converted from the audio of the text to be synthesized, and then analyzes the emotion type and the role of the text to be synthesized through the emotion recognition model and the semantic analysis model, wherein the emotion scene of the text to be synthesized can be accurately and efficiently located through the emotion recognition model and the semantic analysis model, which facilitates the subsequent query of the pronunciation parameters corresponding to the role and the emotion type from the pre-built voice blockchain node, and by synthesizing the pronunciation data with the basic audio data, the basic audio data can be tuned according to the pronunciation parameters, and finally emotional audio data with emotional characteristics can be obtained. Therefore, the speech synthesis method, device, electronic device and computer-readable storage medium proposed in the present invention can solve the problem that the speech generated by speech synthesis is mechanical and rigid, and improve the flexibility of synthesized speech. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 A flowchart of a speech synthesis method provided by an embodiment of the present invention;
[0051] Figure 2 A detailed flowchart of a step in a speech synthesis method provided by an embodiment of the present invention;
[0052] Figure 3 A detailed flowchart of a step in a speech synthesis method provided by an embodiment of the present invention;
[0053] Figure 4 A functional module diagram of a speech synthesis device provided by an embodiment of the present invention;
[0054] Figure 5 A schematic diagram of the structure of an electronic device for implementing the speech synthesis method provided by an embodiment of the present invention.
[0055] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0056] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0057] The embodiment of the present application provides a method for speech synthesis. The execution subject of the speech synthesis method includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the speech synthesis method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.
[0058] Reference Figure 1 , which is a flow chart of a speech synthesis method provided by an embodiment of the present invention.
[0059] In this embodiment, the speech synthesis method includes:
[0060] S1. Obtain pre-constructed text to be synthesized, and convert the text to be synthesized into basic audio data.
[0061] In the embodiment of the present invention, the text to be synthesized can be one sentence or multiple sentences. Specifically, the text to be synthesized can be a news text to be dubbed, or can be the text content of a novel or a storybook. Among them, the news text is required to be neutral in emotion, while the emotion types of novels and stories are relatively rich. In the embodiment of the present invention, the dubbing of novels and stories is mainly used as an example for explanation.
[0062] In addition, the text to be synthesized can be converted into basic audio data through speech recognition technology, and the speech recognition technology can be a text-to-speech engine, such as an online dictionary, etc., for converting the text to be synthesized into basic audio data that only contains the pronunciation of each word in the text and does not contain overall emotional features.
[0063] Specifically, the basic audio data contains default voice feature parameters such as timbre, speech rate, pitch, etc.
[0064] In detail, in the embodiments of the present invention, the obtaining of the pre-constructed text to be synthesized and the conversion of the text to be synthesized into basic audio data include:
[0065] Phoneticize the text to be synthesized using the Yan's phonetic notation method to obtain a phoneme sequence;
[0066] Extract the audio segments of each syllable in the phoneme sequence from a pre-constructed basic pronunciation database;
[0067] Splice the audio segments of each syllable according to the order of the phoneme sequence to obtain basic audio data.
[0068] Among them, the Yan's phonetic notation method generally uses the International Phonetic Alphabet (abbreviated as IPA) as a marking tool to convert each character in the text to be synthesized into phonemes. For example, convert the text [ā, ài, dài...] into phonemes [ā, ài, dài...]. Among them, the phoneme is the smallest speech unit divided according to the natural attributes of speech. Analyzing according to the pronunciation actions in the syllable, phonemes can be classified into single phonemes and multiple phonemes. For example, [ā, ài, dài...] are single-phoneme, double-phoneme, and triple-phoneme syllables respectively.
[0069] In addition, the basic pronunciation database can be any speech dictionary database issued by an authoritative institution.
[0070] Specifically, diphones (from the center of one phoneme to the center of the next phoneme) can be used as units to connect the audio segments of each syllable unit in the phoneme sequence to obtain basic audio data, thereby improving the coherence of the synthesized speech.
[0071] S2. Use a pre-trained emotion recognition model to perform emotion recognition on the text to be synthesized to obtain the emotion type of the text to be synthesized.
[0072] In the embodiments of the present invention, the emotion recognition model is a neural network model constructed by a Bert neural network and a decision tree classification network, and is used to judge the emotion type of the text to be synthesized according to the context of each word and phrase in the text to be synthesized. Among them, the emotion type includes the emotion category and emotion intensity level of the text to be synthesized.
[0073] Detailed, such as Figure 2 As shown, in the embodiment of the present invention, the emotion recognition of the text to be synthesized by using the pre-trained emotion recognition model to obtain the emotion type of the text to be synthesized includes:
[0074] S21, quantizing the text to be synthesized to obtain a text quantization matrix.
[0075] Furthermore, in an embodiment of the present invention, quantizing the text to be synthesized to obtain a text quantization matrix includes:
[0076] Performing a word segmentation operation on the text to be synthesized to obtain a word set;
[0077] Quantizing the word set by using the Word2Vec model and the positional encoding preconfigured in the emotion recognition model to obtain a word vector set;
[0078] According to a preset combination strategy, the word vector set is split, and the split results are spliced to obtain a text quantization matrix.
[0079] Among them, the position encoding is a position vector generated according to the order and number of word vectors in the text to be synthesized, such as [E0, E1, E2, ...], which is mainly configured in the initial layer of the BERT neural network of the emotion recognition model to ensure that the text to be synthesized can be sequentially and completely processed for subsequent text recognition.
[0080] Specifically, the embodiment of the present invention utilizes 2-character, 3-character and 4-character windows in the jieba tool to sequentially intercept the text to be synthesized from front to back to obtain a word block set, and utilizes an authoritative common word database to perform existence judgment on each word block in the word block set, and outputs the word blocks existing in the common word database to obtain a word set.
[0081] In the embodiment of the present invention, the Word2Vec model is a group of related models used to generate word vectors, which can map each word in the word set to a vector to obtain an initial word vector set. However, the initial word vector set generated by the Word2Vec model is discrete. In order to enhance the features in the subsequent text, the embodiment of the present invention processes each sentence separately, so the extra-sentence encoding is not considered. The initial word vector set is positionally encoded only through the pre-constructed intra-sentence encoding [E0, E1, E2, ...] to obtain a word vector set. For example, the word vector set may be [E0+E1, E2, ...] 我们 、E1+E 马上 ……】.
[0082] In the embodiment of the present invention, if the word vector set is long, the word vector set is cut into L bytes to form a text quantization matrix of L×N (number of columns), wherein the remaining vector data in the Nth row can be padded with '0' to complete the matrix.
[0083] S22. Using the convolutional layer of the emotion recognition model to perform feature extraction on the text quantization matrix to obtain a feature matrix set.
[0084] In the embodiment of the present invention, the convolution layer has a preset number M of convolution kernels, and each convolution kernel is used to perform a sliding convolution calculation on the text quantization matrix to obtain a feature matrix set including M feature matrices;
[0085] S23. Use the pooling layer and flatten layer of the emotion recognition model to reduce the dimension of the feature matrix set to obtain a feature sequence.
[0086] In the embodiment of the present invention, the feature matrix set is pooled by using the maximum pooling operation, the main features are retained, and the secondary features are excluded, so as to obtain a reduced-dimensional feature matrix set, and the reduced-dimensional feature matrix set is flattened by using the flatten layer to obtain a feature sequence including each one-dimensional feature;
[0087] S24, importing the feature sequence into the decision tree classification network of the emotion recognition model for classification, and obtaining the emotion type of the text to be synthesized.
[0088] Specifically, the embodiment of the present invention performs a full connection operation on the text feature sequence in the feature sequence, identifies the sentence scene corresponding to each text feature, and then performs a decision tree classification operation on each sentence scene to obtain the emotion type of the text to be synthesized.
[0089] In addition, in the embodiment of the present invention, before the step S2, the method further includes:
[0090] Step I: Connect the Bert neural network with the decision tree classification network to obtain the emotion recognition model to be trained;
[0091] Step II: using the emotion recognition model to be trained to perform emotion recognition on the pre-constructed sentence sample set to obtain a recognition result;
[0092] For example, a preset number of sentences are randomly extracted from a large number of novels for output storage to obtain a sentence sample set;
[0093] Step III, using a preset loss function to calculate the loss value of the recognition result and the real quality inspection result corresponding to the sentence sample set;
[0094] The actual quality inspection result is the result label after professionals evaluate the extracted sentences when randomly extracting a preset number of sentences from a large number of novels in the above-mentioned step.
[0095] In the embodiment of the present invention, the sentence sample set is imported into the emotion recognition model to be trained for text recognition, wherein the process is similar to the above-mentioned process from S21 to S24, and will not be described in detail here;
[0096] The loss value in the embodiment of the present invention is:
[0097] Loss = αLoss 识别 +βLoss 决策树
[0098] Among them, the Loss 识别 is the loss function of the Bert neural network. 决策树 is the loss function corresponding to the decision tree classification network, and α and β are weight coefficients, which are configured according to the training effect in the embodiment of the present invention.
[0099] Step IV: when the loss value is greater than the preset qualified parameter, the model parameters of the emotion recognition model to be trained are automatically updated according to the Adaboost algorithm, and the process returns to the above step II;
[0100] The Adaboost is an iterative algorithm that continuously changes the weight position of each decision tree in the classification process in the emotion recognition model through iterative training, and finally makes the classification result more accurate. Specifically, the embodiment of the present invention uses the Adaboost algorithm to change the weight coefficient of each classifier in the decision tree classification network during each training process, and the weight of the correctly classified sentence samples will be reduced when the next classification is performed, and the weight of the incorrectly classified sentence samples will be increased when the next classification is performed, and according to the loss function, the model parameters in the classification network are updated, so that the number of incorrectly classified sentence samples is reduced and the loss value is reduced.
[0101] Step V: When the loss value is less than or equal to the qualified parameter, a trained emotion recognition model is obtained.
[0102] In the embodiment of the present invention, training is performed according to the operation steps of step I to step V, and finally a trained emotion recognition model is obtained.
[0103] S3. Using a pre-trained semantic analysis model, identify the role to which the text to be synthesized belongs.
[0104] The character is a male or female news anchor character, or a character with various types of voices in a novel or a storybook, such as "an elderly male voice, a middle-aged female voice, a loli voice..." In the embodiment of the present invention, the pre-constructed sentence samples are dubbed by voice actors with various types of voices.
[0105] Detailed, such as Figure 3 As shown, in another embodiment of the present invention, the identifying the role of the text to be synthesized by using a pre-trained semantic analysis model includes:
[0106] S31, when the text to be synthesized includes narration text and dialogue text, using a pre-trained semantic analysis model to identify the semantics of the narration text, and determining the scores of the narration text belonging to each role according to the semantics of the narration text;
[0107] S32, determining whether there is a score greater than a preset qualified threshold among the scores;
[0108] S33, when there is a belonging score greater than or equal to the preset qualified threshold, taking the role corresponding to the belonging score greater than or equal to the qualified threshold as the belonging role of the text to be synthesized;
[0109] S34, when there is no belonging score greater than or equal to the preset qualified threshold, using the semantic analysis model to identify the dialogue text, and obtain the score queue of each role of the dialogue text;
[0110] S35, performing a score superposition operation on the scores of the narration text belonging to each role and each score in the score queue, and extracting the role corresponding to the highest score from the superposition result as the role to which the text to be synthesized belongs.
[0111] Specifically, when analyzing the character corresponding to the text to be synthesized, the narration text and dialogue text in the text to be synthesized can be extracted first, such as [……, the scout said: "Report! General, an enemy food team was found thirty miles ahead."], and the "scout said" in the narration text (i.e., the text outside the quotation marks) is extracted to obtain the score of the character to which the text to be synthesized belongs as [military officer: 0%, general: 0%, scout: 100%], and among them, the scout's score of 100% is greater than the preset qualified threshold of 80%, then the character of the text to be synthesized is a "scout".
[0112] In addition, when the text to be synthesized is […, a man hurried in from outside the tent: "General, an enemy grain team was found thirty miles ahead."], the narration text (that is, the text outside the quotation marks "A man hurried in from outside the tent") is identified, and the scores may be [Military: 30%, General: 0%, Scout: 50%, ...], among which the maximum is [Scout: 50%] but less than the qualified threshold of 80%, then it is necessary to identify the dialogue text [General, an enemy grain team was found thirty miles ahead] again, and the score queue is [Military: 10%, Captain: 20%, Scout 40%, ...] Then, by adding up the scores of the corresponding roles, we get [Military: 40%, ..., Scout: 100%, ...], and select the role with the highest score, that is, "Scout" as the role to which the text to be synthesized belongs.
[0113] S4. Query the pronunciation parameters corresponding to the role and the emotion type from the pre-built voice blockchain node.
[0114] The voice blockchain node is a database that extracts and stores the multi-dimensional pronunciation parameters of the sentence samples through blockchain technology.
[0115] In an embodiment of the present invention, when the role and emotion type of the text to be synthesized are identified, the role and emotion type can be used as keywords to query the voice blockchain node to obtain multi-dimensional pronunciation parameters corresponding to the role and emotion type, such as voice fundamental frequency and harmonic frequency, unvoiced possibility (between 0 and 1), voiced possibility (between 0 and 1), speech speed or duration of pronunciation state, intonation and rhythm, etc.
[0116] Specifically, the pronunciation parameters corresponding to the role and the pronunciation parameters corresponding to the emotion type are queried during the query. For example, if the role of the sentence text identified by the embodiment of the present invention is a scout, and the emotion type is a high-level tension type including haste and tension, then the timbre information of the scout role is first queried through the voice blockchain node, and then the high-level tension type under the scout role is queried to obtain the pronunciation parameters including speech speed and pause habit information.
[0117] In the embodiment of the present invention, before querying the pronunciation parameters corresponding to the role and the emotion type from the pre-built voice blockchain node, the method further includes:
[0118] Obtaining a pre-constructed multi-dimensional pronunciation parameter set, and calculating the pronunciation parameter set using a message digest algorithm to obtain a character label;
[0119] The pronunciation parameter set is encrypted and chained to a pre-built voice blockchain node using the character tag.
[0120] Among them, the information summary algorithm is a very important branch of cryptographic algorithms. It extracts fingerprint information from all data to achieve functions such as data signature and data integrity verification. The algorithm has its irreversibility and can prevent deciphering by brute force enumeration.
[0121] Specifically, the embodiment of the present invention obtains the voice actors of different roles in the preset text (such as novels and storybooks) in advance, dubs the preset sentence samples under various preset emotion type entries, obtains sample audio, and then extracts the multi-dimensional pronunciation parameters such as timbre, pitch, tone, word pause interval, etc. in the sample audio to obtain multi-dimensional pronunciation parameters, and then encrypts the multi-dimensional pronunciation parameters through the message digest algorithm and chains them to the blockchain node for storage. Among them, the embodiment of the present invention uses blockchain technology to store the multi-dimensional pronunciation parameters of various types of voices to prevent important data from being lost or changed.
[0122] S5. Input the pronunciation parameters and the basic speech data into an audio synthesizer for synthesis to obtain emotional audio data.
[0123] Specifically, the audio synthesizer may be a TrakAxPC audio processor.
[0124] For example, the basic audio data is imported into a pre-built TrakAxPC audio processor, and after obtaining pronunciation parameters including speech speed and pause habit information according to the advanced tension type under the queried scout character, the basic audio data is tuned according to the pronunciation parameters to obtain the pronunciation audio of the scout character with an adult male voice and a fast speaking speed.
[0125] Furthermore, when the text to be synthesized contains narration text and non-narration text, during speech synthesis, only the style reconstruction operation can be performed on the dialogue text belonging to individual characters contained in the text to be synthesized, that is, the dialogue text belonging to individual characters contained in the text to be synthesized is adjusted according to the role and emotion type, without the need to adjust the narration text part (such as changing the timbre and emotion).
[0126] The embodiment of the present invention first uses the basic audio data converted from the audio of the text to be synthesized, and then analyzes the emotion type and the role of the text to be synthesized through the emotion recognition model and the semantic analysis model, wherein the emotion scene of the text to be synthesized can be accurately and efficiently located through the emotion recognition model and the semantic analysis model, which facilitates the subsequent query of the pronunciation parameters corresponding to the role and the emotion type from the pre-built voice blockchain node, and by synthesizing the pronunciation data with the basic audio data, the basic audio data can be tuned according to the pronunciation parameters, and finally emotional audio data with emotional characteristics can be obtained. Therefore, the speech synthesis method, device, electronic device and computer-readable storage medium proposed in the present invention can solve the problem that the speech generated by speech synthesis is mechanical and rigid, and improve the flexibility of synthesized speech.
[0127] like Figure 4 , which is a functional module diagram of a speech synthesis device provided in one embodiment of the present invention.
[0128] The speech synthesis device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the speech synthesis device 100 can include an audio acquisition module 101, an emotion recognition module 102, a role recognition module 103, a pronunciation parameter query module 104 and an audio style setting module 105. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.
[0129] In this embodiment, the functions of each module / unit are as follows:
[0130] The audio acquisition module 101 is used to acquire pre-constructed text to be synthesized and convert the text to be synthesized into basic audio data;
[0131] The emotion recognition module 102 is used to perform emotion recognition on the text to be synthesized using a pre-trained emotion recognition model to obtain the emotion type of the text to be synthesized;
[0132] The role identification module 103 is used to identify the role of the text to be synthesized by using a pre-trained semantic analysis model;
[0133] The pronunciation parameter query module 104 is used to query the pronunciation parameters corresponding to the role and the emotion type from the pre-built voice blockchain node;
[0134] The audio style setting module 105 is used to input the pronunciation parameters and the basic voice data into an audio synthesizer for synthesis to obtain emotional audio data.
[0135] In detail, each module described in the speech synthesis device 100 in the embodiment of the present invention is used in the same manner as described above. Figures 1 to 3 The present invention has the same technical means as the speech synthesis method described in , and can produce the same technical effects, which will not be repeated here.
[0136] like Figure 5 FIG. 1 is a schematic diagram of the structure of an electronic device for implementing a speech synthesis method provided by an embodiment of the present invention.
[0137] The electronic device 1 may include a processor 10 , a memory 11 , a communication bus 12 , and a communication interface 13 , and may also include a computer program stored in the memory 11 and executable on the processor 10 , such as a speech synthesis program.
[0138] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core (ControlUnit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules (such as executing speech synthesis programs, etc.) stored in the memory 11, and calls data stored in the memory 11 to execute various functions of the electronic device and process data.
[0139] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 may not only be used to store application software and various types of data installed in the electronic device, such as the code of a speech synthesis program, etc., but may also be used to temporarily store data that has been output or is to be output.
[0140] The communication bus 12 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.
[0141] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.
[0142] Figure 5 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.
[0143] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.
[0144] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.
[0145] The speech synthesis program stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve:
[0146] Obtaining pre-constructed text to be synthesized, and converting the text to be synthesized into basic audio data;
[0147] Using a pre-trained emotion recognition model to perform emotion recognition on the text to be synthesized, to obtain the emotion type of the text to be synthesized;
[0148] Using a pre-trained semantic analysis model to identify the role to which the text to be synthesized belongs;
[0149] Query the pronunciation parameters corresponding to the role and the emotion type from the pre-built voice blockchain node;
[0150] The pronunciation parameters and the basic speech data are input into an audio synthesizer for synthesis to obtain emotional audio data.
[0151] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.
[0152] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).
[0153] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:
[0154] Obtaining pre-constructed text to be synthesized, and converting the text to be synthesized into basic audio data;
[0155] Using a pre-trained emotion recognition model to perform emotion recognition on the text to be synthesized, to obtain the emotion type of the text to be synthesized;
[0156] Using a pre-trained semantic analysis model to identify the role to which the text to be synthesized belongs;
[0157] Query the pronunciation parameters corresponding to the role and the emotion type from the pre-built voice blockchain node;
[0158] The pronunciation parameters and the basic speech data are input into an audio synthesizer for synthesis to obtain emotional audio data.
[0159] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.
[0160] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0161] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.
[0162] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.
[0163] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.
[0164] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.
[0165] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0166] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.
[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.
Claims
1. A speech synthesis method, characterized in that: The method comprises: Obtaining pre-constructed text to be synthesized, and converting the text to be synthesized into basic audio data; Using a pre-trained emotion recognition model to perform emotion recognition on the text to be synthesized, to obtain the emotion type of the text to be synthesized; Using a pre-trained semantic analysis model to identify the role to which the text to be synthesized belongs; Query the pronunciation parameters corresponding to the role and the emotion type from the pre-built voice blockchain node; Inputting the pronunciation parameters and the basic audio data into an audio synthesizer for synthesis to obtain emotional audio data; The step of using a pre-trained semantic analysis model to identify the role to which the text to be synthesized belongs includes: When the text to be synthesized includes narration text and dialogue text, the semantics of the narration text is identified using a pre-trained semantic analysis model, and the scores of the narration text belonging to each character are determined according to the semantics of the narration text; Determine whether there is a score greater than a preset qualified threshold among the scores; When there is a belonging score greater than or equal to the preset qualified threshold, the role corresponding to the belonging score greater than or equal to the qualified threshold is used as the belonging role of the text to be synthesized; When there is no belonging score greater than or equal to the preset qualified threshold, the semantic analysis model is used to identify the dialogue text to obtain a score queue of each role of the dialogue text; The scores of the narration texts belonging to the respective roles and the scores in the score queue are superimposed on the scores of the corresponding roles, and the role corresponding to the highest score is extracted from the superposition result as the role to which the text to be synthesized belongs.
2. The speech synthesis method according to claim 1, characterized in that: The step of converting the text to be synthesized into basic audio data comprises: Using Yan's phonetic notation method to phonetically mark the text to be synthesized to obtain a phoneme sequence; Extracting audio segments of each syllable in the phoneme sequence from a pre-built basic pronunciation database; The audio segments of the syllables are concatenated according to the order of the phoneme sequence to obtain basic audio data.
3. The speech synthesis method according to claim 1, characterized in that: The using of the pre-trained emotion recognition model to perform emotion recognition on the text to be synthesized to obtain the emotion type of the text to be synthesized includes: quantizing the text to be synthesized to obtain a text quantization matrix; Using the convolutional layer of the emotion recognition model to extract features from the text quantization matrix, to obtain a feature matrix set; Using the pooling layer and the flatten layer of the emotion recognition model to reduce the dimension of the feature matrix set to obtain a feature sequence; The feature sequence is imported into the decision tree classification network of the emotion recognition model for classification, so as to obtain the emotion type of the text to be synthesized.
4. The speech synthesis method according to claim 3, characterized in that: The step of quantizing the text to be synthesized to obtain a text quantization matrix includes: Performing a word segmentation operation on the text to be synthesized to obtain a word set; Quantizing the word set by using the Word2Vec model and the positional encoding preconfigured in the emotion recognition model to obtain a word vector set; The word vector set is split according to a preset combination strategy, and the split results are spliced to obtain a text quantization matrix.
5. The speech synthesis method according to claim 1, characterized in that: Before performing emotion recognition on the text to be synthesized by using the pre-trained emotion recognition model to obtain the emotion type of the text to be synthesized, the method further includes: Step I: Connect the Bert neural network with the decision tree classification network to obtain the emotion recognition model to be trained; Step II: using the emotion recognition model to be trained to perform emotion recognition on the pre-constructed sentence sample set to obtain a recognition result; Step III, using a preset loss function to calculate the loss value of the recognition result and the real quality inspection result corresponding to the sentence sample set; Step IV: when the loss value is greater than the preset qualified parameter, the model parameters of the emotion recognition model to be trained are updated according to the Adaboost algorithm, and the process returns to the above step II; Step V: When the loss value is less than or equal to the qualified parameter, a trained emotion recognition model is obtained.
6. The speech synthesis method according to claim 1, characterized in that: Before querying the pronunciation parameters corresponding to the role and the emotion type from the pre-built voice blockchain node, the method further includes: Obtaining a pre-constructed multi-dimensional pronunciation parameter set, and calculating the pronunciation parameter set using a message digest algorithm to obtain a character label; The pronunciation parameter set is encrypted and chained to a pre-built voice blockchain node using the character tag.
7. A speech synthesis device, characterized in that: The device comprises: An audio acquisition module, used to acquire pre-constructed text to be synthesized and convert the text to be synthesized into basic audio data; An emotion recognition module, used to perform emotion recognition on the text to be synthesized using a pre-trained emotion recognition model to obtain the emotion type of the text to be synthesized; A role identification module, used to identify the role to which the text to be synthesized belongs using a pre-trained semantic analysis model; A pronunciation parameter query module, used to query the pronunciation parameters corresponding to the role and the emotion type from the pre-built voice blockchain node; An audio style setting module, used for inputting the pronunciation parameters and the basic audio data into an audio synthesizer for synthesis to obtain emotional audio data; The step of using a pre-trained semantic analysis model to identify the role to which the text to be synthesized belongs includes: When the text to be synthesized includes narration text and dialogue text, the semantics of the narration text is identified using a pre-trained semantic analysis model, and the scores of the narration text belonging to each character are determined according to the semantics of the narration text; Determine whether there is a score greater than a preset qualified threshold among the scores; When there is a belonging score greater than or equal to the preset qualified threshold, the role corresponding to the belonging score greater than or equal to the qualified threshold is used as the belonging role of the text to be synthesized; When there is no belonging score greater than or equal to the preset qualified threshold, the semantic analysis model is used to identify the dialogue text to obtain a score queue of each role of the dialogue text; The scores of the narration texts belonging to the respective roles and the scores in the score queue are superimposed on the scores of the corresponding roles, and the role corresponding to the highest score is extracted from the superposition result as the role to which the text to be synthesized belongs.
8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the speech synthesis method according to any one of claims 1 to 6.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 6 is implemented.
