A voice playing method and computer device
By combining the joint input of pronunciation feature prediction network and acoustic model, the problem of low voice quality when text is converted to speech in the prior art is solved, and better voice quality and user experience are achieved.
Patent Information
- Application Number
- CN202111678366.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-31
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2041-12-31
AI Technical Summary
The prior art when converting text into speech, the voice quality is not high, affecting the user experience.
Using trained pronunciation feature prediction network and acoustic models, context dependency mining is carried out through the joint input of text features and pronunciation features, and Mel spectrum is output to improve speech quality.
It improves the voice quality, makes the voice content played better and improves the user experience.
Smart Images
Figure CN114495896B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a voice playing method, a voice playing device, a computer device and a computer readable storage medium. Background Art
[0002] Nowadays, with the popularity of smart electronic products, more and more electronic devices have voice interaction functions, such as smart speakers and smart reading machines. Correspondingly, the quality requirements for voice playback by electronic devices are getting higher and higher.
[0003] In the prior art, when implementing voice interaction, electronic devices first convert the acquired text files into voice files before playing them. Although, when converting text files into voice files, existing conversion models can be sampled to map text content into voice content, this solution only maps the phonemes of each word in the text, resulting in a relatively flat tone in the voice content, that is, poor voice quality, which affects the user experience. Summary of the invention
[0004] In view of this, the embodiments of the present application provide a voice playback method, a voice playback device, a computer device and a computer-readable storage medium to solve the problem of poor voice quality after converting text into voice in the voice playback solution in the prior art.
[0005] A first aspect of an embodiment of the present application provides a voice playback method, including:
[0006] Using the trained prosodic feature prediction network, prosodic prediction is performed according to the text features of the text sequence to obtain the prosodic features of the text sequence;
[0007] The text feature and the rhythm feature are used as joint features, input into the trained acoustic model to perform context dependency mining, and output a Mel spectrum;
[0008] The speech content corresponding to the text sequence is played based on the Mel spectrum.
[0009] Optionally, before the step of using the trained prosodic feature prediction network to perform prosodic prediction according to the text features of the text sequence to obtain the prosodic features of the text sequence, the method further includes:
[0010] Analyzing and processing the text sequence samples and the speech samples corresponding to the text sequence samples respectively to obtain mutually related text sample features and speech sample features;
[0011] The text sample features and the speech sample features are used as first training samples to train a prosodic feature prediction network to obtain a trained prosodic feature prediction network.
[0012] Optionally, before the step of taking the text feature and the prosodic feature as joint features, inputting the trained acoustic model to perform context dependency mining, and outputting the mel spectrum, the step further includes:
[0013] The text sample features and the speech sample features are used as second training samples to train the acoustic model to obtain a trained acoustic model.
[0014] Optionally, the using a trained prosody feature prediction network to perform prosody prediction according to text features of a text sequence to obtain the prosody features of the text sequence includes:
[0015] Extracting time information and context dependency based on the encoding result of the text sequence through a long short-term memory network to obtain text features of the text sequence; wherein the encoding result of the text sequence is obtained by encoding the text sequence using a deep neural network; the text features carry the phoneme information of the text sequence;
[0016] The trained prosodic feature prediction network is used to perform prosodic prediction based on the phoneme information carried by the text features to obtain the prosodic features of the text sequence.
[0017] Optionally, the text feature and the rhythm feature are used as joint features, input into a trained acoustic model to perform context dependency mining, and output a Mel spectrum, including:
[0018] Splicing the position code for the joint features of each frame to obtain the joint features carrying the position code;
[0019] Through the multi-head attention module in the trained acoustic model, context dependency mining is performed based on the joint features carrying position encoding, and a Mel spectrum is output.
[0020] Optionally, the multi-head attention module in the trained acoustic model performs context dependency mining based on the joint features carrying the positional encoding and outputs a Mel spectrum, including:
[0021] Extracting features of the joint features carrying the positional encoding through a first multi-head attention layer in the trained acoustic model, and outputting feature extraction results through a first fully connected layer;
[0022] Using the length regulator in the trained acoustic model and the preset duration model, the feature extraction result is extended along the time dimension to obtain an extended feature set;
[0023] Through the second multi-head attention layer in the trained acoustic model, a linear transformation is performed based on the extended feature set to output a Mel spectrum.
[0024] Optionally, the performing a linear transformation based on the extended feature set through the second multi-head attention layer in the trained acoustic model to output a Mel spectrum includes:
[0025] splicing the position code for each frame of extended features in the extended feature set to obtain an extended feature set carrying the position code;
[0026] Performing a nonlinear transformation on the extended feature set carrying the position code through a second multi-head attention layer in the trained acoustic model, and outputting the nonlinear transformation result through a second fully connected layer;
[0027] The convolution layer in the trained acoustic model performs residual calculation on the nonlinear transformation result, and the linear layer performs regression prediction based on the residual calculation result to output the Mel spectrum.
[0028] A second aspect of an embodiment of the present application provides a voice playback device, including:
[0029] A prediction unit, configured to use a trained prosodic feature prediction network to perform prosodic prediction according to text features of a text sequence to obtain prosodic features of the text sequence;
[0030] A mining unit, used for taking the text feature and the prosodic feature as joint features, inputting them into a trained acoustic model to perform context dependency mining, and outputting a mel spectrum;
[0031] An output unit is used to play the speech content corresponding to the text sequence based on the Mel spectrum.
[0032] A third aspect of an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, each step of the voice playback method described in the first aspect or any optional method of the first aspect is implemented.
[0033] The fourth aspect of the embodiments of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the various steps in the voice playback method described in the first aspect or any optional manner of the first aspect.
[0034] A fifth aspect of an embodiment of the present application provides a computer program product. When the computer program product runs on a computer device, the computer device executes each step of the voice playback method described in the first aspect or any optional manner of the first aspect.
[0035] The implementation of a voice playback method, a voice playback device, a computer device, a computer-readable storage medium, and a computer program product provided by the embodiments of the present application has the following beneficial effects:
[0036] In the embodiment of the present application, the prosody prediction is performed based on the text features of the text sequence by using a trained prosody feature prediction network to obtain the prosody features of the text sequence. Since the prosody features can be used to characterize the acoustic features of the unit text content in the text sequence, the text features and the prosody features are used as joint features and input into the trained acoustic model for context dependency mining, so that the output Mel spectrum can be used to characterize the acoustic features, context dependencies and pronunciation timing of the unit text content in the text sequence. Finally, the speech content corresponding to the text sequence is played based on the Mel spectrum, which can improve the speech quality in the scenario where text is converted to speech. In addition, since a speech playback method provided in the embodiment of the present application can improve the speech quality in the scenario where text is converted to speech, that is, the played speech content sounds better, and the user experience is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0038] Figure 1 This is a flow chart of an implementation of a voice playback method provided in an embodiment of the present application;
[0039] Figure 2 is a flow chart of an implementation of a voice playback method provided by another embodiment of the present application;
[0040] Figure 3 is a flow chart of an implementation of a voice playback method provided in yet another embodiment of the present application;
[0041] Figure 4 It is a structural block diagram of a voice playback device provided in an embodiment of the present application;
[0042] Figure 5 It is a structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0044] The present embodiment provides a method for voice playback, the execution subject is a computer device, including but not limited to a personal computer, a mobile phone or a tablet computer, etc. By configuring a target script file for the computer device or an application in the computer device, the target script file describes the voice playback method provided in the present embodiment, so that the computer device executes the target script file when implementing voice interaction, and then executes each step in the voice playback method. Alternatively, the voice playback method provided in the present embodiment is also described by the target script file, the target script file is configured in the computer device, and a corresponding calling interface is provided for the application or browser installed on the computer device, so that after the computer device loads a page file with a voice interaction function, it can call and execute the target script file, and then perform each step in the voice playback method.
[0045] During implementation, the computer device uses a trained prosodic feature prediction network to perform prosody prediction based on the text features of a text sequence to obtain the prosodic features of the text sequence. Since the prosodic features can be used to characterize the acoustic features of unit text content in a text sequence, the text features and prosodic features are used as joint features and input into a trained acoustic model for context dependency mining, so that the output Mel spectrum can be used to characterize the acoustic features, context dependencies and pronunciation timing of unit text content in a text sequence. Finally, the computer device plays the speech content corresponding to the text sequence based on the Mel spectrum, which can improve the speech quality in the scenario of converting text into speech.
[0046] In addition, due to the voice playback method provided in the embodiment of the present application, the voice quality can be improved in the scenario of converting text into voice, that is, the played voice content sounds better, thereby improving the user experience.
[0047] The following is a detailed description of the voice playback method provided in the embodiment of the present application through a specific implementation method.
[0048] See also Figure 1 , is a flow chart of an implementation of a voice playback method provided in an embodiment of the present application, such as Figure 1 As shown, the voice playing method may include S11 to S13, which are described in detail as follows:
[0049] S11: Using the trained prosodic feature prediction network, prosodic prediction is performed according to the text features of the text sequence to obtain the prosodic features of the text sequence.
[0050] In step S11, the text sequence represents the text content to be converted to speech. The trained prosodic feature prediction network is used to describe the correspondence between text features and prosodic features, that is, it can be used to perform prosodic prediction based on text features to obtain prosodic features of the text sequence.
[0051] In this embodiment, the prosodic features are also called suprasegmental features. Prosodic features are all features including pitch, intensity, duration and their relationships, in addition to timbre features such as vowels and consonants. Prosodic features are manifested in tone, intonation, stress and rhythm in phonetics.
[0052] In a specific implementation, the trained prosodic feature prediction network can be obtained by constructing a prosodic feature prediction network based on a deep neural network and then training the prosodic feature prediction network on a mapping relationship using pre-configured training samples. Here, the training samples include text sample features and prosodic annotations, and the text sample features are used as the input of the prosodic feature prediction network, and the prosodic annotations are used as the target for training, so that the trained prosodic feature prediction network can be used to describe the corresponding relationship between text features and prosodic features.
[0053] In some existing technologies, when playing voice, after extracting text features from a text sequence, the text features are directly mapped to the voice content, and then the voice content corresponding to the text content is output. When the voice content is adjusted or optimized later, it can only be adjusted in terms of timbre. For example, after extracting timbre features from a voice sample of a target person, it is embedded in the voice content so that the voice content sounds close to the voice of the target person. However, the optimized voice content is only close to the target person in terms of timbre, and cannot be optimized or polished from the perspective of text rhythm.
[0054] In this embodiment, it is considered that in some scenarios where text is converted to speech, the prosodic features determine the rendering ability and listening comfort of the speech content. For example, when converting ancient poetry text into speech, more prosodic features need to be considered. If the text content is simply mapped to the speech template, it is easy to cause the converted speech content to be unable to better express the emotions in the text content. Based on this, the prosodic features of the text sequence are predicted based on the text features of the text sequence using a trained prosodic feature prediction network, and the prosodic features of the text sequence are obtained. When converting the text sequence to speech, the prosodic features can be used as one of the elements considered in the speech conversion process, thereby providing a basis for improving speech quality.
[0055] As an embodiment, step S11 includes the following steps:
[0056] The time information and context dependency are extracted based on the encoding result of the text sequence through a long short-term memory network to obtain the text features of the text sequence; wherein the encoding result of the text sequence is obtained by encoding the text sequence using a deep neural network; the text features carry the phoneme information of the text sequence; and a trained rhythm feature prediction network is used to perform rhythm prediction based on the phoneme information carried by the text features to obtain the rhythm features of the text sequence.
[0057] In this embodiment, the trained prosodic feature prediction network is used to perform prosodic prediction based on the text features of the text sequence, specifically, the prosodic prediction is performed based on the phoneme information of the text features. Here, phoneme is the smallest speech unit divided according to the natural attributes of speech, and is analyzed based on the pronunciation action in the syllable, and one action constitutes one phoneme. At the same time, phoneme is also the smallest unit or the smallest speech segment that constitutes a syllable, and is the smallest linear speech unit divided from the perspective of sound quality.
[0058] It should be noted that since the encoding result of the text sequence is obtained by encoding the text sequence using a deep neural network, the time information and context dependency relationship of the encoding result of the text sequence can be directly extracted through the long short-term memory network. Here, the time information represents the temporal order of the phoneme information, which is related to the time dimension of the prosodic feature. The context dependency relationship represents the influence of the interrelated parts of the text content on the phoneme information, which is the prosodic correlation relationship between different contents in the prosodic feature.
[0059] It is easy to understand that in specific implementation, other deep neural networks or other rule models can also be used to directly predict the rhythm of text. For example, by pre-configuring the rhythm relationship network between different texts, the rhythm features between different texts are characterized, and then the rhythm relationship network is used to directly perform rhythm mining or rhythm prediction on the text sequence to obtain the rhythm features corresponding to the text sequence.
[0060] S12: The text feature and the rhythm feature are used as joint features, input into the trained acoustic model to perform context dependency mining, and output the Mel spectrum.
[0061] In step S12, the joint features include the text features and the prosodic features of the text sequence. The trained acoustic model mines the context dependency of the joint features to avoid the phenomenon of multiple words and missing words.
[0062] In this embodiment, the text feature and the prosodic feature are used as a joint feature. Specifically, the text feature and the prosodic feature are concatenated, that is, the vector of the text feature and the vector of the prosodic feature are concatenated to obtain a two-dimensional vector, which is used as the prosodic feature.
[0063] In the specific implementation, since the rhythmic features are two-dimensional vectors, the trained acoustic model uses the two-dimensional vector representing the joint features as input, and can express the two-dimensional vector in a higher-dimensional space. Here, the trained acoustic model contains a duration model for duration prediction, which can output a more accurate pronunciation time, that is, predict and express the acoustic features, context dependencies, and pronunciation timing of the unit text content, so that the overall synthesized speech rhythm is more in line with the real voice characteristics, and the voice fluctuations are more obvious, and the pauses between words are more reasonable.
[0064] It is easy to understand that since the Mel spectrum is obtained by mining the context dependency of the joint features through the trained acoustic model, and the joint features are obtained by concatenating the text features and the rhythmic features, when predicting the phase spectrum based on the Mel spectrum, the text features and the rhythmic features of the text sequence can be considered at the same time, so that the predicted phase spectrum can have both the text feature part and the rhythmic feature part, so that the speech corresponding to the phase spectrum has a sense of rhythm and becomes more vivid.
[0065] As an embodiment, step S12 specifically includes the following steps:
[0066] Step 1: Concatenate the position code of the joint feature of each frame to obtain the joint feature carrying the position code.
[0067] Step 2: Through the multi-head attention module in the trained acoustic model, context dependency mining is performed based on the joint features carrying position encoding, and a Mel spectrum is output.
[0068] In this embodiment, the position code is used to characterize the position of each frame joint feature in the overall joint feature. Here, the position code can be calculated based on the position code of each frame joint feature. For example, the cosine value is calculated based on the sequence value of each frame joint feature, and the calculated cosine value is used as the position code corresponding to the frame joint feature.
[0069] Here, the joint features carrying the position code are input into the trained acoustic model, and the multi-head attention module in the model performs context dependency mining on the joint features carrying the position code. In the specific implementation, it can be implemented by setting multiple groups of multi-head attention modules. For example, the first group of multi-head attention modules encodes the joint features carrying the position code, and after splicing the encoded results through the fully connected layer, the second group of multi-head attention modules decodes and linearly transforms them to output the Mel spectrum.
[0070] As a possible implementation method of this embodiment, the above step: performing context dependency mining based on the joint features carrying position encoding through the multi-head attention module in the trained acoustic model and outputting the Mel spectrum includes the following steps:
[0071] Step 1: Extract the joint features carrying the position code through the first multi-head attention layer in the trained acoustic model, and output the feature extraction results through the first fully connected layer.
[0072] Step 2: Using the length adjuster in the trained acoustic model and the preset duration model, perform feature expansion on the feature extraction result along the time dimension to obtain an extended feature set.
[0073] Step 3: Perform a linear transformation based on the extended feature set through the second multi-head attention layer in the trained acoustic model to output a Mel spectrum.
[0074] In this embodiment, a multi-layer attention extraction layer is configured in the first multi-head attention module, and the extracted feature content is output as a feature extraction result through the first fully connected layer with an activation function. The feature extraction result is passed to the duration adjuster and the preset duration model. Here, the preset duration model is used to copy and expand the feature extraction result, that is, to predict the duration corresponding to each frame of feature extraction result. For example, when the predicted duration of a certain frame feature extraction result is 2, the feature extraction result of the frame is copied to obtain two consecutive frames of feature extraction results.
[0075] It should be noted that, since the preset time length model replicates and expands the feature extraction results, the feature extraction results of each frame are expanded, and the overall feature extraction results are correspondingly enlarged. Therefore, it is necessary to use the length adjuster to coordinate the expansion of the time dimension.
[0076] It can be understood that the extended feature set is a matrix in a high-dimensional space, so it is necessary to perform a linear transformation based on the extended feature set through the second multi-head attention layer in the trained acoustic model to output the Mel spectrum.
[0077] As an embodiment, the above step of: performing a linear transformation based on the extended feature set through the second multi-head attention layer in the trained acoustic model and outputting a Mel spectrum includes the following steps:
[0078] Step 1: splice the position code for each frame of extended features in the extended feature set to obtain an extended feature set carrying the position code.
[0079] Step 2: Perform a nonlinear transformation on the extended feature set carrying the position code through the second multi-head attention layer in the trained acoustic model, and output the nonlinear transformation result through the second fully connected layer.
[0080] Step 3: Perform residual calculation on the nonlinear transformation result through the convolution layer in the trained acoustic model, and perform regression prediction based on the residual calculation result through the linear layer to output the Mel spectrum.
[0081] In this embodiment, since the extended feature set is obtained by feature expansion along the time dimension based on the feature extraction result, it is necessary to re-position-code the content of each frame in the extended feature set. Here, the position code is spliced for each frame of the extended feature in the extended feature set, which is the same as the position code for the joint feature. The position cosine value of each frame of the extended feature can be measured, and the measured value is used as the position code to obtain a control feature set carrying the position code.
[0082] Here, the second multi-head attention layer in the trained acoustic model performs a nonlinear transformation on the extended feature set with position encoding, and the nonlinear transformation result is output through the second fully connected layer. Similar to the first fully connected layer, the second fully connected layer also has an activation function, and the nonlinear transformation result is output after the extended feature set nonlinear transformation structure is connected. The convolution layer in the trained acoustic model performs convolution and residual calculation on the linear transformation result, and the linear layer performs regression prediction based on the residual calculation result, and outputs the Mel spectrum.
[0083] S13: Playing the speech content corresponding to the text sequence based on the Mel spectrum.
[0084] In step S13, the Mel spectrum is used to describe the sound features of the text sequence. Here, since the Mel spectrum is obtained by mining the context dependency of the joint features using the trained acoustic model, and the joint features are obtained by concatenating the text features and prosodic features of the text sequence, the Mel spectrum has both the expression of the contextual relationship of the text sequence and the prosodic association expression of the text contextual relationship.
[0085] In practical applications, since the obtained spectrogram is relatively large, in order to obtain the sound features of appropriate size, it is usually converted into a Mel-scale spectrum through a Mel-scale filter bank. Correspondingly, the audio phase spectrum can also be obtained through reverse prediction based on the Mel-scale spectrum, and the audio phase spectrum can be used as a complete mapping of the speech to play the speech content.
[0086] In the specific implementation, the speech content corresponding to the text sequence is played based on the Mel spectrum. Specifically, the audio phase spectrum of the Mel spectrum can be predicted by using a vocoder, that is, the Mel spectrum is converted into a corresponding audio phase spectrum, and then the audio phase spectrum is used for speech playback, that is, the speech content corresponding to the text sequence is played.
[0087] The above scheme uses the trained prosodic feature prediction network to perform prosody prediction based on the text features of the text sequence to obtain the prosodic features of the text sequence. Since the prosodic features can be used to characterize the acoustic features of the unit text content in the text sequence, the text features and the prosodic features are used as joint features and input into the trained acoustic model for context dependency mining, so that the output Mel spectrum can be used to characterize the acoustic features, context dependencies and pronunciation timing of the unit text content in the text sequence. Finally, the speech content corresponding to the text sequence is played based on the Mel spectrum, which can improve the speech quality in the scenario of converting text into speech.
[0088] In addition, due to the voice playback method provided in the embodiment of the present application, the voice quality can be improved in the scenario of converting text into voice, that is, the played voice content sounds better, thereby improving the user experience.
[0089] See also Figure 2 , which is a flowchart of a voice playback method provided in another embodiment of the present application. Figure 2 The illustrated embodiment, with Figure 1 The difference between the illustrated embodiments is that before step S11, steps S21 to S22 are also included, which are described in detail as follows:
[0090] S21: Analyze and process the text sequence sample and the speech sample corresponding to the text sequence sample respectively to obtain mutually related text sample features and speech sample features.
[0091] S22: Using the text sample features and the speech sample features as first training samples, training a prosody feature prediction network to obtain a trained prosody feature prediction network.
[0092] In this embodiment, the text sequence sample and the speech sample correspond to each other, that is, the speech content of the speech sample is the same as the content of the text sequence. In other words, the speech content in the speech sample is recorded when a person reads the text sequence. Based on this, the prosodic features in the speech sample are the output targets of the prosodic feature prediction network.
[0093] It should be noted that the text sequence sample and the speech sample corresponding to the text sequence sample are analyzed and processed separately to obtain the interrelated text sample features and speech sample features, which can be achieved through two different analysis modules. Here, the analysis module may include a deep neural network, which is used to encode the text sequence, and then the encoding result of the text sequence is extracted with the long short-term memory network to obtain the text sample features and speech sample features respectively. Since the encoding result of the text sequence is obtained by encoding the text sequence with a deep neural network, the encoding result of the text sequence can be directly extracted with the long short-term memory network to extract the time information and context dependency.
[0094] It is easy to understand that, unlike training the acoustic model, when the prosodic feature prediction network is trained using the first training sample, the text sample features serve as the input of the prosodic feature prediction network, and the speech sample features serve as the target output of the prosodic feature prediction network, so that the trained prosodic feature prediction network can be used to characterize the correspondence between text features and prosodic features.
[0095] See also Figure 3 , which is a flowchart of a voice playback method provided in yet another embodiment of the present application. Figure 3 The embodiment shown in Figure 2 Based on the embodiment shown, as an embodiment, after step S21 and before step S12, step S31 is further included, which is described in detail as follows:
[0096] S31: Using the text sample features and the speech sample features as second training samples, training the acoustic model to obtain a trained acoustic model.
[0097] In this embodiment, the text sequence sample and the voice sample correspond to each other, that is, the voice content of the voice sample is the same as the content of the text sequence. In other words, the voice content in the voice sample is recorded when a person reads the text sequence. Therefore, there is also a correlation between the text sample features and the voice sample features. Here, the voice sample features are the rhythm features of the voice sample. The second training sample is a spliced sample obtained by splicing the text sample features and the voice sample features.
[0098] Here, the text sample features and the speech sample features are concatenated by concatenating the vector of the text sample features and the vector of the speech sample features to obtain a two-dimensional vector, and then using the two-dimensional vector as a joint feature sample in the second training sample.
[0099] It should be noted that since the text sequence samples and the speech samples correspond to each other, that is, the speech content of the speech sample is the same as the content of the text sequence, the text sample features can be obtained from the text sequence samples, and the speech sample features, that is, the prosodic features of the speech sample, can be obtained from the speech samples. By splicing the text sample features with the speech sample features, the training samples can be obtained.
[0100] It is easy to understand that in practical applications, in order to improve the generalization of the trained acoustic model, this can be achieved by enriching the content of the second training sample. For example, different text sample features can be combined with corresponding speech sample features to enrich the content of the second training sample, so that the second training sample can be used to train the acoustic model, which can improve the Mel spectrum output efficiency of the trained acoustic model.
[0101] See also Figure 4 , is a structural block diagram of a voice playback device provided in an embodiment of the present application. In this embodiment, the voice playback device includes various units for executing Figures 1 to 3 For details, please refer to the steps in the corresponding embodiment. Figures 1 to 3 as well as Figures 1 to 3 For the convenience of explanation, only the parts related to this embodiment are shown. Figure 4 The speech playback device 40 includes: a prediction unit 41, a mining unit 42 and an output unit 43. Specifically:
[0102] The prediction unit 41 is used to use the trained prosodic feature prediction network to perform prosodic prediction according to the text features of the text sequence to obtain the prosodic features of the text sequence.
[0103] The mining unit 42 is used to use the text feature and the prosodic feature as joint features, input them into the trained acoustic model to perform context dependency mining, and output a mel spectrum.
[0104] The output unit 43 is configured to play the speech content corresponding to the text sequence based on the Mel spectrum.
[0105] As an embodiment, the voice playback device 40 further includes:
[0106] The analysis unit is used to analyze and process the text sequence samples and the speech samples corresponding to the text sequence samples respectively to obtain mutually related text sample features and speech sample features.
[0107] The first training unit is used to train the prosodic feature prediction network by using the text sample features and the speech sample features as first training samples to obtain a trained prosodic feature prediction network.
[0108] As an embodiment, the voice playback device 40 further includes:
[0109] The second training unit is used to train the acoustic model by using the text sample features and the speech sample features as second training samples to obtain a trained acoustic model.
[0110] As an embodiment, the prediction unit 41 includes:
[0111] The first feature acquisition unit is used to extract time information and context dependency based on the encoding result of the text sequence through a long short-term memory network to obtain text features of the text sequence; wherein the encoding result of the text sequence is obtained by encoding the text sequence using a deep neural network; and the text features carry phoneme information of the text sequence.
[0112] The prosody prediction unit is used to use the trained prosody feature prediction network to perform prosody prediction based on the phoneme information carried by the text features to obtain the prosody features of the text sequence.
[0113] As an embodiment, the mining unit 42 includes:
[0114] The second feature acquisition unit is used to splice the position code for the joint feature of each frame to obtain the joint feature carrying the position code.
[0115] The mel spectrum mining unit is used to perform context dependency mining based on the joint features carrying the position code through the multi-head attention module in the trained acoustic model, and output the mel spectrum.
[0116] As an embodiment, the mel spectrum mining unit includes:
[0117] A feature extraction unit is used to extract features of the joint features carrying position codes through the first multi-head attention layer in the trained acoustic model, and output feature extraction results through the first fully connected layer.
[0118] The feature acquisition unit is used to use the length adjuster in the trained acoustic model and the preset duration model to perform feature expansion on the feature extraction result along the time dimension to obtain an extended feature set.
[0119] The feature transformation unit is used to perform linear transformation based on the extended feature set through the second multi-head attention layer in the trained acoustic model to output a Mel spectrum.
[0120] As an embodiment, the feature transformation unit is specifically used for:
[0121] splicing the position code for each frame of extended features in the extended feature set to obtain an extended feature set carrying the position code;
[0122] Performing a nonlinear transformation on the extended feature set carrying the position code through a second multi-head attention layer in the trained acoustic model, and outputting the nonlinear transformation result through a second fully connected layer;
[0123] The convolution layer in the trained acoustic model performs residual calculation on the nonlinear transformation result, and the linear layer performs regression prediction based on the residual calculation result to output the Mel spectrum.
[0124] It should be understood that Figure 4 In the structural block diagram of the voice playback device shown in FIG. 1 , each unit is used to execute Figures 1 to 3 The steps in the corresponding embodiments, and for Figures 1 to 3 Each step in the corresponding embodiment has been explained in detail in the above embodiment. Figures 1 to 3 as well as Figures 1 to 3 The relevant descriptions in the corresponding embodiments are not repeated here.
[0125] See also Figure 5 , is a structural block diagram of a computer device provided in an embodiment of the present application. Figure 5 As shown, the computer device 50 of this embodiment includes: a processor 51, a memory 52, and a computer program 53 stored in the memory 52 and executable on the processor 51, such as a program of the voice playing method. When the processor 51 executes the computer program 53, the steps in each embodiment of the above-mentioned voice playing method are implemented, such as Figure 1 S11 to S13 as shown, or Figure 2 S21 to S13 shown, or Figure 3 The processor 51 executes the computer program 53 to implement the above-mentioned S21 to S13 and S31. Figure 4 The functions of each unit in the corresponding embodiment are, for example, Figure 4 For details on the functions of the units 41 to 43 shown, please refer to Figure 4 The relevant descriptions in the corresponding embodiments are not repeated here.
[0126] Exemplarily, the computer program 53 may be divided into one or more units, which are stored in the memory 52 and executed by the processor 51 to complete the present application. The one or more units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 53 in the computer device 50. For example, the computer program 53 may be divided into a prediction unit, a mining unit, and an output unit, and the specific functions of each unit are as described above.
[0127] The computer device 50 may include, but is not limited to, a processor 51 and a memory 52. Those skilled in the art will appreciate that Figure 5 It is only an example of the computer device 50 and does not constitute a limitation of the computer device 50. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device 50 may also include input and output devices, network access devices, buses, etc.
[0128] The processor 51 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0129] The memory 52 may be an internal storage unit of the computer device 50, such as a hard disk or memory of the computer device 50. The memory 52 may also be an external storage device of the computer device 50, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 50. Further, the memory 52 may also include both an internal storage unit of the computer device 50 and an external storage device. The memory 52 is used to store the computer program and other programs and data required by the computer device. The memory 52 may also be used to temporarily store data that has been output or is to be output.
[0130] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A voice playback method, It is characterized in that include: Using the trained prosodic feature prediction network, prosodic prediction is performed according to the text features of the text sequence to obtain the prosodic features of the text sequence; The text feature and the rhythm feature are used as joint features, input into the trained acoustic model to perform context dependency mining, and the Mel spectrum is output, including: Splicing the position code for the joint features of each frame to obtain the joint features carrying the position code; Extracting features of the joint features carrying the positional encoding through a first multi-head attention layer in the trained acoustic model, and outputting feature extraction results through a first fully connected layer; Using the length regulator in the trained acoustic model and the preset duration model, the feature extraction result is extended along the time dimension to obtain an extended feature set; splicing the position code for each frame of extended features in the extended feature set to obtain an extended feature set carrying the position code; Performing a nonlinear transformation on the extended feature set carrying the position code through a second multi-head attention layer in the trained acoustic model, and outputting the nonlinear transformation result through a second fully connected layer; Performing residual calculation on the nonlinear transformation result through the convolution layer in the trained acoustic model, and performing regression prediction according to the residual calculation result through the linear layer, and outputting the Mel spectrum; The speech content corresponding to the text sequence is played based on the Mel spectrum.
2. The voice playing method according to claim 1, It is characterized in that Before the step of using the trained prosodic feature prediction network to perform prosodic prediction according to the text features of the text sequence to obtain the prosodic features of the text sequence, the method further includes: Analyzing and processing the text sequence samples and the speech samples corresponding to the text sequence samples respectively to obtain mutually related text sample features and speech sample features; The text sample features and the speech sample features are used as first training samples to train a prosodic feature prediction network to obtain a trained prosodic feature prediction network.
3. The voice playing method according to claim 2, It is characterized in that Before the step of using the text feature and the prosodic feature as joint features, inputting the trained acoustic model to perform context dependency mining, and outputting the Mel spectrum, the method further includes: The text sample features and the speech sample features are used as second training samples to train the acoustic model to obtain a trained acoustic model.
4. The voice playing method according to any one of claims 1 to 3, It is characterized in that The method of using the trained prosodic feature prediction network to perform prosodic prediction according to the text features of the text sequence to obtain the prosodic features of the text sequence includes: Extracting time information and context dependency based on the encoding result of the text sequence through a long short-term memory network to obtain text features of the text sequence; wherein the encoding result of the text sequence is obtained by encoding the text sequence using a deep neural network; the text features carry the phoneme information of the text sequence; The trained prosodic feature prediction network is used to perform prosodic prediction based on the phoneme information carried by the text features to obtain the prosodic features of the text sequence.
5. A voice playback device, It is characterized in that include: A prediction unit, configured to use a trained prosodic feature prediction network to perform prosodic prediction according to text features of a text sequence to obtain prosodic features of the text sequence; A mining unit, used for taking the text feature and the prosodic feature as joint features, inputting them into a trained acoustic model to perform context dependency mining, and outputting a mel spectrum; An output unit, configured to play speech content corresponding to the text sequence based on the Mel spectrum; The excavation unit comprises: A second feature acquisition unit is used to concatenate the position code of the joint feature of each frame to obtain the joint feature carrying the position code; A feature extraction unit, configured to extract the joint features carrying the position code through a first multi-head attention layer in the trained acoustic model, and output the feature extraction results through a first fully connected layer; A feature acquisition unit, used to perform feature expansion on the feature extraction result along the time dimension by using the length adjuster in the trained acoustic model and the preset duration model to obtain an extended feature set; A feature transformation unit is used to concatenate position codes for each frame of extended features in the extended feature set to obtain an extended feature set carrying position codes; perform nonlinear transformation on the extended feature set carrying position codes through the second multi-head attention layer in the trained acoustic model, and output the nonlinear transformation result through the second fully connected layer; perform residual calculation on the nonlinear transformation result through the convolution layer in the trained acoustic model, and perform regression prediction based on the result of the residual calculation through the linear layer, and output a Mel spectrum.
6. A computer device, It is characterized in that The computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the steps of the method according to any one of claims 1 to 4 when executing the computer program.
7. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.