Hierarchical coding and decoding speech synthesis method, device, equipment and medium

Abstract

The invention relates to the technical field of speech synthesis, can be applied to business system platforms such as financial science and technology, medical health and the like, and discloses a layered coding and decoding speech synthesis method, device, equipment and medium, and the method comprises the following steps: carrying out semantic feature extraction on a pre-acquired original text to obtain text semantic features; performing prosodic feature extraction on the original text to obtain a text prosodic feature; performing feature fusion compression on the text semantic features and the text rhythm features to obtain fusion compression features; performing low-frequency decoding on the fusion compression features to obtain low-frequency decoding features; performing high-frequency decoding on the fusion compression feature to obtain a high-frequency decoding feature; and combining the low-frequency decoding features and the high-frequency decoding features, and converting the combined features into target synthetic speech. Through high and low frequency decoding, richer and more accurate voice details are restored, so that the synthesized voice is listened more naturally and smoothly, and the credibility and fluency of the voice are improved.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis technology, and in particular to a layered coding and decoding speech synthesis method, device, equipment and medium. Background Art

[0002] Speech synthesis technology, as a critical link in the field of artificial intelligence, is committed to converting text into natural and fluent speech output. It plays an indispensable role in many fields in today's digital age, especially in the fields of healthcare and financial technology, which are closely related to people's lives. Speech synthesis technology has greatly improved the efficiency and experience of information interaction. For example, in the healthcare field, with the help of speech synthesis technology, accurate and friendly voice navigation can be generated to guide patients through a series of processes such as registration, waiting, examination, and medication collection. This will greatly improve medical treatment efficiency and alleviate patients' anxiety when seeking medical treatment. In the field of financial technology, through speech synthesis, customer service systems can answer various questions such as account inquiries, financial product consultations, loan processing procedures, etc. in clear and friendly voices, providing 24-hour uninterrupted service, greatly improving customer service efficiency and response speed, and reducing labor costs.

[0003] However, existing end-to-end TTS models generate waveforms directly from text when generating high-quality speech, avoiding the intermediate processing of the pipeline structure. However, this approach has poor processing capabilities for irregular text input (such as punctuation errors and spelling problems), and is prone to speech interruptions or anomalies. When processing long sentences, speech distortion or information loss may occur, making it difficult to maintain the naturalness and fluency of the speech.

[0004] Therefore, in the face of the growing problem of high-quality speech synthesis, current speech synthesis methods urgently need to be improved to solve the problem that existing methods are difficult to maintain the naturalness and fluency of speech. Summary of the Invention

[0005] The present invention provides a layered coding and decoding speech synthesis method, device, equipment and medium, which restores richer and more accurate speech details through high and low frequency decoding, makes the synthesized speech sound more natural and smooth, and improves the credibility and fluency of the speech.

[0006] In a first aspect, a layered coding and decoding speech synthesis method is provided, comprising:

[0007] Extract semantic features from the pre-acquired original text to obtain text semantic features;

[0008] Extracting rhythmic features from the original text to obtain text rhythmic features;

[0009] Performing feature fusion compression on the text semantic features and the text prosodic features to obtain fused compressed features;

[0010] Performing low-frequency decoding on the fused compressed features to obtain low-frequency decoding features;

[0011] Performing high-frequency decoding on the fused compressed features to obtain high-frequency decoding features;

[0012] The low-frequency decoded features and the high-frequency decoded features are merged, and the merged features are converted into target synthesized speech.

[0013] In a second aspect, a layered coding and decoding speech synthesis device is provided, comprising:

[0014] The semantic extraction module is used to extract semantic features from the pre-acquired original text to obtain text semantic features;

[0015] A rhythm extraction module, configured to extract rhythm features from the original text to obtain text rhythm features;

[0016] A compression module, configured to perform feature fusion compression on the text semantic features and the text prosodic features to obtain fused compressed features;

[0017] A low-frequency decoding module, configured to perform low-frequency decoding on the fused compressed features to obtain low-frequency decoding features;

[0018] A high-frequency decoding module, configured to perform high-frequency decoding on the fused compressed features to obtain high-frequency decoding features;

[0019] The merging and conversion module is used to merge the low-frequency decoding features and the high-frequency decoding features, and convert the merged features into a target synthesized speech.

[0020] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned layered coding and decoding speech synthesis method are implemented.

[0021] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned layered coding and decoding speech synthesis method are implemented.

[0022] In the scheme implemented by the aforementioned layered encoding and decoding speech synthesis method, apparatus, computer device, and storage medium, high-dimensional text data can be converted into low-dimensional semantic feature vectors through semantic feature extraction. This not only reduces data storage space, but also reduces the computational complexity of subsequent processing, improving the training and inference efficiency of the model. Simultaneously, low-frequency and high-frequency decoding of the fused compressed features can better align with human perception and cognition of speech, while restoring detailed information lost due to compression to save space and bandwidth. Finally, by merging the low-frequency and high-frequency decoded features and converting them into synthesized speech, this information can be comprehensively utilized to restore richer and more accurate speech details, making the synthesized speech closer to real speech and sounding more natural and fluent, thereby improving the credibility and fluency of the speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 This is a schematic diagram of an application environment of a layered coding and decoding speech synthesis method according to an embodiment of the present invention;

[0025] Figure 2 This is a flowchart of a layered coding and decoding speech synthesis method according to an embodiment of the present invention;

[0026] Figure 3 This is a structural diagram of a layered coding and decoding speech synthesis device according to an embodiment of the present invention;

[0027] Figure 4 is a structural diagram of a computer device according to an embodiment of the present invention;

[0028] Figure 5 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0030] The embodiment of the present invention provides a layered coding and decoding speech synthesis method, which can be applied in the following situations: Figure 1 In an application environment, the client communicates with the server through a network. The server can extract semantic features from the original text obtained in advance to obtain text semantic features; extract prosodic features from the original text to obtain text prosodic features; perform feature fusion compression on the text semantic features and the text prosodic features to obtain fused compressed features; perform low-frequency decoding on the fused compressed features to obtain low-frequency decoding features; perform high-frequency decoding on the fused compressed features to obtain high-frequency decoding features; merge the low-frequency decoding features and the high-frequency decoding features, and convert the merged features into target synthesized speech, and feed the target synthesized speech back to the client. The present invention provides a layered coding and decoding speech synthesis device, which, for the target synthesized speech service, restores richer and more accurate speech details through high and low frequency decoding, making the synthesized speech sound more natural and smooth, and improving the credibility and fluency of the speech. The client can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0031] See also Figure 2 As shown, Figure 2 A schematic flow chart of a layered coding and decoding speech synthesis method provided in an embodiment of the present invention includes the following steps:

[0032] S1. Extract semantic features from the original text obtained in advance to obtain text semantic features.

[0033] In the embodiment of the present invention, the semantic feature extraction refers to the process of extracting key features that can represent the semantic information of the text from the pre-acquired original text.

[0034] Specifically, the original text is a natural language text used by humans, and it is difficult for computers to directly understand its meaning. Semantic feature extraction is to convert these texts into a form that computers can process and understand, and use a set of features to represent the semantic content of the text.

[0035] In specific medical and health scenarios, these texts can be converted into structured information through semantic feature extraction. For example, features such as symptom type (such as fever, cough), severity (such as mild, severe) and other features can be extracted to assist doctors in disease diagnosis and treatment plan formulation.

[0036] In the FinTech scenario, through semantic feature extraction, features related to customer credit can be extracted from these texts, such as income level, job stability, debt status, etc. These features can be used as inputs to risk assessment models to help financial institutions more accurately assess customer credit risks and make reasonable credit decisions.

[0037] In the embodiment of the present invention, extracting semantic features from the pre-acquired original text to obtain text semantic features includes:

[0038] Perform word segmentation processing on the pre-acquired original text to obtain word segmentation text, and convert the word segmentation text into word segmentation low-dimensional vectors one by one;

[0039] Performing forward and backward sequence splicing processing on the word segmentation low-dimensional vectors through a bidirectional long short-term memory network to obtain a splicing vector;

[0040] The attention weight of each splicing vector is calculated, and the splicing vector and the attention weight of the splicing vector are weighted summed to obtain the text semantic feature.

[0041] In an embodiment of the present invention, the word segmentation processing refers to dividing the pre-acquired original text into individual words or phrases according to certain rules and algorithms. The forward and backward sequence splicing processing refers to using a bidirectional LSTM (bidirectional long short-term memory network) to process the word segmentation low-dimensional vector, learning and encoding the sequence information from the forward and reverse directions respectively, and then splicing the results obtained from these two directions to obtain a splicing vector that more comprehensively contains text context information. The calculation refers to measuring the importance of each splicing vector in representing the semantics of the text.

[0042] Specifically, the data is first cleaned, which includes removing noise, missing values and outliers from the pre-acquired original text; then some dictionary-based methods are used, that is, using the constructed dictionary to match the words in the text. For example, starting from the beginning of the text, each character combination is compared with the words in the dictionary in turn, and the matching words are used as a segmented word; there are also some statistical-based methods, which analyze the frequency of occurrence and contextual information of words in a large amount of text to determine which character combinations are more likely to be a word. For example, if "beautiful" and "flowers" often appear together, they will be identified as two different segmented words.

[0043] Specifically, for a given sequence of low-dimensional word segmentation vectors, the forward LSTM starts at the beginning of the sequence and inputs each low-dimensional word segmentation vector into the network in turn. When processing each vector, it updates the hidden state based on the current input and the previous hidden state, thereby remembering the previous information. The reverse LSTM starts at the end of the sequence and processes each low-dimensional word segmentation vector in reverse order. It also updates the hidden state based on the current input and the previous (in reverse order) hidden state, thereby remembering the subsequent information. Finally, the hidden state obtained at each position by the forward LSTM is concatenated with the hidden state obtained at the corresponding position by the reverse LSTM to obtain the concatenated vector.

[0044] Furthermore, the attention mechanism model is used to calculate the attention weight of the splicing vector. By taking the splicing vector as input and performing some linear transformations, activation functions and other operations, a score representing the importance of each splicing vector is calculated; then, these scores are normalized so that their sum is 1, thus obtaining the attention weight of each splicing vector.

[0045] In the embodiment of the present invention, the bidirectional LSTM can better handle dependencies in long sequences through a two-directional memory mechanism, remember the long-term semantic associations between different positions in the text, and adaptively highlight the key parts related to the semantics of the current text by calculating the attention weights, while ignoring irrelevant or minor information.

[0046] In the embodiment of the present invention, high-dimensional text data can be converted into low-dimensional semantic feature vectors through semantic feature extraction; this not only reduces the storage space of the data, but also reduces the computational complexity of subsequent processing, and improves the training and reasoning efficiency of the model.

[0047] S2. Extracting rhythmic features from the original text to obtain text rhythmic features.

[0048] In the embodiment of the present invention, the prosodic feature extraction refers to analyzing and extracting information related to these speech characteristics from the text.

[0049] Specifically, semantic features reflecting the meaning of the text and prosodic features reflecting the prosodic characteristics of the speech are extracted from the original text.

[0050] In the specific scenario of medical health, by analyzing the fusion information of the rhythmic characteristics and semantic characteristics of the doctor-patient dialogue, the quality and effectiveness of the doctor-patient communication can be evaluated; it helps to discover problems in communication, adjust the communication method in time, and improve patients' treatment compliance and satisfaction.

[0051] At the same time, by integrating and analyzing the rhythmic and semantic features of a large number of customer consultation texts, the intelligent customer service system can better understand customer intentions and emotions and provide more personalized and accurate services.

[0052] In the embodiment of the present invention, extracting the prosodic features of the original text to obtain the prosodic features of the text includes:

[0053] Performing dilated convolution processing on the original text to obtain convolution features;

[0054] Linearly projecting the convolution feature onto a preset subspace to obtain a feature projection corresponding to the subspace;

[0055] Calculate the scaled dot product attention of the feature projections one by one, and concatenate the attention results into a concatenated feature sequence;

[0056] The concatenated feature sequences are subjected to nonlinear transformation and fusion to obtain text prosodic features.

[0057] In an embodiment of the present invention, the atrous convolution processing refers to skipping some elements at a certain interval when operating on each element in the original text and convolving the elements in the original text. The linear projection refers to mapping the convolution feature from the original feature space to a preset subspace. The calculation refers to calculating the similarity between feature projections to determine the importance weight of each feature projection. The nonlinear transformation fusion refers to performing nonlinear transformation and fusion on the calculated scaled dot product attention results to further extract the complex relationship between the features and combine them into a feature sequence that can better represent the rhythmic features of the text.

[0058] Specifically, suppose we have an original text sequence, which is represented as a one-dimensional vector sequence. A convolution kernel with a void rate of 2 is calculated every other element during each slide, so that text information at a longer distance can be captured. Let the convolution feature be the matrix X, and the preset subspace consists of a set of basis vectors {v1,v2,...v n Linear projection is to calculate the projection of convolution feature X on these basis vectors. For each feature vector x in X, its projection y on the subspace can be calculated by We get , where · represents the dot product of the vectors.

[0059] Specifically, each feature projection is mapped into two different spaces, resulting in a query vector and a key vector, typically through a linear transformation. The dot product between the query and key vectors is then calculated, yielding a score matrix representing the similarity between them. This score matrix is then converted into a probability distribution using a softmax function, yielding an attention weight for each feature projection relative to the other feature projections. The attention weights are then multiplied by the value vector and summed to yield the scaled dot product attention result for each feature projection.

[0060] Furthermore, for each scaled dot product attention result a i , transformed by a nonlinear activation function f, and obtained b i =f(a i ); Then, all the results after nonlinear transformation {b1,b2,...b m} are fused, and the fusion method can be simple concatenation, summation, or combination through a fully connected layer.

[0061] In an embodiment of the present invention, prosodic features are extracted from the original text to extract prosodic features such as stress, intonation, pauses, and rhythm in the text, so that a speech synthesis system (such as TTS) can generate speech that is closer to real-person pronunciation and avoids a mechanical feel.

[0062] S3. Perform feature fusion compression on the text semantic features and the text prosodic features to obtain fused compressed features.

[0063] In the embodiment of the present invention, the feature fusion compression refers to merging text semantic features and text prosodic features, and compressing the merged features to obtain more concise and representative fusion compression features.

[0064] Specifically, the text semantic features and text prosodic features are merged, and the merged features are compressed, ultimately obtaining a fused compressed feature that combines semantic and prosodic information and is more compact and efficient.

[0065] In the embodiment of the present invention, the step of fusing and compressing the text semantic features and the text prosodic features to obtain fused compressed features includes:

[0066] Performing feature alignment on the text semantic features and the text prosodic features to obtain an alignment feature;

[0067] Performing gated fusion on the alignment features to obtain fusion features;

[0068] Feature compression is performed on the fused features to obtain fused compressed features.

[0069] In an embodiment of the present invention, the feature alignment refers to matching and adjusting text semantic features and text prosodic features in feature dimensions or feature spaces so that they are consistent and comparable to a certain extent. The gated fusion refers to controlling the weights and contributions of text semantic features and text prosodic features in the fusion process through a gating mechanism, thereby more flexibly fusing the two features. The feature compression refers to processing the fused features to reduce the dimension or data volume of the features while retaining their key information as much as possible.

[0070] Specifically, semantic features and prosodic features of text may have different dimensions, distributions, or representations. First, we need to analyze these two features and determine their correspondence. This can be achieved by using feature engineering techniques, such as linear transformations, mapping functions, or deep learning layers (such as fully connected layers and convolutional layers) to adjust the feature dimensions and distribution.

[0071] Furthermore, the aligned semantic features and prosodic features are taken as input and respectively subjected to a linear transformation (which can be a fully connected layer) to obtain two intermediate representations S and R, which correspond to the transformed results of the semantic features and prosodic features respectively. Then, a gating function is used to calculate the gating value g, which ranges from [0 to 1]. The gating value g is used to control the fusion ratio of the semantic features and prosodic features. The final fusion feature F can be obtained by Get, among them Represents element-wise multiplication.

[0072] Furthermore, the covariance matrix of the fused features is calculated, and then the eigenvalues and eigenvectors of the covariance matrix are obtained. The eigenvectors with larger eigenvalues are selected as principal components, and the fused features are projected onto these principal components to achieve feature compression. For example, a convolution layer with a convolution kernel size of 1*1 is used to perform a convolution operation on the fused features, and the dimension of the output features is controlled by adjusting the number of convolution kernels to achieve feature compression.

[0073] In an embodiment of the present invention, calculating the scaled dot product attention can measure the importance of each feature projection, allowing the model to automatically focus on the feature parts that are more relevant to the text rhythm and suppress unimportant information. Through the action of nonlinear functions, the attention information of different feature projections is fused and transformed, thereby extracting more advanced features that can more accurately represent the text rhythm.

[0074] In the embodiment of the present invention, semantic and prosodic features may partially overlap. Compression can remove redundancy and reduce model complexity. At the same time, the compression process forces the model to learn more essential feature representations, thereby improving adaptability to unseen texts.

[0075] S4. Perform low-frequency decoding on the fused compressed features to obtain low-frequency decoding features.

[0076] In the embodiment of the present invention, the low-frequency decoding refers to the operation of mapping the fused compressed features to the low-frequency feature space, which is intended to extract information related to the low-frequency characteristics of speech.

[0077] Specifically, speech signals contain different frequency components in the frequency domain. Low-frequency components are often associated with important speech features, such as fundamental pitch and formant structure. These features play a key role in speech perception and understanding. Low-frequency decoding extracts information corresponding to these low-frequency speech characteristics from the fused compressed features and converts them into a representation more suitable for describing speech low-frequency characteristics.

[0078] In healthcare scenarios, doctors and nurses rely on voice commands to operate medical equipment, record patient records, and more. Low-frequency decoding features can better capture key information in speech, improve speech recognition accuracy, and reduce medical accidents caused by speech recognition errors.

[0079] In financial scenarios, customers can interact with intelligent customer service or financial applications through voice. Low-frequency decoding features help generate more natural and friendly voice responses, improving the customer interaction experience and increasing customer satisfaction and loyalty to financial services.

[0080] In the embodiment of the present invention, performing low-frequency decoding on the fused compressed features to obtain low-frequency decoding features includes:

[0081] Performing a duration analysis on the segmented text to obtain a duration analysis result corresponding to each segmented text;

[0082] Performing feature expansion on the fused compressed feature according to the duration analysis result to obtain an expanded feature;

[0083] Performing context aggregation on the extended features to obtain aggregated features;

[0084] Low-frequency extraction is performed on the aggregated features to obtain low-frequency decoding features.

[0085] In an embodiment of the present invention, the duration analysis refers to the analysis and statistics of the speech duration corresponding to each segmented word in the segmented text, the feature expansion refers to the process of integrating the duration analysis results into the fused compressed features to increase the dimension and information content of the features, the context aggregation refers to considering the information of the extended features in the context, and aggregating the extended features in adjacent or related positions to capture the contextual dependencies in the speech, and the low-frequency extraction refers to extracting the feature information of the low-frequency part from the aggregated features.

[0086] Specifically, with the help of a pre-trained speech model, the segmented text is aligned with the corresponding speech data to determine the start and end time of each segmented word in the speech. For example, for the segmented text "I / like / apple", the time intervals of "I", "like" and "apple" in the speech are found through speech alignment, and their durations are calculated respectively, assuming they are 2s, 3s, and 2.5s. At the same time, the average duration can be calculated as (2+3+2.5) / 3≈2.5s, and the standard deviation is used to measure the discreteness of the duration, etc.

[0087] Furthermore, the duration analysis results are appropriately encoded. For example, the duration of each word segment is converted into a numerical vector. If the duration ranges from 0 to 5 seconds, it can be divided into five equal intervals. Each interval corresponds to a dimension of the vector. The value of the dimension in which the duration falls is set to 1, and the remaining dimensions are set to 0. The encoded duration vector is then concatenated with the fused compressed feature. Assuming that the fused compressed feature is a 10-dimensional vector and the duration encoding vector is 5-dimensional, the resulting expanded feature is a 15-dimensional vector, thus expanding the feature to include information related to speech duration.

[0088] Furthermore, a sliding window approach can be used, for example, setting a window size of 3. For each extended feature at a position, it is combined with the extended features at the preceding and following positions. Assuming the extended feature sequence is [E1, E2, E3, E4, E5] and the window size is 3, then during the first aggregation, E1, E2, and E3 are aggregated, and a new aggregated feature C1 can be obtained by averaging, summing, or using a neural network for fusion. The window is then slid right one position, and E2, E3, and E4 are aggregated to obtain C2. This is repeated in this way, resulting in a series of aggregated features. These aggregated features incorporate contextual information and can better reflect the overall characteristics of speech at different positions.

[0089] Furthermore, the aggregated features are regarded as a signal sequence. By designing a suitable low-pass filter and setting the cutoff frequency, for example, if the cutoff frequency is set to 10 Hz, the part of the aggregated features with a frequency higher than 10 Hz will be weakened or removed, and the low-frequency decoding features will be obtained.

[0090] In an embodiment of the present invention, by aggregating context information, the characteristics of co-pronunciation can be better captured, thereby improving the recognition accuracy of the word. Using a low-pass filter to filter out high-frequency components during the low-frequency extraction process can reduce the impact of noise, make the characteristics cleaner and clearer, and help improve the accuracy of subsequent processing.

[0091] In the embodiment of the present invention, the fused compressed features are low-frequency decoded, and the obtained low-frequency decoded features are more consistent with the physiological and psychological characteristics of the human auditory system and can better match the human perception and cognition of speech.

[0092] S5. Perform high-frequency decoding on the fused compressed features to obtain high-frequency decoding features.

[0093] In the embodiment of the present invention, the high-frequency decoding refers to the process of extracting feature components related to high-frequency information from the fused compressed features.

[0094] Specifically, the feature vector obtained after a certain processing of the fused compressed features contains information related to the different frequency components of the speech signal; and high-frequency decoding is to specifically analyze and extract the high-frequency part to obtain high-frequency decoding features that can represent the high-frequency characteristics of the speech.

[0095] In the specific scenario of medical health, the respiratory sounds of asthma patients may have specific wheezing characteristics in the high-frequency band, and the high-frequency cough sounds of patients with chronic obstructive pulmonary disease may also have their own unique manifestations, which can help doctors judge the condition more accurately.

[0096] In the FinTech scenario, the intelligent customer service system of financial institutions can understand customers' questions and intentions more accurately by analyzing the high-frequency decoding features of customer consultation voices.

[0097] In the embodiment of the present invention, performing high-frequency decoding on the fused compressed features to obtain high-frequency decoding features includes:

[0098] Performing residual analysis on the fused compressed features to obtain feature residuals;

[0099] Performing residual enhancement on the feature residual to obtain an enhanced residual;

[0100] Reconstructing high-frequency features of the fused compressed features according to the enhanced residual to obtain initial high-frequency decoding features;

[0101] The initial high-frequency decoding feature and the preset true high-frequency decoding feature are subjected to spectrum consistency loss optimization to obtain a high-frequency decoding feature.

[0102] In an embodiment of the present invention, the residual analysis refers to analyzing the fused compressed features to find the differences between them and a certain expected state or ideal model to obtain feature residuals. The residual enhancement refers to processing the obtained feature residuals to highlight or amplify the useful information in the residuals and suppress noise or irrelevant information. The high-frequency feature reconstruction refers to reconstructing the high-frequency features based on the enhanced residuals, combined with the fused compressed features or other relevant information, to restore or extract the feature information in the high-frequency part of the original data to obtain the initial high-frequency decoding features. The spectrum consistency loss refers to comparing the difference in spectrum between the initial high-frequency decoding features and the preset true high-frequency decoding features, and minimizing this difference through an optimization algorithm, thereby obtaining more accurate and more practical high-frequency decoding features.

[0103] Specifically, the fused compressed features are input into a predefined model or reference framework. The model can be an autoencoder based on deep learning, a generative adversarial network, etc., or a simple mathematical model. The model will try to "restore" or "predict" the fused compressed features to obtain an output result. Then, the original fused compressed features are subtracted from the output result element by element, and the difference is the feature residual.

[0104] In detail, some traditional signal enhancement methods are used, such as performing wavelet transform on the residual, adjusting the coefficients in the wavelet domain, enhancing the coefficients of the high-frequency part, and then obtaining the enhanced residual through inverse wavelet transform.

[0105] Furthermore, the enhanced residual is element-wise added to the fused compressed features to produce a new feature vector. This new feature vector is then fed into a specially designed high-frequency feature generation model, which can be a deconvolutional network or other structure capable of generating high-frequency features from low-dimensional features. Through a series of deconvolution layers and activation functions, the deconvolutional network gradually increases the dimensionality and resolution of the feature vector, restoring the structure and details of the high-frequency features, ultimately yielding the initial high-frequency decoded features.

[0106] Furthermore, the spectra of the initial high-frequency decoding features and the true high-frequency decoding features are calculated. This can be achieved by converting the features from the time domain to the frequency domain through methods such as the fast Fourier transform (FFT) to obtain their spectral representations. Then, a spectrum consistency loss function is defined. Common loss functions include the mean square error (MSE) loss and the mean absolute error (MAE) loss, which are used to measure the difference between the two spectra. For example, the mean square error loss function is used to calculate the average of the squared difference of the amplitudes at the corresponding frequency points of the two spectra. Finally, an optimization algorithm, such as stochastic gradient descent (SGD), Adagrad, Adadelta, etc., is used to minimize the loss function. In each iteration, the parameters of the initial high-frequency decoding features are updated according to the gradient of the loss function, so that the value of the loss function gradually decreases until the preset convergence condition is reached. At this time, the initial high-frequency decoding features obtained after optimization are the final high-frequency decoding features.

[0107] In an embodiment of the present invention, by combining the enhanced residual with the fused compressed features to reconstruct high-frequency features, the detailed information in the residual can be used to restore the high-frequency features that were originally lost or weakened during compression, thereby obtaining the initial high-frequency decoding features containing more high-frequency details, providing a basis for subsequent more accurate analysis and processing of data.

[0108] In the embodiment of the present invention, high-frequency decoding is used to recover detailed information lost during compression to save space and bandwidth, thereby achieving efficient data compression and ensuring the availability and accuracy of data in medical diagnosis.

[0109] S6. Merge the low-frequency decoding features and the high-frequency decoding features, and convert the merged features into target synthesized speech.

[0110] In an embodiment of the present invention, the merging refers to combining low-frequency decoding features with high-frequency decoding features in the feature space or time dimension to form a joint feature representation containing complete frequency domain information, and the conversion refers to the process of generating target synthetic speech by using a specific algorithm or model for low-frequency decoding features and high-frequency decoding features.

[0111] Specifically, low-frequency and high-frequency decoding features represent features obtained by decoding speech signals after certain processing (such as encoding and compression). They contain information about different aspects of speech. For example, low-frequency features may be related to the fundamental frequency and formants of speech, determining the general outline and timbre of the speech; high-frequency features are related to speech details and harmonics, affecting the clarity and sound quality of the speech. By inputting these features into the speech synthesis system, the system will map them into the actual speech waveform according to preset rules and algorithms, thereby generating the target synthesized speech.

[0112] In the embodiment of the present invention, merging the low-frequency decoding features and the high-frequency decoding features, and converting the merged features into a target synthesized speech, includes:

[0113] Merging the low-frequency decoding features and the high-frequency decoding features to obtain a merged feature;

[0114] Inputting the combined features into a preset vocoder model to generate a preliminary waveform signal;

[0115] Performing spectrum analysis on the preliminary waveform signal, and filtering the waveform signal according to the spectrum analysis result to obtain an optimized waveform signal;

[0116] The optimized waveform signal is amplitude-adjusted and phase-corrected to obtain target synthesized speech.

[0117] In an embodiment of the present invention, the feature alignment refers to combining low-frequency decoding features and high-frequency decoding features to form a feature vector or feature matrix containing more comprehensive information. The generation refers to the process of converting the merged frequency domain features (such as Mel spectrum, linear spectrum or residual features) into a time domain speech waveform. The spectrum analysis refers to converting the time domain waveform into a frequency domain representation and analyzing the energy distribution of each frequency component to detect anomalies in the waveform (such as noise, harmonic distortion) or optimize synthesis parameters. The filtering processing refers to removing noise, interference or unnecessary frequency components in the signal to improve the quality and clarity of the speech. The amplitude adjustment and phase correction refers to modifying the amplitude (energy) distribution of the frequency domain or time domain signal while correcting the phase information of the waveform to optimize the time domain envelope and formant transition smoothness of the speech.

[0118] Specifically, merging is usually performed based on the representation of the features. If the low-frequency decoding features and the high-frequency decoding features are in vector form, they may be directly concatenated together in sequence to form a longer vector. If the features are in matrix form, they may be merged based on the dimension of the matrix and the logical relationship of the features, such as concatenation by columns or rows.

[0119] Furthermore, the merged features are used as input and passed through a series of computational layers in the model, such as convolutional layers and recurrent neural network layers, to gradually generate the corresponding initial waveform signal. For example, the WaveNet model gradually generates audio sample points by performing multiple layers of causal convolution operations on the input features, ultimately forming a complete initial waveform signal.

[0120] Furthermore, the waveform is divided into short time windows (such as 25ms), and the signal in each window is Fourier transformed to obtain a spectrum diagram (the horizontal axis is time, the vertical axis is frequency, and the color represents energy). The spectrum diagram is Mel-filtered to convert the linear frequency axis into a Mel frequency axis (simulating the auditory characteristics of the human ear), compressing high-frequency information and highlighting low-frequency key features. When adjusting the amplitude, the frequency domain amplitude is adjusted first to retain a specific frequency range (such as the main energy of speech is concentrated in 200-4000Hz), attenuating high-frequency noise or low-frequency redundant energy, and then adjusting the amplitude according to the spectrum. Based on the analysis results, the amplitude of specific frequency bands (such as the formant area) is increased or decreased to correct the timbre deviation of the synthesized speech (such as too bright or too dull). Then, time domain amplitude adjustment is performed, specifically adjusting the peak amplitude of the waveform to a preset range (such as [-1,1]) to avoid digital signal clipping distortion and compress excessive volume differences in the speech. Finally, phase spectrum correction is performed to compensate for the phase delay introduced during vocoder generation, ensuring the phase consistency of each frequency component. Interpolation or filtering (such as Gaussian smoothing) is used to reduce the mutation of the phase spectrum and make the phase transition of adjacent time windows more natural.

[0121] At the same time, various digital filters are used to achieve this, such as low-pass filters, high-pass filters, band-pass filters, etc., and appropriate filter types and parameters are selected according to specific needs; in this solution, a low-pass filter can be used to remove high-frequency noise, which allows low-frequency signals to pass through and attenuates high-frequency signals; each sample point of the waveform signal is calculated through the filter algorithm, and the signal is weighted and summed according to the filter coefficients to obtain the filtered waveform signal.

[0122] In this embodiment of the present invention, low-frequency decoded features contain information such as the basic outline and timbre of speech, while high-frequency decoded features focus on speech details, such as subtle variations in pronunciation and the harmonic structure of speech. By converting both into synthesized speech, this information can be comprehensively utilized to restore richer and more accurate speech details, making the synthesized speech closer to real speech and sounding more natural and fluent, thereby improving the credibility and fluency of the speech.

[0123] It can be seen that in the above scheme, for the target synthetic speech service, semantic features are extracted from the pre-acquired original text to obtain text semantic features; rhythmic features are extracted from the original text to obtain text rhythmic features; the text semantic features and the text rhythmic features are fused and compressed to obtain fused compressed features; the fused compressed features are low-frequency decoded to obtain low-frequency decoding features; the fused compressed features are high-frequency decoded to obtain high-frequency decoding features; the low-frequency decoding features and the high-frequency decoding features are merged, and the merged features are converted into target synthetic speech. Through high- and low-frequency decoding, richer and more accurate speech details are restored, making the synthetic speech sound more natural and smooth, thereby improving the credibility and fluency of the speech.

[0124] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0125] In one embodiment, a layered coding and decoding speech synthesis device is provided, and the layered coding and decoding speech synthesis device corresponds one-to-one to a layered coding and decoding speech synthesis method in the above embodiment. Figure 3 As shown, the layered coding and decoding speech synthesis device includes a semantic extraction module 101, a prosody extraction module 102, a compression module 103, a low-frequency decoding module 104, a high-frequency decoding module 105, and a merging and conversion module 106. The functional modules are described in detail as follows:

[0126] Semantic extraction module 101, used to extract semantic features from the pre-acquired original text to obtain text semantic features;

[0127] A rhythm extraction module 102 is used to extract rhythm features from the original text to obtain text rhythm features;

[0128] A compression module 103 is configured to perform feature fusion compression on the text semantic features and the text prosodic features to obtain fused compressed features;

[0129] A low-frequency decoding module 104 is configured to perform low-frequency decoding on the fused compressed features to obtain low-frequency decoding features;

[0130] A high frequency decoding module 105 is configured to perform high frequency decoding on the fused compressed features to obtain high frequency decoding features;

[0131] The merging and conversion module 106 is configured to merge the low-frequency decoding features and the high-frequency decoding features, and convert the merged features into a target synthesized speech.

[0132] In one embodiment, when the semantic extraction module 101 extracts semantic features from the pre-acquired original text to obtain text semantic features, it is used to:

[0133] Perform word segmentation processing on the pre-acquired original text to obtain word segmentation text, and convert the word segmentation text into word segmentation low-dimensional vectors one by one;

[0134] Performing forward and backward sequence splicing processing on the word segmentation low-dimensional vectors through a bidirectional long short-term memory network to obtain a splicing vector;

[0135] The attention weight of each splicing vector is calculated, and the splicing vector and the attention weight of the splicing vector are weighted summed to obtain the text semantic feature.

[0136] In one embodiment, when the prosody extraction module 102 extracts prosody features from the original text to obtain the text prosody features, it is configured to:

[0137] Performing dilated convolution processing on the original text to obtain convolution features;

[0138] Linearly projecting the convolution feature onto a preset subspace to obtain a feature projection corresponding to the subspace;

[0139] Calculate the scaled dot product attention of the feature projections one by one, and concatenate the attention results into a concatenated feature sequence;

[0140] The concatenated feature sequences are subjected to nonlinear transformation and fusion to obtain text prosodic features.

[0141] In one embodiment, when the compression module 103 performs feature fusion compression on the text semantic features and the text prosodic features to obtain fused compressed features, it is configured to:

[0142] Performing feature alignment on the text semantic features and the text prosodic features to obtain an alignment feature;

[0143] Performing gated fusion on the alignment features to obtain fusion features;

[0144] Feature compression is performed on the fused features to obtain fused compressed features.

[0145] In one embodiment, when the low-frequency decoding module 104 performs low-frequency decoding on the fused compressed features to obtain low-frequency decoding features, it is configured to:

[0146] Performing a duration analysis on the segmented text to obtain a duration analysis result corresponding to each segmented text;

[0147] Performing feature expansion on the fused compressed feature according to the duration analysis result to obtain an expanded feature;

[0148] Performing context aggregation on the extended features to obtain aggregated features;

[0149] Low-frequency extraction is performed on the aggregated features to obtain low-frequency decoding features.

[0150] In one embodiment, when the high frequency decoding module 105 performs high frequency decoding on the fused compressed features to obtain high frequency decoding features, it is configured to:

[0151] Performing residual analysis on the fused compressed features to obtain feature residuals;

[0152] Performing residual enhancement on the feature residual to obtain an enhanced residual;

[0153] Reconstructing high-frequency features of the fused compressed features according to the enhanced residual to obtain initial high-frequency decoding features;

[0154] The initial high-frequency decoding feature and the preset true high-frequency decoding feature are subjected to spectrum consistency loss optimization to obtain a high-frequency decoding feature.

[0155] In one embodiment, when the conversion module 106 combines the low-frequency decoding features and the high-frequency decoding features and converts the combined features into the target synthesized speech, it is configured to:

[0156] Merging the low-frequency decoding features and the high-frequency decoding features to obtain a merged feature;

[0157] Inputting the combined features into a preset vocoder model to generate a preliminary waveform signal;

[0158] Performing spectrum analysis on the preliminary waveform signal, and filtering the waveform signal according to the spectrum analysis result to obtain an optimized waveform signal;

[0159] The optimized waveform signal is amplitude-adjusted and phase-corrected to obtain target synthesized speech.

[0160] The present invention provides a layered coding and decoding speech synthesis device. For a target synthesized speech service, semantic features are extracted from a pre-acquired original text to obtain text semantic features; prosodic features are extracted from the original text to obtain text prosodic features; the text semantic features and the text prosodic features are subjected to feature fusion compression to obtain fused compressed features; low-frequency decoding is performed on the fused compressed features to obtain low-frequency decoding features; high-frequency decoding is performed on the fused compressed features to obtain high-frequency decoding features; the low-frequency decoding features and the high-frequency decoding features are merged, and the merged features are converted into a target synthesized speech. Through high- and low-frequency decoding, richer and more accurate speech details are restored, so that the synthesized speech sounds more natural and smooth, thereby improving the credibility and fluency of the speech.

[0161] For the specific definition of a layered coding and decoding speech synthesis device, please refer to the definition of a layered coding and decoding speech synthesis method above, and will not be repeated here. The various modules in the above-mentioned layered coding and decoding speech synthesis device can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0162] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a layered coding and decoding speech synthesis method.

[0163] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the client side of a layered coding and decoding speech synthesis method.

[0164] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0165] Extract semantic features from the pre-acquired original text to obtain text semantic features;

[0166] Extracting rhythmic features from the original text to obtain text rhythmic features;

[0167] Performing feature fusion compression on the text semantic features and the text prosodic features to obtain fused compressed features;

[0168] Performing low-frequency decoding on the fused compressed features to obtain low-frequency decoding features;

[0169] Performing high-frequency decoding on the fused compressed features to obtain high-frequency decoding features;

[0170] The low-frequency decoded features and the high-frequency decoded features are merged, and the merged features are converted into target synthesized speech.

[0171] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0172] Extract semantic features from the pre-acquired original text to obtain text semantic features;

[0173] Extracting rhythmic features from the original text to obtain text rhythmic features;

[0174] Performing feature fusion compression on the text semantic features and the text prosodic features to obtain fused compressed features;

[0175] Performing low-frequency decoding on the fused compressed features to obtain low-frequency decoding features;

[0176] Performing high-frequency decoding on the fused compressed features to obtain high-frequency decoding features;

[0177] The low-frequency decoded features and the high-frequency decoded features are merged, and the merged features are converted into target synthesized speech.

[0178] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0179] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0180] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0181] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. If software tools or components other than those of the company appear in the application embodiments, they are merely used for illustration and do not represent actual use. Although the present invention has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above-mentioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A layered coding and decoding speech synthesis method, characterized in that: include: Extract semantic features from the pre-acquired original text to obtain text semantic features; Extracting rhythmic features from the original text to obtain text rhythmic features; Performing feature fusion compression on the text semantic features and the text prosodic features to obtain fused compressed features; Performing low-frequency decoding on the fused compressed features to obtain low-frequency decoding features; Performing high-frequency decoding on the fused compressed features to obtain high-frequency decoding features; The low-frequency decoded features and the high-frequency decoded features are merged, and the merged features are converted into target synthesized speech.

2. The layered coding and decoding speech synthesis method according to claim 1, wherein: The semantic feature extraction of the pre-acquired original text to obtain the text semantic features includes: Perform word segmentation processing on the pre-acquired original text to obtain word segmentation text, and convert the word segmentation text into word segmentation low-dimensional vectors one by one; Performing forward and backward sequence splicing processing on the word segmentation low-dimensional vectors through a bidirectional long short-term memory network to obtain a splicing vector; The attention weight of each splicing vector is calculated, and the splicing vector and the attention weight of the splicing vector are weighted summed to obtain the text semantic feature.

3. The layered coding and decoding speech synthesis method according to claim 1, wherein: The prosodic feature extraction of the original text to obtain the text prosodic feature includes: Performing dilated convolution processing on the original text to obtain convolution features; Linearly projecting the convolution feature onto a preset subspace to obtain a feature projection corresponding to the subspace; Calculate the scaled dot product attention of the feature projections one by one, and concatenate the attention results into a concatenated feature sequence; The concatenated feature sequences are subjected to nonlinear transformation and fusion to obtain text prosodic features.

4. The layered coding and decoding speech synthesis method according to claim 1, wherein: The step of fusing and compressing the text semantic features and the text prosodic features to obtain fused compressed features includes: Performing feature alignment on the text semantic features and the text prosodic features to obtain an alignment feature; Performing gated fusion on the alignment features to obtain fusion features; Feature compression is performed on the fused features to obtain fused compressed features.

5. The layered coding and decoding speech synthesis method according to claim 2, wherein: The performing low-frequency decoding on the fused compressed features to obtain low-frequency decoding features includes: Performing a duration analysis on the segmented text to obtain a duration analysis result corresponding to each segmented text; Performing feature expansion on the fused compressed feature according to the duration analysis result to obtain an expanded feature; Performing context aggregation on the extended features to obtain aggregated features; Low-frequency extraction is performed on the aggregated features to obtain low-frequency decoding features.

6. The layered coding and decoding speech synthesis method according to claim 1, wherein: The performing high-frequency decoding on the fused compressed features to obtain high-frequency decoding features includes: Performing residual analysis on the fused compressed features to obtain feature residuals; Performing residual enhancement on the feature residual to obtain an enhanced residual; Reconstructing high-frequency features of the fused compressed features according to the enhanced residual to obtain initial high-frequency decoding features; The initial high-frequency decoding feature and the preset true high-frequency decoding feature are subjected to spectrum consistency loss optimization to obtain a high-frequency decoding feature.

7. The layered coding and decoding speech synthesis method according to claim 1, wherein: The merging of the low-frequency decoding features and the high-frequency decoding features and converting the merged features into target synthesized speech includes: Merging the low-frequency decoding features and the high-frequency decoding features to obtain a merged feature; Inputting the combined features into a preset vocoder model to generate a preliminary waveform signal; Performing spectrum analysis on the preliminary waveform signal, and filtering the waveform signal according to the spectrum analysis result to obtain an optimized waveform signal; The optimized waveform signal is amplitude-adjusted and phase-corrected to obtain target synthesized speech.

8. A layered coding and decoding speech synthesis device, characterized in that: include: The semantic extraction module is used to extract semantic features from the pre-acquired original text to obtain text semantic features; A rhythm extraction module, configured to extract rhythm features from the original text to obtain text rhythm features; A compression module, configured to perform feature fusion compression on the text semantic features and the text prosodic features to obtain fused compressed features; A low-frequency decoding module, configured to perform low-frequency decoding on the fused compressed features to obtain low-frequency decoding features; A high-frequency decoding module, configured to perform high-frequency decoding on the fused compressed features to obtain high-frequency decoding features; The merging and conversion module is used to merge the low-frequency decoding features and the high-frequency decoding features, and convert the merged features into a target synthesized speech.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the layered coding and decoding speech synthesis method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the layered coding and decoding speech synthesis method according to any one of claims 1 to 7 is implemented.

Images