Phoneme alignment model training and speech synthesis method and device, equipment and medium

Abstract

The application relates to the technical field of speech synthesis, in particular to a phoneme alignment model training and speech synthesis method and device, equipment and a medium. The method comprises the following steps: performing convolution attention alignment on spectral feature information and text feature information obtained according to speech training data to obtain a first alignment matrix; performing monotone alignment search based on the first alignment matrix to generate a second alignment matrix, the second alignment matrix being a binary hard attention matrix; calculating a relative entropy loss according to the first alignment matrix and the second alignment matrix; extending the text feature information to the length of a mel spectrum frame according to the second alignment matrix, and performing linear transformation on the extended text feature information to generate predicted mel spectrum; calculating a mel loss according to the spectral feature information and the predicted mel spectrum; and training a phoneme alignment model according to the relative entropy loss and the mel loss until a preset convergence condition is met. The method can improve phoneme alignment accuracy and thus improve speech synthesis accuracy.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, and in particular to a phoneme alignment model training and speech synthesis method, apparatus, computer equipment, storage medium and computer program product. Background Technology

[0002] In speech synthesis technology, text information needs to be converted into a phoneme sequence, then a spectrogram is generated, and finally a speech signal is synthesized. To ensure that the synthesized speech corresponds strictly to the original text content in time, a clear alignment relationship needs to be established between phonemes and speech frames. Therefore, the quality of phoneme alignment directly determines the speech rhythm, pronunciation clarity, and accuracy of semantic expression.

[0003] Currently, alignment methods in related technologies primarily rely on attention weights to automatically learn the relationship between speech and text, thereby generating alignment paths. While this addresses the alignment issue to some extent, in practical applications, these methods are prone to significant alignment errors at phoneme boundaries, particularly at the junctions of consonants and vowels. Due to the drastic energy changes and complex speech frame features at these locations, the model struggles to accurately determine phoneme boundaries, easily leading to unclear pronunciation, swallowed consonants, and pronunciation deviations. Furthermore, the transition regions between silence frames and effective pronunciation frames at the beginning and end of speech segments are frequently misjudged, causing the first phoneme to be pronounced prematurely or the last phoneme to be elongated into a silence segment. This misalignment significantly impacts the naturalness and intelligibility of the speech.

[0004] It is evident that the relevant technologies suffer from problems such as large alignment errors between consonant and vowel boundaries and misalignment of silence segments during speech synthesis, and their accuracy in speech synthesis is insufficient to cope with the ever-changing real-world application scenarios. Summary of the Invention

[0005] Therefore, it is necessary to provide a phoneme alignment model training and speech synthesis method, device, computer equipment, storage medium and computer program product that can improve the accuracy of phoneme alignment and thus improve the accuracy of speech synthesis, in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides a method for training a phoneme alignment model, including:

[0007] Acquire speech training data, and obtain spectral feature information and text feature information based on the speech training data; wherein, the spectral feature information includes multiple Mel spectrum frames, and the text feature information includes multiple phonemes;

[0008] The spectral feature information and the text feature information are convolved and attention-aligned to obtain a first alignment matrix;

[0009] Based on the first alignment matrix, a monotonic alignment search is performed to generate a second alignment matrix, which is a binary hard attention matrix;

[0010] Calculate the relative entropy loss based on the first alignment matrix and the second alignment matrix;

[0011] The text feature information is extended to the length of the Mel spectrum frame according to the second alignment matrix, and the extended text feature information is linearly transformed to generate the predicted Mel spectrum.

[0012] The Mel loss is calculated based on the spectral feature information and the predicted Mel spectrum.

[0013] The phoneme alignment model is trained based on the relative entropy loss and the Mel loss until the preset convergence condition is met.

[0014] In one embodiment, the method further includes:

[0015] The phoneme probability sequence is calculated based on the logarithmic probability distribution function on the first alignment matrix; the phoneme probability sequence is the logarithmic value of the phoneme classification probability corresponding to each Mel spectrum frame;

[0016] Calculate the connection temporal classification loss based on the text feature information and the phoneme probability sequence;

[0017] The phoneme alignment model is trained based on the connection-time classification loss, the relative entropy loss, and the Mel loss until the preset convergence condition is met.

[0018] In one embodiment, the method further includes:

[0019] The predicted Mel frequency band probability is generated based on the text feature information; the predicted Mel frequency band probability includes the classification probability distribution of each phoneme corresponding to the Mel frequency band;

[0020] Calculate the cross-entropy loss between the predicted Mel band probability and the actual Mel spectrum;

[0021] The phoneme alignment model is trained based on the cross-entropy loss, the connection temporal classification loss, the relative entropy loss, and the Mel loss until the preset convergence condition is met.

[0022] In one embodiment, generating the predicted Mel band probability based on the text feature information includes:

[0023] The text feature information is mapped to the predicted Mel spectrum using a linear projection head;

[0024] The predicted Mel band probability is calculated based on the logarithmic probability distribution function of the predicted Mel spectrum.

[0025] In one embodiment, the step of extending the text feature information to the length of a Mel spectrum frame according to the second alignment matrix, and performing a linear transformation on the extended text feature information to generate a predicted Mel spectrum, includes:

[0026] The duration of each phoneme is extracted based on the second alignment matrix;

[0027] The text feature information is extended to the length of a Mel spectrum frame according to the duration.

[0028] The expanded text feature information is mapped to the Mel spectrum frame dimension using a Mel projection head to generate a predicted Mel spectrum.

[0029] In one embodiment, the step of performing convolutional attention alignment on the spectral feature information and the text feature information to obtain a first alignment matrix includes:

[0030] The first alignment matrix is ​​generated based on the negative Euclidean distance between the text feature information and the spectral feature information.

[0031] Secondly, this application provides a speech synthesis method, including:

[0032] Obtain the text information to be processed;

[0033] The text information to be processed is processed using the phoneme alignment model as described in the first aspect to obtain target spectrum information;

[0034] The target speech signal is obtained based on the target spectrum information.

[0035] Thirdly, this application also provides a phoneme alignment model training device, comprising:

[0036] The data acquisition module is used to acquire speech training data and obtain spectral feature information and text feature information based on the speech training data; wherein, the spectral feature information includes multiple Mel spectrum frames, and the text feature information includes multiple phonemes;

[0037] The data alignment module is used to perform convolutional attention alignment on the spectral feature information and the text feature information to obtain a first alignment matrix;

[0038] The first processing module is used to perform a monotonic alignment search based on the first alignment matrix to generate a second alignment matrix, wherein the second alignment matrix is ​​a binarized hard attention matrix.

[0039] The first calculation module is used to calculate the relative entropy loss based on the first alignment matrix and the second alignment matrix;

[0040] The second processing module is used to extend the text feature information to the length of the Mel spectrum frame according to the second alignment matrix, and to perform a linear transformation on the extended text feature information to generate a predicted Mel spectrum.

[0041] The second calculation module is used to calculate the Mel loss based on the spectral feature information and the predicted Mel spectrum.

[0042] The model training module is used to train the phoneme alignment model based on the relative entropy loss and the Mel loss until a preset convergence condition is met.

[0043] Fourthly, this application provides a speech synthesis apparatus, the apparatus comprising:

[0044] The information acquisition module is used to acquire text information to be processed;

[0045] The information processing module is used to process the text information to be processed using the phoneme alignment model as described in the first aspect to obtain target spectrum information;

[0046] A speech synthesis module is used to obtain a target speech signal based on the target spectrum information.

[0047] Fifthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the steps as described in the first or second aspect.

[0048] Sixthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps described in the first or second aspect.

[0049] In a seventh aspect, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the steps as described in the first or second aspect.

[0050] The aforementioned phoneme alignment model training method, apparatus, computer equipment, storage medium, and computer program product acquire speech training data and, based on this data, obtain spectral feature information including multiple Mel-spectrum frames and text feature information including multiple phonemes. The spectral and text feature information are then convolved with attention alignment to obtain a first alignment matrix. This first alignment matrix expresses the model's natural tendency towards boundary positions. Based on this, a second alignment matrix is ​​generated by performing a monotonic alignment search on the first alignment matrix. The second alignment matrix is ​​a binarized hard attention matrix. Binarization discretizes the first alignment matrix, clearly identifying which specific phoneme each frame should align to, ensuring that the alignment path structurally satisfies strict monotonicity. Next, by calculating the relative entropy loss based on the first and second alignment matrices, the difference between them can be measured. This encourages the hard alignment path to maintain a clear structure while closely approximating the model's perception of boundaries, thereby minimizing boundary distortion caused by alignment path offset during binarization. More reasonable time allocation is achieved at phoneme boundaries (such as the boundary between consonants and vowels), mitigating misalignment issues such as premature or delayed boundary alignment. Subsequently, the text feature information is extended to the length of the Mel spectrum frame based on the second alignment matrix, and the extended text feature information is linearly transformed to generate a predicted Mel spectrum. Then, the Mel loss is calculated based on the spectral feature information and the predicted Mel spectrum, thereby optimizing the spectral reconstruction quality and further strengthening the temporal correspondence between frames and phonemes. This reverse optimization of the spectral reconstruction loss effectively reduces the phenomenon of silent frames being incorrectly assigned to phoneme regions, improving the alignment accuracy of the model in silent transition regions such as sentence beginnings and endings. Finally, the phoneme alignment model is trained based on relative entropy loss and Mel loss until a preset convergence condition is met, improving the accuracy of phoneme alignment. Furthermore, the above-mentioned speech synthesis method, apparatus, computer equipment, storage medium, and computer program products, by acquiring the text information to be processed, using the above-mentioned phoneme alignment model to process the text information to be processed, obtaining target spectral information, and then obtaining the target speech signal based on the target spectral information, can apply the above-mentioned phoneme alignment model to actual speech synthesis scenarios, enabling the target speech signal to have higher accuracy under complex requirements. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1This is a diagram illustrating the application environment of a phoneme alignment model training method in one embodiment.

[0053] Figure 2 This is a flowchart illustrating a phoneme alignment model training method in one embodiment;

[0054] Figure 3 This is a flowchart illustrating the multi-loss joint training method used in one embodiment of the phoneme alignment model training method;

[0055] Figure 4 This is a flowchart illustrating the multi-loss joint training process used in another embodiment of the phoneme alignment model training method.

[0056] Figure 5 This is a structural block diagram of a phoneme alignment model training device in one embodiment;

[0057] Figure 6 This is an internal structural diagram of a computer device in one embodiment;

[0058] Figure 7 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] The phoneme alignment model training method and speech synthesis method provided in the embodiments of this application can both be applied to, for example, Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. Terminal 102 can be used to acquire speech training data, send the speech training data to server 104 for phoneme alignment model training, and can also be used to acquire text information to be processed, send the text information to server 104 for speech synthesis, and then receive and output the target speech signal processed by server 104. The terminal 102 used to acquire speech training data and the terminal 102 used to acquire text data to be processed can be the same terminal device or different terminal devices. The data storage system can store the data that server 104 needs to process. The data storage system can be integrated on server 104 or placed on the cloud or other network servers. The data storage system can be used to store data such as speech training data. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster composed of multiple servers.

[0061] In one exemplary embodiment, such as Figure 2 As shown, a phoneme alignment model training method is provided, which is then applied to... Figure 1 Taking server 104 as an example, the explanation includes steps S202 to S214. It is understood that this phoneme alignment model training method can also be applied to terminal 102. Wherein:

[0062] Step S202: Obtain speech training data, and based on the speech training data, obtain spectral feature information and text feature information.

[0063] The spectral feature information includes multiple Mel-spectral frames, and the text feature information includes multiple phonemes. For example, the speech training data includes raw speech information and raw text information.

[0064] For example, server 104 can first obtain training samples containing raw speech and text from the speech training dataset obtained by terminal 102. The speech training data may include one or more standard speech audio files and their corresponding text content. For example, a piece of training data may be a raw speech information file, and its accompanying raw text information may be "Hello, today is Wednesday". Based on this data, server 104 can further extract structured feature information, namely spectral feature information and text feature information, from the speech and text respectively for use by the model learning.

[0065] Spectral feature information refers to the transformation of the original speech signal into a feature representation with both time and frequency dimensions through acoustic processing methods. Specifically, it can take the form of a Mel spectrogram. A Mel spectrogram is a spectral transformation method that is more sensitive to human hearing. It performs a short-time Fourier transform on the original waveform and then maps it to a Mel scale to express the frequency energy distribution of the audio at different points in time. Server 104 can divide a speech segment into frames, obtaining multiple Mel spectrogram frames. Each frame can be understood as the sound performance within a certain time window of the speech. For example, the audio "Hello" might be divided into 80 frames after Mel transformation, each representing a 12.5-millisecond speech signal, lasting approximately 1 second, constituting the spectral feature information of that sample.

[0066] Simultaneously, server 104 can preprocess the text information corresponding to the audio to extract text feature information. Text feature information mainly refers to phoneme sequences, which are the smallest phonetic units that constitute text pronunciation. Phonemes are the basic pronunciation units of language; for example, the Chinese pinyin “nǐ hǎo” can be divided into the phoneme sequence [n,i,h,ao]. Server 104 can perform standardization, word segmentation, pinyin transcription, and phoneme annotation on the original text through the text analysis module, converting the natural language text into a machine-processable phoneme number sequence. This process can be implemented using a Chinese phoneme annotation dictionary or a method based on preset rules, ultimately obtaining a phoneme list, such as [5,13,42,8], where each number corresponds to a specific phoneme meaning.

[0067] For example, server 104 can extract audio variation features from the original speech information to obtain spectral feature information; determine the phoneme number sequence based on the original text information; input the phoneme number sequence into the phoneme encoder to obtain the phoneme encoding vector, and use the phoneme encoding vector as text feature information.

[0068] The audio variation features can include frequency variation features and rate variation features. Server 104 can extract audio variation features from the original speech information. These features can include two dimensions: first, the frequency variation itself over time, i.e., frequency variation features; and second, the rate of frequency variation, i.e., rate variation features. Both serve as important signals revealing temporal structures in speech, such as boundary changes, speech rate transitions, and silence transitions. For example, server 104 can extract the frequency variation features of audio frequencies over time from the original speech information; extract the rate of audio frequency variation over time from the original speech information to obtain rate variation features; and obtain spectral feature information containing first-order difference features based on the frequency variation features and rate variation features. Specifically, server 104 can perform a short-time Fourier transform (STFT) on the original speech waveform and map it to a Mel scale to obtain a basic spectral frame sequence as the basis for the frequency variation features. Based on this, the server 104 can further perform a difference calculation on these spectrum frame sequences in the frame order direction, that is, perform a vector difference operation on each frame spectrum and its previous frame to obtain the time derivative of the spectrum, thereby capturing the speed of speech change in the frequency dimension with time. This difference result is the rate change feature, which can also be called the first-order difference or delta feature.

[0069] For example, if the Mel spectrum of a certain frame is [0.3, 0.4, 0.6, ...], and the next frame is [0.35, 0.38, 0.58, ...], then subtracting the two yields [+0.05, -0.02, -0.02, ...], representing whether the energy is increasing or decreasing in these frequency bands, and the rate of change. This type of information changes more dramatically at the boundaries between phonemes, especially between consonants and vowels, or when switching from a silent segment to a spoken segment, significantly revealing the changing trend of the sound structure. Server 104 can combine this first-order difference feature with the original Mel spectrum as the final spectral feature information to enhance the model's sensitivity to boundaries.

[0070] On the other hand, the server 104 can perform structured processing on the original text information to determine a sequence of phoneme numbers. This process can include normalizing the original text and converting it into pinyin, and then mapping the pinyin to a set of phoneme ID numbers according to a phoneme dictionary. Taking the phrase "ni hao" as an example, the server 104 can decompose it into phonemes [n, i, h, ao], and assign numbers according to the phoneme dictionary, such as [12, 4, 19, 31], to form a clear phoneme sequence. Subsequently, the server 104 can input the above phoneme number sequence into a phoneme encoder for modeling. The phoneme encoder can include an embedding layer and multiple one-dimensional convolutional layers, which learn to map each phoneme number to a high-dimensional semantic vector and capture the context connection between phonemes. For example, the number "12" represents the phoneme / n / , which becomes a 512-dimensional vector [0.21, -0.04, 0.38, …] after embedding. After convolutional extraction of the context, a final phoneme encoding vector is formed. The server 104 can combine the encoding vectors of all phonemes into an ordered sequence as the above text feature information.

[0071] In this step, the server 104 not only completes the extraction of basic features of the spectrum and phonemes, but also introduces dynamic information on the audio change trend (such as delta features) and the context expression ability between phonemes, thus laying a more solid temporal foundation and semantic expression ability for subsequent attention alignment and duration prediction. Especially for speech segments with剧烈语速变化或模糊边界, the change features introduced in this step can effectively improve the recognition accuracy and alignment robustness of the model for the boundary region.

[0072] Step S204: Perform convolutional attention alignment on the spectral feature information and the text feature information to obtain a first alignment matrix.

[0073] Among them, the first alignment matrix is used to represent the alignment relationship between the Mel spectrum frames and the phonemes. The spectral feature information can be expressed as [B, T, d], the text feature information can be expressed as [B, L, d], and the first alignment matrix can be expressed as [B, T, L], where B is the batch size, T is the number of spectrum frames (for example, the number of time steps included in each speech signal), L is the length of the phoneme sequence, and d is the feature dimension.

[0074] It should be noted that in the original text, there is an unclear expression "剧烈语速变化", which may need to be further clarified according to the specific context for a more accurate translation.For example, server 104 can input spectral feature information as a query and text feature information as a key into two sets of structurally identical but parameter-independent one-dimensional convolutional networks. Each convolutional network uses multiple one-dimensional convolutional kernels to slide along the time dimension to extract local contextual information of the features. The spectral feature information [B,T,d] is transformed into an enhanced query representation [B,T,d] after convolution, while the text feature information [B,L,d] is transformed into an enhanced key representation [B,L,d] through another set of convolutional operations. While preserving the temporal order structure of the original features, the receptive fields of the convolutional kernels capture the correlation patterns between local phonemes or spectra.

[0075] Next, server 104 can perform similarity calculation between the processed Query and Key. For example, server 104 can generate a first alignment matrix based on the negative Euclidean distance of textual and spectral feature information. Alternatively, server 104 can expand the Query and Key into a three-dimensional tensor that can be computed item by item using a broadcast mechanism, and calculate the sum of squared differences between them along the feature dimension d to obtain a distance matrix of shape [B,T,L].

[0076] Furthermore, to transform this distance matrix into a probabilistic attention matrix, server 104 can then negative it and normalize it using the softmax function. The softmax operation can be performed on L (phoneme dimension), ensuring that the sum of all phoneme matching scores for each frame is 1. The normalized output is the first alignment matrix [B,T,L], where each element represents the attention weight of a spectral feature for a particular phoneme in a given frame, i.e., the normalized matching probability. For example, a row [0.65,0.3,0.05,0] indicates that the spectral frame is primarily determined by the first phoneme, but may also be related to the second phoneme.

[0077] The final output first alignment matrix not only possesses probability distribution properties, facilitating subsequent calculation of the loss function and generation of hard-aligned paths, but also exhibits differentiability, serving as a crucial bridge for backpropagation optimization. This matrix preserves the temporal features of the original input while also expressing the matching tendency of each phoneme in each frame, providing a high-resolution probability matrix foundation for constructing stable, monotonic, and semantically consistent phoneme alignment paths.

[0078] Furthermore, server 104 can also calculate the first alignment matrix using a cosine distribution scheme. Server 104 can first perform L2 normalization on each Mel spectrum frame vector (Query) and each phoneme vector (Key) so that they are all on a unit sphere, and then calculate the dot product between them as a similarity score. The value range of this cosine similarity is [–1, 1], representing the degree of consistency in the direction of the two vectors. Subsequently, server 104 can normalize these similarity scores again through a softmax operation to obtain the first alignment matrix of shape [B, T, L], where each row represents the probability distribution of the frame's association with all phonemes. Cosine similarity focuses more on direction than amplitude, so it is easier to maintain alignment stability when facing samples with different pronunciation intensities or large amplitude variations, especially suitable for situations where the boundaries between phonemes are blurred and the energy changes drastically.

[0079] On the other hand, server 104 can also employ a multi-head attention mechanism to compute the first alignment matrix, splitting each input vector into multiple subspace representations and performing attention matching independently for each. Server 104 can split the input spectral features [B,T,d] and text features [B,L,d] into h sub-vectors, for example, splitting d=512 into 8 sub-heads, each with 64 dimensions, and then calculating attention scores separately to obtain multiple alignment maps of [B,T,L]. These sub-maps capture the frame-phoneme relationships under different representation subspaces, and finally, the server performs weighted fusion of them to obtain the final first alignment matrix. The multi-head attention mechanism allows the model to understand the alignment relationship between frames and phonemes from multiple perspectives at the same time, thereby constructing a more robust and diverse attention map, suitable for synthesis scenarios such as long sentences, complex speech rates, and multiple speaker styles, and can significantly reduce the situation where a single attention head cannot align or misaligns.

[0080] Step S206: Perform a monotonic alignment search based on the first alignment matrix to generate a second alignment matrix.

[0081] The second alignment matrix is ​​a binary hard attention matrix.

[0082] For example, server 104 can perform Monotonic Alignment Search (MAS) based on a first alignment matrix, which is a probability distribution tensor of shape [B, T, L], where each value represents the matching probability of a given frame's spectrum across all phonemes. During MAS, server 104 can construct a dynamic programming table D and search for the path with the highest cumulative probability in both the temporal dimension T and the phoneme dimension L, following a monotonically increasing path rule. This algorithm adheres to the monotonic constraint that frames must match phonemes sequentially and cannot jump back to the previous phoneme, ensuring that the generated alignment path conforms to the temporal pattern of phoneme emission frame by frame in natural speech.

[0083] After the search is complete, server 104 can backtrack from the dynamic programming table to find the path with the highest probability and generate a second alignment matrix accordingly. This matrix is ​​still in the shape of [B,T,L], but it is in a strictly binary form, with only one element in each row being 1 and the rest being 0, clearly representing the unique phoneme corresponding to each frame. This binary structure is the hard attention matrix, which no longer retains the probabilistic uncertainty in the first alignment matrix, but instead constructs a precise distribution of phonemes on the time axis based on a clear frame-phoneme attribution relationship.

[0084] Step S208: Calculate the relative entropy loss based on the first alignment matrix and the second alignment matrix.

[0085] For example, server 104 can calculate the relative entropy loss (Kullback-Leibler divergence, KL Loss) based on the first alignment matrix and the second alignment matrix obtained in the previous steps. The first alignment matrix is ​​a probabilistic soft attention matrix A_soft, calculated by the model through the convolutional attention mechanism, with a shape of [B,T,L]. Each frame has a normalized probability representation for each phoneme, is differentiable, and can participate in gradient calculation. The second alignment matrix is ​​a binarized result generated by monotonic alignment search (MAS), i.e., a hard attention matrix A_hard, also [B,T,L], but each row has only one position set to 1, with the rest set to 0, representing a unique phoneme corresponding to each frame. Since the second alignment matrix is ​​inherently discrete, to ensure its participation in backpropagation, server 104 can relax this binary matrix using the Gumbel-Softmax mechanism to approximate it as a differentiable form. Server 104 can use the first alignment matrix as the "predicted distribution" and the relaxed second alignment matrix as the "target distribution" to perform frame-by-frame and phoneme-by-phoneme KL divergence calculations on both, thereby measuring the difference between the model's current attention output and the ideal hard alignment path.

[0086] Furthermore, the loss function can be expressed as:

[0087]

[0088] KL divergence measures the information redundancy or offset of the model's current predicted distribution relative to the target distribution; a smaller value indicates that the model is closer to the ideal alignment path. For example, if a frame's distribution in the first alignment matrix is ​​[0.7, 0.2, 0.1], while the corresponding second alignment matrix is ​​[1, 0, 0], then KL Loss will penalize the non-maximum probability parts, guiding the model to further focus its attention on the correct phonemes in the next iteration. During training, Server104 does not directly replace the output of the soft alignment matrix with the hard alignment matrix, but instead uses KL divergence as the learning target, thus preserving the model's tolerance for blurred boundary regions. Simultaneously, the monotonic path provided by hard alignment guides the model's convergence direction, avoiding attention drift or jump mismatches. Through relative entropy loss, Server104 can guide the model to continuously approach the ideal path in each training round, improving the accuracy of frame-level attention distribution. Especially in segments where the boundaries between consonants and vowels are blurred, pronunciation changes are drastic, or there are silent transitions, the combination of the guiding nature of the soft alignment matrix and the structural nature of the hard alignment matrix allows the model to maintain flexibility while possessing clear and definite boundary judgment capabilities, fundamentally reducing problems such as phoneme pronunciation misalignment, boundary slippage, and silent frame crowding.

[0089] Step S210: Extend the text feature information to the length of the Mel spectrum frame according to the second alignment matrix, and perform a linear transformation on the extended text feature information to generate the predicted Mel spectrum.

[0090] For example, server 104 can expand the text feature vector from phoneme granularity to frame-level granularity based on the binarized second alignment matrix, aligning it with the spectrum in the temporal dimension. Specifically, server 104 can copy the corresponding phoneme vector to the position of each "1" in the second alignment matrix, ultimately constructing a frame-level text feature sequence of shape [B,T,d], where each frame has a high-dimensional semantic representation of the corresponding phoneme. This expansion process ensures that the phoneme representation is strictly aligned with each Mel spectrum frame in time, providing structural support for subsequent spectrum prediction.

[0091] After the frame-level text features are constructed, server 104 can input them into a linear transformation layer, namely a fully connected projection layer, to map the frame-level phoneme vectors from the encoding dimension d to the Mel spectrum dimension n_mels (e.g., 80). The output shape is [B, T, n_mels], which is the Mel band energy prediction value of each frame spectrum. This output is the predicted Mel spectrum. Through this linear mapping, server 104 realizes the transformation from structured text semantics to specific speech energy distribution, enabling the model to have the ability to generate actual speech. Through the above steps, based on the soft alignment to hard alignment transformation, the phoneme information is extended to the length of a complete speech frame, and the spectrum corresponding to the real speech is predicted.

[0092] For example, server 104 can extract the duration of each phoneme according to the second alignment matrix; extend the text feature information to the length of the Mel spectrum frame according to the duration; and map the extended text feature information to the Mel spectrum frame dimension through the Mel projection head to generate the predicted Mel spectrum.

[0093] For example, server 104 can extract the duration of each phoneme from the second alignment matrix. The second alignment matrix is ​​a binary attention matrix of shape [B,T,L], where each row represents the unique phoneme position corresponding to a frame, and "1" in the matrix indicates that the frame belongs to a certain phoneme, while "0" indicates that it does not. The server traverses the matrix and counts the total number of "1" values ​​in each column, that is, the number of frames each phoneme is assigned in the speech segment. This statistical result can be organized into a phoneme duration vector of shape [B,L], where each element represents the frame-level duration of the corresponding phoneme. For example, if the phoneme " / a / " is labeled with 10 frames in the matrix, then its duration value is 10.

[0094] After obtaining the phoneme duration vector, server 104 can extend the text feature information in the time dimension. The original text feature information has a shape of [B,L,d], representing the semantic vector corresponding to each phoneme. Based on the duration of each phoneme, the server copies its corresponding vector a certain number of times in the time dimension, thus obtaining a frame-level text vector sequence [B,T,d], where T is the total number of Mel spectrum frames. For example, if the phoneme " / t / " lasts for 4 frames, its encoding vector will be copied 4 times and filled into 4 consecutive positions in the generated sequence. Next, server 104 can input the extended frame-level text features into a Mel projection head. The Mel projection head is essentially a fully connected layer or a set of linear transformation modules, whose function is to map the input d-dimensional phoneme representation vector to an n_mels-dimensional Mel spectrum space (e.g., 80-dimensional), with an output shape of [B,T,n_mels], representing the Mel spectrum energy distribution corresponding to each frame. Through the above steps, server 104 can obtain a complete predicted Mel spectrum, which is based on text semantic information and expanded by phoneme duration control, possessing both linguistic content and strictly aligned with the actual pronunciation rhythm.

[0095] In addition, the server 104 can also concatenate the expanded text feature information with the speaker timbre vector. This timbre vector can be extracted by the server according to the speaker number and represents the pronunciation style and speech characteristics corresponding to the speech sample. By concatenating the timbre information, the server 104 can provide more complete acoustic condition input, so that the subsequently generated spectrum results can accurately reflect the timbre characteristics of a specific speaker.

[0096] Step S212: Calculate the Mel loss based on the spectral feature information and the predicted Mel spectrum.

[0097] For example, server 104 can calculate Mel loss based on the predicted Mel spectrum and the real spectrum data to constrain the accuracy of spectrum generation. The predicted Mel spectrum is the [B,T,n_mels] tensor generated by the Mel projection head in the above steps, while the real spectrum is the original Mel spectrum extracted from the training speech samples, which have the same shape. Server 104 can square the element-wise difference between the two and average it over all time frames and frequency bands to obtain the mean squared error (MSE) value, which is the Mel loss. This loss reflects the degree of matching between the spectrum generated by the model and the real speech spectrum and is an important indicator for measuring the naturalness and clarity of the synthesized speech.

[0098] Furthermore, the loss function can be expressed as:

[0099]

[0100] This involves calculating the squared difference of the energy values ​​for each frame and each frequency band and averaging them. The smaller the value, the closer the model's predicted spectrum is to the actual speech spectrum. For example, if the actual spectrum value of a certain frame is [0.3, 0.5, 0.4], and the model predicts it as [0.28, 0.52, 0.39], then the error for that frame is ((0.3-0.28)^2+(0.5-0.52)^2+(0.4-0.39)^2) / 3. Finally, the errors of all frames are averaged to obtain the overall mel loss. By using hard alignment to guide spectrum reconstruction and applying mel loss constraints, the model can accurately control the spectrum boundaries, preventing silence from being mistakenly occupied by phonemes, and avoiding excessive extension of phonemes at the end, thus improving the naturalness and clarity of speech boundaries.

[0101] Step S214: Train the phoneme alignment model based on relative entropy loss and Mel loss until the preset convergence condition is met.

[0102] For example, after obtaining the two types of loss values ​​mentioned above, server 104 can combine them according to preset weights to form a joint loss function, such as total_loss = 1*mel_loss + 1*kl_loss, and perform backpropagation training accordingly. During training, server 104 can continuously iteratively update various learnable parameters in the phoneme alignment model, including the convolutional kernel weights of the convolutional attention module, the embedding parameters of the phoneme encoder, the projection matrix of the Mel projector, etc. After each training round, the server evaluates the decreasing trend of the total loss value and compares it with preset convergence conditions, which may include the loss value decreasing by less than a threshold, the validation set performance stabilizing, and the number of training rounds reaching a set upper limit.

[0103] During model training, server 104 can continuously monitor the trend of total loss and set preset convergence conditions to determine when to terminate training. These conditions can include reaching the maximum number of training epochs, the total loss decreasing by less than a threshold for several consecutive epochs, or stopping the improvement in validation set accuracy. For example, if the total loss decreases by less than 0.001 in five consecutive training epochs, server 104 can determine that the model has converged stably. At this point, the current model parameter state can be recorded as the final model parameters, and the output is the trained phoneme alignment model.

[0104] Finally, when the total loss meets the preset convergence condition, server 104 can consider the model training complete, and the current model is a phoneme alignment model with high-quality alignment and spectral representation capabilities. Through this step, the server not only achieves explicit optimization of the phoneme-speech frame alignment structure, but also ensures a high degree of fidelity in the predicted spectrum to the actual pronunciation.

[0105] In the aforementioned phoneme alignment model training method, speech training data is acquired, and based on this data, spectral feature information including multiple Mel-spectrum frames and text feature information including multiple phonemes are obtained. The spectral and text feature information are then convolved with attention alignment to obtain a first alignment matrix. This first alignment matrix expresses the model's natural tendency towards boundary positions. Based on this, a second alignment matrix is ​​generated by performing a monotonic alignment search on the first alignment matrix. The second alignment matrix is ​​a binarized hard attention matrix. Binarization discretizes the first alignment matrix, thus clearly identifying which specific phoneme each frame should be aligned to, ensuring that the alignment path structurally satisfies strict monotonicity. Next, by calculating the relative entropy loss based on the first and second alignment matrices, the difference between them can be measured. This encourages the hard alignment path to maintain a clear structure while closely approximating the model's perception of boundaries, thereby minimizing boundary distortion caused by alignment path offset during binarization. This results in more reasonable time allocation at phoneme boundaries (such as the boundary between consonants and vowels), alleviating misalignment issues such as premature or delayed boundary alignment. Subsequently, the textual feature information is extended to the length of the Mel spectrum frame based on the second alignment matrix. The extended textual feature information is then linearly transformed to generate a predicted Mel spectrum. The Mel loss is calculated based on the spectral feature information and the predicted Mel spectrum, thereby optimizing the spectral reconstruction quality and further strengthening the temporal correspondence between frames and phonemes. This reverse optimization of the spectral reconstruction loss effectively reduces the phenomenon of silent frames being incorrectly assigned to phoneme regions, improving the alignment accuracy of the model in silent transition regions such as sentence beginnings and endings. Finally, the phoneme alignment model is trained using relative entropy loss and Mel loss until the preset convergence condition is met, improving the accuracy of phoneme alignment. Furthermore, the aforementioned speech synthesis method, apparatus, computer equipment, storage medium, and computer program product, by acquiring text information to be processed, processing the text information using the aforementioned phoneme alignment model to obtain target spectral information, and then obtaining the target speech signal based on the target spectral information, can apply the aforementioned phoneme alignment model to actual speech synthesis scenarios, enabling the target speech signal to have higher accuracy under complex requirements. This, in turn, improves the pronunciation clarity and temporal naturalness of the synthesized speech.

[0106] In one exemplary embodiment, such as Figure 3 As shown, the above method further includes steps S302 to S306. Wherein:

[0107] Step S302: Calculate the phoneme probability sequence based on the logarithmic probability distribution function on the first alignment matrix.

[0108] The phoneme probability sequence is the logarithmic value of the phoneme classification probability corresponding to each Mel spectrum frame.

[0109] For example, the first alignment matrix is ​​a three-dimensional tensor of shape [B,T,L], representing the matching probability distribution of each phoneme for each Mel-spectrum frame. To make it recognizable by the connection-time classification loss function, server 104 can perform a log_softmax operation on each row of this matrix to obtain a phoneme classification distribution in log probability form, forming a phoneme probability sequence. This sequence still maintains the [B,T,L] structure, where each value of each frame represents the log probability that the frame belongs to the corresponding phoneme. Compared to the ordinary softmax output, the log probability has better numerical stability and is suitable for subsequent cumulative calculation of the total path probability.

[0110] Step S304: Calculate the connection temporal classification loss based on text feature information and phoneme probability sequence.

[0111] For example, server 104 can use the phoneme probability sequence as input to the model's prediction path and the real phoneme number sequence in the text feature information as the target label to perform the calculation of connection-temporal classification loss. The connection-temporal classification loss (CTC loss) does not require explicit labeling of the boundary positions between frames and phonemes, but instead enumerates all possible paths that map the spectral frame sequence to the target phoneme sequence through a forward-backward algorithm and accumulates their total probabilities.

[0112] Furthermore, server 104 can allow the insertion of special whitespace symbols in the path during this step to accommodate natural speech rate variations and frame redundancy, making the aligned path temporally flexible. The objective of CTC loss calculation is to minimize the distance between the predicted path distribution and the true phoneme sequence, i.e., to maximize the total path probability, thereby guiding the model to automatically establish the optimal frame-to-phoneme mapping structure during the learning process. The loss function can be expressed as:

[0113]

[0114] Here, π represents the set of all valid alignment paths, and z represents the true phoneme sequence (including silence markers). The CTC loss maximizes the probability that the model generates the correct phoneme sequence, or equivalently, minimizes the total negative logarithmic probability. If the path distribution predicted by the model deviates significantly from the true phoneme order, or if it incorrectly classifies silence frames as valid phonemes, the loss value will increase significantly, reflecting the irrationality of the alignment path. Therefore, the CTC loss has advantages such as no need for manual boundary labeling, automatic alignment, and preservation of temporal monotonicity, making it particularly suitable for tasks where the speech sequence length is significantly longer than the phoneme sequence.

[0115] Step S306: Train the phoneme alignment model based on connection-time classification loss, relative entropy loss, and Mel loss until the preset convergence condition is met.

[0116] For example, server 104 can combine the connection-based temporal classification loss with the relative entropy loss (KL Loss) and Mel loss calculated in the preceding steps to form a joint training objective, constituting a complete multi-task loss function. For example, this joint loss can be set as: total_loss = 1*mel_loss + 1*kl_loss + 0.1*ctc_loss. Server 104 can perform gradient backpropagation based on this joint loss to update the parameter weights in the model, particularly the learnable parameters of the convolutional attention module, phoneme encoder, and CTC path prediction head.

[0117] This embodiment introduces CTC loss to enhance the modeling ability of monotonic paths between frame sequences and phoneme sequences, compensates for the local instability of the attention mechanism in boundary processing, and improves the global rationality and stability of the phoneme alignment model in the selection of frame-phoneme alignment paths. In particular, in areas with multi-syllable words, continuous pronunciation, or speech rate changes, CTC loss can effectively suppress phenomena such as attention jumps, uneven frame distribution, and premature speech. By controlling the distribution position of blank tags, it can effectively reduce the problem of silent frames being misidentified as valid phonemes and also alleviate the deviation of consonants being prematurely pronounced.

[0118] In one exemplary embodiment, such as Figure 4 As shown, the above method further includes steps S402 to S406. Wherein:

[0119] Step S402: Generate the predicted Mel band probability based on text feature information.

[0120] Among them, the predicted Mel band probability includes the classification probability distribution of each phoneme corresponding to the Mel band.

[0121] For example, server 104 can receive text feature information output by a phoneme encoder, with a shape of [B, L, d], representing a high-dimensional embedding representation of the phoneme sequence in each batch. The vector encoding of each phoneme carries abstract features such as semantics and timbre in the current context. Server 104 can introduce a linear transformation module (fully connected layer) to project each d-dimensional phoneme vector onto a space of n_mels, with an output shape of [B, L, n_mels]. This output can be understood as the predicted score of the energy of each frequency band corresponding to the phoneme. For example, if a phoneme " / a / " has high values ​​in the 4th, 17th, and 40th dimensions, it means that the model believes the main spectral energy of this phoneme is concentrated in these frequency bands. Server 104 can then normalize the above prediction results through a log_softmax operation to obtain a log probability distribution in the n_mels dimension. The output is still [B, L, n_mels], where the position of each phoneme represents the probability that it belongs to a certain Mel band morphological category. This is the “predicted Mel band probability”, which is essentially a categorical distribution rather than a continuous spectral curve. It can be seen as the model’s preference for choosing which type of sound template this phoneme belongs to.

[0122] For example, server 104 can map text feature information into a predicted Mel spectrum using a linear projection head; the predicted Mel band probability is calculated based on the log probability distribution function. Server 104 can input the text feature information [B,L,d] output by the phoneme encoder into a linear projection head, which is a fully connected layer whose function is to map the high-dimensional semantic vector of each phoneme to the spectral dimension space. Subsequently, in order to transform the prediction result into a probability distribution, server 104 performs a log_softmax operation on the predicted Mel spectrum using the log probability distribution function. During the operation, server 104 can log-normalize the spectral prediction value of each phoneme in the n_mels dimension, thereby obtaining a [B,L,n_mels] tensor with unchanged shape but different semantics, where each term represents the log probability of the phoneme belonging to a certain spectral category.

[0123] Through this classification probability modeling method, server 104 can transform the original spectrum modeling problem into a phoneme acoustic feature recognition problem, introducing a more accurate and interpretable supervision signal into the training process, making the relationship between phoneme semantics and acoustic features clearer and more stable, and providing a more discriminative intermediate representation for subsequent speech synthesis.

[0124] Step S404: Calculate the cross-entropy loss of the predicted Mel band probability and the actual Mel spectrum.

[0125] For example, server 104 can extract the corresponding real mel spectrum [B,T,n_mels] from the training speech samples, and then, based on the previously obtained phoneme duration (derived from the hard alignment matrix), segment the spectrum according to the phoneme boundaries and perform average pooling. For instance, if a phoneme corresponds to 10 frames, the server can take the average of the spectra of these 10 frames to obtain an n_mels-dimensional spectrum vector, and finally construct the real spectral feature matrix [B,L,n_mels].

[0126] In some embodiments, to enhance the effectiveness of classification supervision, the server can also perform K-means clustering on these real spectrum vectors to obtain spectrum category labels, transforming continuous values ​​into discrete categories so that each phoneme has a corresponding classification label. For example, if the average spectrum of " / a / " falls near cluster center #4, it can be labeled as "Category 4".

[0127] Next, server 104 can introduce a cross-entropy loss function (ce loss) to calculate the cross-entropy loss between the predicted melodic band probabilities [B,L,n_mels] and the aforementioned true labels, comparing the deviation between the predicted probability distribution of each phoneme and the true label on a phoneme-by-phoneme basis. Further, the cross-entropy loss function is as follows:

[0128]

[0129] Where y is the real Mel one-hot label, and p is the Mel frequency band probability predicted by the model. The smaller the cross-entropy, the more accurate the model prediction; the larger the cross-entropy, the lower the degree of matching between the phoneme semantics and the actual spectrum.

[0130] Step S406: Train the phoneme alignment model based on cross-entropy loss, connection temporal classification loss, relative entropy loss, and Mel loss until the preset convergence condition is met.

[0131] For example, server 104 can use the cross-entropy loss calculated in this embodiment, together with the other three types of losses obtained in the aforementioned steps, to form a joint training objective: wherein, connection-temporal classification loss (CTC Loss) is used to supervise the model to learn the global path structure between speech frames and phonemes, and solve the problem of inconsistent time lengths; relative entropy loss (KLLoss) is used to constrain the consistency between the soft attention and discrete hard alignment path of the model output, and improve the stability of alignment; Mel loss (Mel Loss) is used to regress the difference between the predicted spectrum and the true spectrum, and directly optimize the speech generation quality; cross-entropy loss (CE Loss) is used to improve the acoustic discriminability of phoneme semantic vectors, and make the spectral classification ability of phonemes stronger.

[0132] Server 104 can sum the four losses according to set weights, for example, total_loss = 1*mel_loss + 1*kl_loss + 0.1*ctc_loss + 1*ce_loss, and update the model parameters with this total loss as the target. Training continues to iterate until the loss value decreases to a stable level or the set termination conditions are met, such as the validation set accuracy no longer improving or the maximum number of training epochs being reached.

[0133] This embodiment enhances the model's ability to understand the classification boundaries of each phoneme in acoustic space, especially between similar phonemes and between the boundaries of silence and consonants. The model can more accurately distinguish between them, effectively alleviating the problems of blurred boundaries and overlapping pronunciations caused by unclear phoneme representation.

[0134] This application also provides a speech synthesis method. This embodiment mainly illustrates the application of this method to a computer device, which can specifically be... Figure 1 The terminal 102 or server 104 can be used independently to execute the speech synthesis method provided in the embodiments of this application. The terminal 102 and server 104 can also be used collaboratively to execute the speech synthesis method provided in the embodiments of this application.

[0135] For example, server 104 can acquire the text information to be processed; process the text information using the aforementioned phoneme alignment model to obtain target spectrum information; and obtain the target speech signal based on the target spectrum information. Specifically, server 104 can acquire the text information to be processed as the starting point for speech synthesis. The text content can be any Chinese phrase or complete sentence, such as "The sun is shining brightly today." Server 104 can call the built-in text processing module to standardize the sentence, including word segmentation, pinyin conversion, phoneme annotation, and other steps, ultimately converting the text into a set of ordered phoneme number sequences, such as [n, i, j, i, n, t, i, a, n, g, sh, a, n, h, e, h, a, o]. This phoneme sequence serves as the text feature part of the model input, entering the phoneme alignment model trained in the above embodiment for semantic representation generation. At the same time, server 104 can also prepare the corresponding speaker's identity number and extract its timbreembedding, i.e., timbre vector, from the speaker database, representing the user's desired pronunciation style. Subsequently, the phoneme encoder can convert the phoneme number sequence into a high-dimensional semantic vector, and through the attention mechanism module already learned within the alignment model, predict the alignment relationship between each spectral frame and the corresponding phoneme to obtain the target spectral information. The structural parameters of this attention module have been optimized using the aforementioned phoneme alignment model training method, and will not be elaborated here.

[0136] Subsequently, server 104 can input the target spectrum information into a trained vocoder, which converts the spectrum information into high-fidelity speech waveform data and outputs a natural speech audio file corresponding to the original text content.

[0137] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0138] Based on the same inventive concept, this application also provides a phoneme alignment model training device for implementing the phoneme alignment model training method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more phoneme alignment model training device embodiments provided below can be found in the limitations of the phoneme alignment model training method described above, and will not be repeated here.

[0139] In one exemplary embodiment, such as Figure 5 As shown, a phoneme alignment model training device is provided, comprising: a data acquisition module 502, a data alignment module 504, a first processing module 506, a first calculation module 508, a second processing module 510, a second calculation module 512, and a model training module 514, wherein:

[0140] The data acquisition module 502 is used to acquire speech training data and obtain spectral feature information and text feature information based on the speech training data; wherein, the spectral feature information includes multiple Mel spectrum frames and the text feature information includes multiple phonemes;

[0141] The data alignment module 504 is used to perform convolutional attention alignment on spectral feature information and text feature information to obtain the first alignment matrix;

[0142] The first processing module 506 is used to perform a monotonic alignment search based on the first alignment matrix to generate a second alignment matrix, wherein the second alignment matrix is ​​a binarized hard attention matrix.

[0143] The first calculation module 508 is used to calculate the relative entropy loss based on the first alignment matrix and the second alignment matrix;

[0144] The second processing module 510 is used to extend the text feature information to the length of the Mel spectrum frame according to the second alignment matrix, and to perform a linear transformation on the extended text feature information to generate a predicted Mel spectrum.

[0145] The second calculation module 512 is used to calculate the Mel loss based on the spectral feature information and the predicted Mel spectrum;

[0146] Model training module 514 is used to train a phoneme alignment model based on relative entropy loss and Mel loss until a preset convergence condition is met.

[0147] In one embodiment, the apparatus further includes: a first joint training module, configured to calculate a phoneme probability sequence based on a logarithmic probability distribution function on a first alignment matrix; the phoneme probability sequence is the logarithmic value of the phoneme classification probability corresponding to each Mel spectrum frame; calculate a connection temporal classification loss based on text feature information and the phoneme probability sequence; and train a phoneme alignment model based on the connection temporal classification loss, relative entropy loss, and Mel loss until a preset convergence condition is met.

[0148] In one embodiment, the method further includes: a second joint training module for generating predicted Mel band probabilities based on text feature information; the predicted Mel band probabilities include the classification probability distribution of each phoneme corresponding to the Mel band; calculating the cross-entropy loss between the predicted Mel band probabilities and the real Mel spectrum; and training a phoneme alignment model based on the cross-entropy loss, the connection temporal classification loss, the relative entropy loss, and the Mel loss until a preset convergence condition is met.

[0149] In one embodiment, the second joint training module is further configured to: map text feature information into a predicted Mel spectrum using a linear projection head; and calculate the predicted Mel band probability based on the predicted Mel spectrum using a log probability distribution function.

[0150] In one embodiment, the second processing module 510 is specifically used to: extract the duration of each phoneme according to the second alignment matrix; extend the text feature information to the length of the Mel spectrum frame according to the duration; and map the extended text feature information to the Mel spectrum frame dimension through the Mel projection head to generate a predicted Mel spectrum.

[0151] In one embodiment, the data alignment module 504 is specifically used to: calculate and generate a first alignment matrix based on the negative Euclidean distance of text feature information and spectral feature information.

[0152] Based on the same inventive concept, this application also provides a speech synthesis apparatus for implementing the speech synthesis method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more speech synthesis apparatus embodiments provided below can be found in the limitations of the speech synthesis method described above, and will not be repeated here.

[0153] In one exemplary embodiment, a speech synthesis device is provided, comprising: an information acquisition module, an information processing module, and a speech synthesis module, wherein:

[0154] The information acquisition module is used to acquire text information to be processed;

[0155] The information processing module is used to process the text information to be processed using the aforementioned phoneme alignment model to obtain the target spectrum information;

[0156] The speech synthesis module is used to obtain the target speech signal based on the target spectrum information.

[0157] Each module in the aforementioned phoneme alignment model training device and speech synthesis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0158] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data used in training phases such as speech training, or data used in the usage phase, such as text information to be processed. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a phoneme alignment model training method.

[0159] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a phoneme alignment model training method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0160] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0161] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: acquiring speech training data, and obtaining spectral feature information and text feature information based on the speech training data; wherein the spectral feature information includes multiple Mel spectrum frames, and the text feature information includes multiple phonemes; performing convolutional attention alignment on the spectral feature information and the text feature information to obtain a first alignment matrix; performing monotonic alignment search based on the first alignment matrix to generate a second alignment matrix, the second alignment matrix being a binarized hard attention matrix; calculating a relative entropy loss based on the first alignment matrix and the second alignment matrix; expanding the text feature information to the length of a Mel spectrum frame based on the second alignment matrix, and performing a linear transformation on the expanded text feature information to generate a predicted Mel spectrum; calculating a Mel loss based on the spectral feature information and the predicted Mel spectrum; and training a phoneme alignment model based on the relative entropy loss and the Mel loss until a preset convergence condition is met.

[0162] In one embodiment, when the processor executes the computer program, it further performs the following steps: calculating a phoneme probability sequence based on the logarithmic probability distribution function of the first alignment matrix; the phoneme probability sequence is the logarithmic value of the phoneme classification probability corresponding to each Mel spectrum frame; calculating the connection temporal classification loss based on the text feature information and the phoneme probability sequence; training the phoneme alignment model based on the connection temporal classification loss, relative entropy loss and Mel loss until the preset convergence condition is met.

[0163] In one embodiment, when the processor executes the computer program, it further performs the following steps: generating predicted Mel band probabilities based on text feature information; the predicted Mel band probabilities include the classification probability distribution of each phoneme's corresponding Mel band; calculating the cross-entropy loss between the predicted Mel band probabilities and the actual Mel spectrum; and training a phoneme alignment model based on the cross-entropy loss, connection-time classification loss, relative entropy loss, and Mel loss until a preset convergence condition is met.

[0164] In one embodiment, when the processor executes the computer program, it further performs the following steps: mapping text feature information to a predicted Mel spectrum using a linear projection head; and calculating the predicted Mel band probability based on the predicted Mel spectrum using a log probability distribution function.

[0165] In one embodiment, when the processor executes the computer program, it further performs the following steps: extracting the duration of each phoneme according to the second alignment matrix; extending the text feature information to the length of the Mel spectrum frame according to the duration; and mapping the extended text feature information to the Mel spectrum frame dimension through the Mel projection head to generate a predicted Mel spectrum.

[0166] In one embodiment, when the processor executes the computer program, it further performs the following steps: calculating and generating a first alignment matrix based on the negative Euclidean distance of text feature information and spectral feature information.

[0167] In one embodiment, when the processor executes the computer program, it further performs the following steps: acquiring text information to be processed; processing the text information to be processed using the above-described phoneme alignment model to obtain target spectrum information; and obtaining a target speech signal based on the target spectrum information.

[0168] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0169] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0170] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0171] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for training a phoneme alignment model, the method comprising: The method comprises: acquiring speech training data, and obtaining spectral feature information and text feature information according to the speech training data; wherein the spectral feature information comprises a plurality of mel-spectrum frames, and the text feature information comprises a plurality of phonemes; performing convolution attention alignment on the spectral feature information and the text feature information to obtain a first alignment matrix; generating a second alignment matrix based on the first alignment matrix by performing monotonic alignment search, the second alignment matrix being a binary hard attention matrix; calculating a relative entropy loss according to the first alignment matrix and the second alignment matrix; extending the text feature information to the length of a mel-spectrum frame according to the second alignment matrix, and generating predicted mel-spectrum by performing linear transformation on the extended text feature information; calculating a mel loss according to the spectral feature information and the predicted mel-spectrum; training a phoneme alignment model according to the relative entropy loss and the mel loss until a preset convergence condition is met; The method further comprises: calculating a phoneme probability sequence based on the first alignment matrix according to a logarithmic probability distribution function; the phoneme probability sequence being a logarithmic value of phoneme classification probability corresponding to each mel-spectrum frame; calculating a connection time sequence classification loss according to the text feature information and the phoneme probability sequence; training the phoneme alignment model based on the connection time sequence classification loss, the relative entropy loss and the mel loss until the preset convergence condition is met.

2. The method of claim 1, wherein, The method further comprises: generating a predicted mel-frequency band probability based on the text feature information; the predicted mel-frequency band probability comprising classification probability distribution of a mel-frequency band corresponding to each phoneme; calculating a cross-entropy loss of the predicted mel-frequency band probability and real mel-spectrum; training the phoneme alignment model based on the cross-entropy loss, the connection time sequence classification loss, the relative entropy loss and the mel loss until the preset convergence condition is met.

3. The method of claim 2, wherein, The generating of the predicted mel-frequency band probability based on the text feature information comprises: mapping the text feature information to the predicted mel-spectrum through a linear projection head; calculating a predicted mel-frequency band probability based on the predicted mel-spectrum according to the logarithmic probability distribution function.

4. The method according to any one of claims 1 to 3, characterized in that, The extending of the text feature information to the length of a mel-spectrum frame according to the second alignment matrix, and the generating of predicted mel-spectrum by performing linear transformation on the extended text feature information, comprises: extracting the duration of each phoneme according to the second alignment matrix; extending the text feature information to the length of a mel-spectrum frame according to the duration; mapping the extended text feature information to the dimension of a mel-spectrum frame through a mel projection head to generate predicted mel-spectrum.

5. The method according to any one of claims 1 to 3, characterized in that, The performing of convolution attention alignment on the spectral feature information and the text feature information to obtain a first alignment matrix comprises: generating a first alignment matrix based on the negative Euclidean distance calculation of the text feature information and the spectral feature information.

6. A speech synthesis method characterized by, The method comprises: acquiring text information to be processed; processing the text information to be processed using the phoneme alignment model training method according to any one of claims 1 to 5 to obtain target spectral information; The target speech signal is obtained according to the target spectrum information.

7. A phoneme alignment model training device, characterized in that, The device comprises: a data acquisition module, configured to acquire speech training data, and obtain spectrum feature information and text feature information according to the speech training data; wherein the spectrum feature information comprises a plurality of mel spectrum frames, and the text feature information comprises a plurality of phonemes; a data alignment module, configured to perform convolution attention alignment on the spectrum feature information and the text feature information to obtain a first alignment matrix; a first processing module, configured to perform monotonic alignment search based on the first alignment matrix to generate a second alignment matrix, the second alignment matrix being a binary hard attention matrix; a first calculation module, configured to calculate a relative entropy loss according to the first alignment matrix and the second alignment matrix; a second processing module, configured to extend the text feature information to the length of a mel spectrum frame according to the second alignment matrix, and perform linear transformation on the extended text feature information to generate predicted mel spectrum; a second calculation module, configured to calculate a mel loss according to the spectrum feature information and the predicted mel spectrum; a model training module, configured to train a phoneme alignment model according to the relative entropy loss and the mel loss until a preset convergence condition is met; a first joint training module, configured to calculate a phoneme probability sequence based on the first alignment matrix using a logarithmic probability distribution function; the phoneme probability sequence being a logarithmic value of phoneme classification probability corresponding to each mel spectrum frame; calculate a connection time sequence classification loss according to the text feature information and the phoneme probability sequence; and train the phoneme alignment model based on the connection time sequence classification loss, the relative entropy loss and the mel loss until the preset convergence condition is met.

8. A speech synthesis apparatus characterized by comprising: The device comprises: an information acquisition module, configured to acquire text information to be processed; an information processing module, configured to process the text information to be processed using the phoneme alignment model training method according to any one of claims 1 to 5 to obtain target spectrum information; a speech synthesis module, configured to obtain a target speech signal according to the target spectrum information. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 6.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 6.

Images