Phoneme alignment method, system, electronic device and storage medium

Abstract

The present invention provides a phoneme alignment method, system, electronic device and storage medium, wherein the method comprises: inputting the initial audio features of the target audio and the initial phoneme features of the target text into a feature extraction module in a phoneme alignment model to obtain the target audio features of the target audio and the target phoneme features of the target text; inputting the target audio features and the target phoneme features into an attention module in the phoneme alignment model to obtain a phoneme alignment matrix; according to the phoneme alignment matrix, obtaining the predicted values ​​of the phoneme sequence labels corresponding to the initial audio features, and performing phoneme alignment processing on the target audio and the target text based on the predicted values ​​of the phoneme sequence labels. The present invention realizes stable and accurate prediction of phoneme sequence labels through refined feature extraction and attention calculation, thereby effectively improving the robustness and generalization of phoneme alignment.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a phoneme alignment method, system, electronic device and storage medium. Background Art

[0002] Phoneme alignment, also known as phoneme duration prediction, aims to predict the duration of phonemes in speech features. This helps acoustic models output the corresponding phoneme numbers alongside speech features, enabling features like resume playback and lip syncing with digital humans. Therefore, accurate and stable phoneme alignment is a critical research topic currently under urgent scrutiny in the industry.

[0003] Current phoneme alignment models typically implicitly generate the duration of phonemes through a sequence generation mechanism, or explicitly predict the number of frames corresponding to phonemes through a simple convolutional neural network. These models have difficulty accurately capturing the phoneme correspondence between audio and text, resulting in poor robustness and generalization of phoneme alignment. Summary of the Invention

[0004] The present invention provides a phoneme alignment method, system, electronic device and storage medium to address the defects in the prior art that it is difficult to accurately capture the phoneme correspondence between audio and text, resulting in poor robustness and generalization of phoneme alignment, thereby improving the robustness and generalization of phoneme alignment.

[0005] The present invention provides a phoneme alignment method, comprising:

[0006] Inputting the initial audio features of the target audio and the initial phoneme features of the target text into a feature extraction module in the phoneme alignment model to obtain the target audio features of the target audio and the target phoneme features of the target text;

[0007] Inputting the target audio feature and the target phoneme feature into the attention module in the phoneme alignment model to obtain a phoneme alignment matrix;

[0008] According to the phoneme alignment matrix, obtaining the phoneme sequence number label prediction value corresponding to the initial audio feature, and performing phoneme alignment processing on the target audio and the target text according to the phoneme sequence number label prediction value;

[0009] The phoneme alignment model is trained based on sample audio, sample text, and the phoneme sequence label values ​​corresponding to the sample text.

[0010] According to a phoneme alignment method provided by the present invention, the initial audio features of the target audio and the initial phoneme features of the target text are input into a feature extraction module in a phoneme alignment model to obtain target audio features of the target audio and target phoneme features of the target text, including:

[0011] Inputting the initial audio features of the target audio into the audio feature extraction branch unit in the feature extraction module to obtain the target audio features;

[0012] The initial phoneme features of the target text are input into the phoneme feature extraction branch unit in the feature extraction module to obtain the target phoneme features.

[0013] According to a phoneme alignment method provided by the present invention, inputting the initial audio features of the target audio into the audio feature extraction branch unit in the feature extraction module to obtain the target audio features includes:

[0014] When the inference mode corresponding to the target audio is a streaming inference mode, dividing the initial audio features of the target audio into a plurality of initial sub-features;

[0015] Inputting each of the initial sub-features into the audio feature extraction branch unit to obtain a target sub-feature corresponding to each of the initial sub-features;

[0016] determining the target audio feature according to the plurality of target sub-features;

[0017] The obtaining, according to the phoneme alignment matrix, a phoneme sequence number label prediction value corresponding to the initial audio feature, includes:

[0018] Obtaining a label prediction subvalue corresponding to each of the initial sub-features according to a sub-matrix corresponding to each of the initial sub-features in the phoneme alignment matrix;

[0019] The label prediction sub-values ​​corresponding to all initial sub-features in the initial audio feature of the target audio are combined to obtain the phoneme sequence number label prediction value corresponding to the initial audio feature.

[0020] According to a phoneme alignment method provided by the present invention, inputting the initial phoneme features of the target text into the phoneme feature extraction branch unit in the feature extraction module to obtain the target phoneme features includes:

[0021] Inputting the initial phoneme features of the target text into the encoding layer in the phoneme feature extraction branch unit, and adding context-aware information to the initial phoneme features by the encoding layer to obtain phoneme encoding features;

[0022] The phoneme coding feature is input into the linear layer in the phoneme feature extraction branch unit, and the linear layer performs scale transformation on the phoneme coding feature to obtain the target phoneme feature.

[0023] According to a phoneme alignment method provided by the present invention, the inputting the target audio feature and the target phoneme feature into the attention module in the phoneme alignment model to obtain a phoneme alignment matrix includes:

[0024] The target audio feature is used as a query parameter, and the target phoneme feature is used as a key parameter, which are input into the attention module. The attention module performs cross-attention calculation on the target audio feature and the target phoneme feature to obtain the phoneme alignment matrix.

[0025] According to a phoneme alignment method provided by the present invention, the phoneme sequence number label value corresponding to the sample text is obtained based on the following steps:

[0026] Obtaining the duration and sequence number information of each phoneme in the sample phoneme sequence corresponding to the sample text;

[0027] performing frame number conversion on the duration of each phoneme in the sample phoneme sequence according to the resolution value corresponding to the initial audio feature of the sample audio, to obtain the duration frame number corresponding to each phoneme in the sample phoneme sequence;

[0028] Each phoneme in the sample phoneme sequence is labeled according to the sequence number information and the number of continuous frames to obtain a phoneme sequence number label value corresponding to the sample text.

[0029] According to a phoneme alignment method provided by the present invention, labeling each phoneme in the sample phoneme sequence according to the sequence number information and the number of continuous frames to obtain a phoneme sequence number label value corresponding to the sample text includes:

[0030] Determining the number of label dimensions corresponding to each phoneme in the sample phoneme sequence according to the number of continuous frames;

[0031] Determining, according to the sequence number information, the values ​​of the elements of each marking dimension corresponding to each phoneme in the sample phoneme sequence;

[0032] According to the value, each phoneme in the sample phoneme sequence is labeled to obtain the phoneme sequence label value corresponding to the sample text.

[0033] The present invention also provides a phoneme alignment system, comprising:

[0034] A first feature extraction unit is configured to input the initial audio features of the target audio and the initial phoneme features of the target text into a feature extraction module in the phoneme alignment model to obtain target audio features of the target audio and target phoneme features of the target text;

[0035] A second feature extraction unit is configured to input the target audio feature and the target phoneme feature into an attention module in the phoneme alignment model to obtain a phoneme alignment matrix;

[0036] A phoneme alignment unit, configured to obtain, based on the phoneme alignment matrix, a phoneme sequence number label prediction value corresponding to the initial audio feature, and perform phoneme alignment processing on the target audio and the target text based on the phoneme sequence number label prediction value;

[0037] The phoneme alignment model is trained based on sample audio, sample text, and the phoneme sequence label values ​​corresponding to the sample text.

[0038] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-described phoneme alignment methods is implemented.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the above-mentioned phoneme alignment methods when executed by a processor.

[0040] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned phoneme alignment methods.

[0041] The phoneme alignment method, system, electronic device and storage medium provided by the present invention obtain more accurate target audio features and target phoneme features by inputting the initial audio features of the target audio and the initial phoneme features of the target text into the feature extraction module in the phoneme alignment model, and then inputting the target audio features and the target phoneme features into the attention module to generate a phoneme alignment matrix, and obtaining the phoneme sequence label prediction value corresponding to the initial audio features through the phoneme alignment matrix, thereby reducing the prediction difficulty of the model and increasing the robustness of the model by converting the phoneme duration and position prediction into the phoneme label prediction, while performing accurate phoneme alignment processing between the target audio and the target text through the feature extraction module and the attention module, thereby increasing the generalization of the model, thereby effectively improving the robustness and generalization of the phoneme alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 It is a flowchart of the phoneme alignment method provided by the present invention.

[0044] Figure 2 It is a flowchart of the training steps of the phoneme alignment model provided by the present invention.

[0045] Figure 3 This is one of the flow charts of the phoneme sequence number labeling step provided by the present invention.

[0046] Figure 4 This is the second flow chart of the phoneme serial number labeling step provided by the present invention.

[0047] Figure 5 Schematic diagram of the structure of the phoneme alignment system provided by the present invention.

[0048] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0050] Phoneme alignment is generally used in acoustic models or non-autoregressive speech synthesis models. Acoustic models accept speech feature input and generate speech features such as mel-spectrograms. In this process, the input and output durations are the same. Non-autoregressive speech synthesis models accept phoneme input and output speech features such as mel-spectrograms. Therefore, a phoneme duration prediction model is required to align the lengths of phoneme features with the lengths of speech features.

[0051] Existing phoneme alignment models fall into two paradigms: autoregressive and non-autoregressive. Autoregressive models, such as the Tacotron model, implicitly align text and phoneme features through a sequence generation mechanism. This model does not require duration labels and supports streaming inference. However, it is difficult to integrate into other models as a functional model. It struggles to accurately capture the phoneme correspondence between audio and text, and is prone to missing or repeating phonemes, resulting in poor robustness and generalization of phoneme alignment.

[0052] In non-autoregressive models, for example, the duration prediction module in the Fast, Robust and Controllable Text to Speech (Fastspeech) model predicts the corresponding number of audio frames through a convolutional network, and the Variational Inference with Adversarial Learning for End-to-End Text-to-Speech (VITS) model predicts the duration of each phoneme through a dynamic programming algorithm. However, such methods have difficulty supporting streaming reasoning and are limited by the acquisition of global information. Therefore, it is difficult to accurately capture the phoneme correspondence between audio and text, resulting in poor robustness and generalization of phoneme alignment.

[0053] In order to solve the above problems, this embodiment provides a phoneme alignment method, which obtains more accurate target audio features and target phoneme features by inputting the initial audio features of the target audio and the initial phoneme features of the target text into the feature extraction module in the phoneme alignment model. Subsequently, the target audio features and the target phoneme features are input into the attention module to generate a phoneme alignment matrix, so as to obtain the phoneme sequence label prediction value corresponding to the initial audio features through the phoneme alignment matrix. By converting the phoneme duration and position prediction into the phoneme label prediction, the prediction difficulty of the model is reduced, and the robustness of the model is increased. At the same time, the feature extraction module and the attention module are used to perform accurate phoneme alignment processing between the target audio and the target text, thereby increasing the generalization of the model, thereby effectively improving the robustness and generalization of the phoneme alignment. Compared with existing autoregressive models, this method is easier to integrate into other models as a functional model, and may have better accuracy and robustness, avoiding the problem of missing or repeating a phoneme. Compared with non-autoregressive models, this method can more accurately capture the phoneme correspondence between audio and text while supporting streaming reasoning, thereby improving the robustness and generalization of phoneme alignment.

[0054] It should be noted that the phoneme alignment method provided in this application can be widely applied to various audio processing scenarios, such as phoneme alignment scenarios that can be adapted to acoustic models or vocoders that output audio features, and this embodiment does not specifically limit this. The method has strong robustness and generalization. By combining it with acoustic models or vocoders in different audio processing scenarios, it can enable the acoustic model or vocoder to output the phoneme number corresponding to the feature while outputting the speech feature, thereby realizing functions such as breakpoint resumption and digital human lip syncing.

[0055] Figure 1 Schematic diagram of the process of the phoneme alignment method provided by the present invention. Figure 1 As shown, the method includes step 110 , step 120 and step 130 .

[0056] Step 110: Inputting the initial audio features of the target audio and the initial phoneme features of the target text into a feature extraction module in a phoneme alignment model to obtain target audio features of the target audio and target phoneme features of the target text;

[0057] Step 120: Input the target audio features and the target phoneme features into the attention module in the phoneme alignment model to obtain a phoneme alignment matrix; wherein, the phoneme alignment model is trained based on sample audio, sample text, and the phoneme serial number label marking value corresponding to the sample text.

[0058] The phoneme alignment model here supports the insertion of other acoustic models or vocoders, and supports both streaming and non-streaming inference modes. This phoneme alignment model can accurately predict the phoneme sequence label prediction value corresponding to the initial audio features based on the initial audio features and initial phoneme features.

[0059] Figure 2 It is a flowchart of the training steps of the phoneme alignment model provided by the present invention.

[0060] Optionally, before executing step 110, a phoneme alignment model may be obtained through training. Figure 2 As shown, specifically, an initialization prediction model can be constructed first, and the initialization prediction model here can include an initial feature extraction module and an initial attention module.

[0061] The initial feature extraction module can be a model prepared for audio feature extraction and phoneme feature extraction after parameter initialization, or it can be a pre-trained model with audio feature extraction and phoneme feature extraction functions; the initial attention module can be a model prepared for cross-attention processing of audio features and phoneme features after parameter initialization, or it can be a pre-trained language model with cross-attention processing functions of audio features and phoneme features. This embodiment does not make specific limitations on this.

[0062] The initial feature extraction module may be an integrated feature extraction module that can further extract audio features and phoneme features, or it may be an initial audio feature extraction branch unit that has the function of further extracting audio features and an initial phoneme feature extraction branch unit that has the function of further extracting phoneme features. This embodiment does not specifically limit this.

[0063] Each branch unit in the initial audio feature extraction branch unit and the initial phoneme feature extraction branch unit may be a branch unit with the same or different structures formed by network layers constructed based on one or more network models. For example, each branch unit may be constructed based on an initial coding layer and an initial linear layer. The initial coding layer here may be initialized based on a transformer model or a convolution-enhanced transformer model, and this embodiment does not specifically limit this.

[0064] The following describes the training steps of the phoneme alignment model provided in this embodiment by taking the initial feature extraction module as a multi-task branch model including an initial audio feature extraction branch unit and an initial phoneme feature extraction branch unit, and each branch unit as a branch network layer constructed based on the initial coding layer and the initial linear layer as an example.

[0065] In addition, sample audio and sample text can be collected and the sample text can be labeled with phoneme serial number tags to obtain the phoneme serial number tag value corresponding to the sample text. The phoneme serial number tag value here can be a label formed by encoding the serial number and duration of each phoneme in the sample text.

[0066] Subsequently, the initialization prediction model can be iteratively trained based on the initial audio features of the sample audio, the initial phoneme features of the sample text, and the phoneme serial number label tag value corresponding to the sample text to obtain a phoneme alignment model that can accurately predict the phoneme serial number label. The iterative training here can be to train and infer the initialization prediction model as an independent model, or to deploy it into the initial acoustic model as a module to add a phoneme alignment function to the initial acoustic model, and to implement it through joint training and inference with the initial acoustic model. This embodiment does not specifically limit this.

[0067] For example, when the initialization prediction model is trained and inferred as an independent model, the initial audio features of the sample audio and the initial phoneme features of the sample text can be used as inputs of the initialization prediction model, and the initial audio feature extraction branch unit of the initial feature extraction module in the initialization prediction model performs feature extraction on the initial audio features of the sample audio to obtain sample audio features, and the initial phoneme feature extraction branch unit of the initial feature extraction module in the initialization prediction model performs feature extraction on the initial phoneme features of the sample text to obtain sample phoneme features.

[0068] The initial audio features of the sample audio here can be audio features obtained by extracting features from the sample audio based on an audio feature extraction algorithm, such as Mel Frequency Cepstrum Coefficient (MFCC) features, spectral centroid, or Perceptual Linear Predictive (PLP) features, etc., or can be intermediate features obtained by extracting features from the sample audio based on a pre-trained deep learning model with audio feature extraction, such as an acoustic model, etc. This embodiment does not specifically limit this.

[0069] The target text here is the content description text corresponding to the target audio. The initial phoneme features of the target text can be obtained by first converting the text into a phoneme sequence using a phoneme annotation tool and then performing phoneme feature extraction.

[0070] For example, for sample audio feature extraction, the initial encoding layer in the initial audio feature extraction branch unit may encode the initial audio feature in combination with the context perception information to obtain the sample audio coding feature, and then the initial linear layer in the initial audio feature extraction branch unit may further transform the sample audio coding feature with information such as scale to transform the sample audio coding feature to a certain feature dimension to obtain the sample audio feature. For example, the dimension of the sample audio coding feature A is After the initial linear layer processing in the initial audio feature extraction branch unit, the sample audio coding feature A can be transformed into a unified dimension , get the sample audio features . Where N is the number of frames of the initial speech feature, and H are the dimensions before and after the transformation, respectively.

[0071] Similarly, for sample phoneme feature extraction, the initial encoding layer in the initial phoneme feature extraction branch unit can encode the initial phoneme feature in combination with the context perception information to obtain the sample phoneme coding feature, and then the initial linear layer in the initial phoneme feature extraction branch unit can further transform the sample phoneme coding feature with information such as scale to transform the sample phoneme coding feature to a certain feature dimension to obtain the sample phoneme feature. For example, the dimension of the sample phoneme coding feature P is After the initial linear layer processing in the initial phoneme feature extraction branch unit, the sample phoneme encoding feature P can be transformed into a uniform dimension , get the sample phoneme features . Where M is the length of the initial phoneme feature, and H are the dimensions before and after the transformation, respectively.

[0072] After obtaining the sample audio features and sample phoneme features, the initial attention module in the initialization prediction model performs attention calculations on the sample audio features and sample phoneme features to obtain a sample phoneme alignment matrix. Finally, the sample phoneme alignment matrix V and the phoneme sequence label values ​​corresponding to the sample text are input into the cross-entropy loss function to calculate the target loss value. The initialization prediction model is iteratively trained based on the target loss value, thereby obtaining a phoneme alignment model that can accurately predict phoneme sequence labels.

[0073] For example, the sample phoneme alignment matrix can be obtained by the initial attention module with the sample audio features is the query parameter (query), with sample phoneme features Perform cross-attention calculation on the key parameter (key) to obtain the value parameter (Value), which is used as the sample phoneme alignment matrix. The specific calculation formula of the value parameter is as follows:

[0074] ;

[0075] in, is the sample phoneme alignment matrix, the dimension of V is , where N is the number of frames of the initial speech feature and M is the length of the initial phoneme feature.

[0076] After the phoneme alignment model is obtained through training, since different reasoning modes correspond to different reasoning processes in actual applications, and thus the model inputs are also different, the initial audio features of the target audio and the initial phoneme features of the target text can be determined based on the actual reasoning mode corresponding to the actual application scenario to determine the first input information corresponding to the actual reasoning mode. The determined first input information is then input into a feature extraction module in the phoneme alignment model, which further extracts audio features and phoneme features based on the first input information, thereby obtaining target audio features of the target audio and target phoneme features of the target text.

[0077] Exemplarily, for determining the first input information, if the actual reasoning mode is a non-streaming reasoning mode, the initial audio features of the target audio are directly combined with the initial phoneme features of the target text to obtain the first input information; if the actual reasoning mode is a streaming reasoning mode, the initial audio features of the target audio are split and combined with the initial phoneme features of the target text to obtain the first input information.

[0078] The initial audio features of the target audio here can also be audio features obtained by extracting features of the target audio based on an audio feature extraction algorithm, such as Mel Frequency Cepstrum Coefficient (MFCC) features, spectral centroid, or Perceptual Linear Predictive (PLP) features, or can be obtained by extracting features based on a pre-trained deep learning model with audio feature extraction, such as an acoustic model. This embodiment does not specifically limit this.

[0079] The target text here is the content description text corresponding to the target audio. The initial phoneme features of the target text can also be obtained by first converting the text into a phoneme sequence using a phoneme annotation tool and then performing phoneme feature extraction.

[0080] The feature extraction module here is constructed after training the initial feature extraction module, and its structure is the same as that of the initial feature extraction module. It can also be an integrated feature extraction module that can further extract audio features and phoneme features, or it can be an audio feature extraction branch unit with the function of further extracting audio features and a phoneme feature extraction branch unit with the function of further extracting phoneme features. This embodiment does not make specific limitations on this.

[0081] After obtaining the target audio features of the target audio and the target phoneme features of the target text through the feature extraction module, input information of the target audio features and the target phoneme features of the target text can also be determined based on the actual inference mode to determine second input information corresponding to the actual inference mode. The determined second input information is then input into the attention module in the phoneme alignment model, and the attention module performs a cross-attention calculation based on the second input information to obtain a phoneme alignment matrix between the initial audio features and the initial phoneme features, i.e., a phoneme alignment matrix.

[0082] Exemplarily, for the determination of the second input information, if the actual reasoning mode is a non-streaming reasoning mode, the target audio feature of the target audio is an overall feature. At this time, the initial audio feature of the target audio can be directly combined with the initial phoneme feature of the target text to obtain the second input information; if the actual reasoning mode is a streaming reasoning mode, the target audio feature of the target audio contains multiple target sub-features. At this time, each target sub-feature can be combined with the initial phoneme feature of the target text to obtain the second input information.

[0083] Step 130: Obtain the predicted values ​​of the phoneme serial number labels corresponding to the initial audio features according to the phoneme alignment matrix, and perform phoneme alignment processing on the target audio and the target text according to the predicted values ​​of the phoneme serial number labels.

[0084] Optionally, after the phoneme alignment matrix is ​​predicted by the phoneme alignment model, the phoneme alignment matrix can be converted according to the conversion process corresponding to the actual inference mode to obtain the phoneme serial label prediction value corresponding to the initial audio feature. By converting the predicted phoneme duration and position into the serial label of the predicted phoneme, the prediction difficulty of the model can be reduced, and continuous identical phonemes can be distinguished, thereby effectively increasing the robustness of the phoneme alignment.

[0085] After obtaining the predicted value of the phoneme serial number label corresponding to the initial audio feature, the target audio and target text can be phoneme aligned based on the serial number information and duration of the phoneme corresponding to the predicted value of the phoneme serial number label corresponding to the initial audio feature.

[0086] Among them, different inference modes correspond to different conversion processes. If the actual inference mode is a non-streaming inference mode, the phoneme alignment matrix can be directly converted, such as using the Argmax function to take the maximum index of the elements under each frame dimension in the phoneme alignment matrix to obtain the phoneme sequence label prediction value corresponding to the initial audio feature. , the specific calculation formula can be expressed as:

[0087] ;

[0088] If the actual inference mode is a streaming inference mode, each sub-matrix in the phoneme alignment matrix can be transformed, such as by using the Argmax function to perform index processing on the elements under each frame dimension in the combination matrix and then combining them to obtain the predicted value of the phoneme serial number label corresponding to the initial audio feature.

[0089] The method provided in this embodiment obtains more accurate target audio features and target phoneme features by inputting the initial audio features of the target audio and the initial phoneme features of the target text into the feature extraction module in the phoneme alignment model, and then inputs the target audio features and the target phoneme features into the attention module to generate a phoneme alignment matrix, and obtains the phoneme serial number label prediction value corresponding to the initial audio features through the phoneme alignment matrix. In this way, by converting the phoneme duration and position prediction into the phoneme label prediction, the prediction difficulty of the model is reduced and the robustness of the model is increased. At the same time, the accurate phoneme alignment processing between the target audio and the target text is performed through the feature extraction module and the attention module, which increases the generalization of the model, thereby effectively improving the robustness and generalization of the phoneme alignment.

[0090] In some embodiments, step 110 specifically includes:

[0091] Inputting the initial audio features of the target audio into the audio feature extraction branch unit in the feature extraction module to obtain the target audio features;

[0092] The initial phoneme features of the target text are input into the phoneme feature extraction branch unit in the feature extraction module to obtain the target phoneme features.

[0093] Optionally, when the feature extraction module is a model with multiple tasks including an audio feature extraction branch unit and a phoneme feature extraction branch unit, the initial audio features of the target audio can be input into the audio feature extraction branch unit, so that the audio feature extraction branch unit can perform further feature extraction on the initial audio features of the target audio to obtain the target audio features of the target audio; and the initial phoneme features of the target text can be input into the phoneme feature extraction branch unit in the feature extraction module, so that the phoneme feature extraction branch unit can perform further feature extraction on the initial phoneme features of the target phonemes to obtain the target phoneme features of the target text.

[0094] Each feature extraction branch unit in the feature extraction module here is constructed based on the training of each initial feature extraction branch unit in the initial feature extraction module, and its structure is the same as that of each initial feature extraction branch unit in the initial feature extraction module, that is, each feature extraction branch unit in the feature extraction module can be a branch unit with the same or different structure formed by a network layer constructed based on one or more network models, such as each branch unit is constructed based on a coding layer and a linear layer. The coding layer here can be constructed based on a transformer model (Transformer) or a convolution-enhanced transformer model (Conformer), and this embodiment does not specifically limit this.

[0095] The method provided in this embodiment inputs the initial audio features of the target audio and the initial phoneme features of the target text into different branch units in the feature extraction module, so that the different branch units can perform refined feature extraction on the corresponding input data to obtain more accurate target audio features and target phoneme features, thereby making the audio and text more matched at the feature level, thereby improving the accuracy of phoneme alignment and further improving the performance of speech recognition and speech synthesis.

[0096] In some embodiments, inputting the initial audio features of the target audio into the audio feature extraction branch unit in the feature extraction module to obtain the target audio features includes:

[0097] When the inference mode corresponding to the target audio is a streaming inference mode, dividing the initial audio features of the target audio into a plurality of initial sub-features;

[0098] Inputting each of the initial sub-features into the audio feature extraction branch unit to obtain a target sub-feature corresponding to each of the initial sub-features;

[0099] determining the target audio feature according to the plurality of target sub-features;

[0100] The obtaining, according to the phoneme alignment matrix, a phoneme sequence number label prediction value corresponding to the initial audio feature, includes:

[0101] Obtaining a label prediction subvalue corresponding to each of the initial sub-features according to a sub-matrix corresponding to each of the initial sub-features in the phoneme alignment matrix;

[0102] The label prediction sub-values ​​corresponding to all initial sub-features in the initial audio feature of the target audio are combined to obtain the phoneme sequence number label prediction value corresponding to the initial audio feature.

[0103] Optionally, since the phoneme alignment model supports streaming reasoning mode and non-streaming reasoning mode. Therefore, in practical applications, it is possible to first determine whether the reasoning mode corresponding to the target audio is a streaming reasoning mode or a non-streaming reasoning mode. If it is determined that the reasoning mode corresponding to the target audio is a streaming reasoning mode, the initial audio features of the target audio are split into arbitrary sizes to obtain multiple initial sub-features, and each initial sub-feature is input into the audio feature extraction branch unit respectively, so that the audio feature extraction branch unit performs feature extraction on each initial sub-feature respectively to obtain each target sub-feature to form a target audio feature containing multiple sub-features, and in the subsequent reasoning process, each target sub-feature is combined with the target phoneme feature and input into the attention module for cross-attention calculation to obtain a sub-matrix corresponding to each initial sub-feature to form a phoneme alignment matrix containing multiple sub-matrices, and each sub-matrix is ​​converted separately to obtain the label prediction sub-value corresponding to each sub-matrix, and then the label prediction sub-values ​​corresponding to all initial sub-features in the initial audio feature of the target audio are combined to obtain the phoneme serial number label prediction value corresponding to the initial audio feature.

[0104] If it is determined that the reasoning mode corresponding to the target audio is a non-streaming reasoning mode, the initial audio features of the target audio are directly input as a whole into the audio feature extraction branch unit, so that the audio feature extraction branch unit extracts the initial audio features of the target audio as a whole to obtain the target audio features. In the subsequent reasoning process, the target audio features as a whole are combined with the target phoneme features as input to the attention module for cross-attention calculation to obtain the phoneme alignment matrix, and the phoneme alignment matrix is ​​transformed as a whole to obtain the phoneme serial number label prediction value corresponding to the initial audio features.

[0105] The so-called cross-attention calculation can be a matrix multiplication calculation.

[0106] It should be noted that the phoneme alignment matrix obtained after the initial audio features of the target audio and the initial phoneme features of the target text enter the attention module through the feature extraction module is It can be expressed as Therefore, according to the matrix block rule, after the initial audio features are segmented and passed through the feature extraction module and then into the attention module, the phoneme alignment matrix is ​​obtained It can be expressed as follows:

[0107] ;

[0108] in, The encoding features of each initial sub-feature corresponding to the initial audio feature of the target audio; The encoding features corresponding to the initial phoneme features of the target text; is the submatrix corresponding to each initial sub-feature corresponding to the initial audio feature; and is the weight coefficient.

[0109] Therefore, in the streaming inference mode, the initial audio features are segmented into arbitrary sizes, the results are concatenated after calculation, and the resulting phoneme sequence label prediction values ​​are exactly the same as those in the non-streaming inference mode. This ensures the consistency and stability of phoneme alignment under different inference modes, thereby improving the robustness, generalization, and flexibility of phoneme alignment.

[0110] The method provided in this embodiment achieves flexible processing of target audio features by supporting streaming inference mode and non-streaming inference mode. Regardless of whether the audio data is input in streaming or holistic form, it can accurately perform phoneme alignment and ensure the consistency and stability of phoneme alignment under different inference modes, thereby significantly improving the robustness, generalization and flexibility of phoneme alignment.

[0111] In some embodiments, inputting the initial phoneme features of the target text into the phoneme feature extraction branch unit in the feature extraction module to obtain the target phoneme features includes:

[0112] Inputting the initial phoneme features of the target text into the encoding layer in the phoneme feature extraction branch unit, and adding context-aware information to the initial phoneme features by the encoding layer to obtain phoneme encoding features;

[0113] The phoneme coding feature is input into the linear layer in the phoneme feature extraction branch unit, and the linear layer performs scale transformation on the phoneme coding feature to obtain the target phoneme feature.

[0114] The phoneme feature extraction branch unit here can be constructed by a coding layer and a linear layer, wherein the coding layer is used to perform feature encoding in combination with context-aware information; and the linear layer is used to perform feature dimension transformation.

[0115] Optionally, when extracting the target phoneme features, the initial phoneme features of the target text may be input into an encoding layer in the phoneme feature extraction branch unit, so that the encoding layer adds context-aware information to the initial phoneme features for feature encoding to obtain phoneme encoding features. After obtaining the phoneme encoding features, the phoneme encoding features may be input into a linear layer in the phoneme feature extraction branch unit, so that the linear layer scales the phoneme encoding features to obtain target phoneme features with normalized feature dimensions.

[0116] The method provided in this embodiment realizes the addition of context-aware information and feature dimension transformation of the initial phoneme features of the target text by constructing a phoneme feature extraction branch unit including an encoding layer and a linear layer, thereby extracting more accurate target phoneme features. It not only improves the accuracy and efficiency of phoneme alignment, but also realizes the main function of phoneme alignment by combining linear transformation and matrix multiplication of cross-attention calculation, so that the phoneme alignment model has the flexible switching between streaming mode reasoning and non-streaming mode reasoning, which effectively improves the flexibility and practicality of phoneme alignment reasoning.

[0117] Similarly, when extracting target audio features, the steps of extracting target phoneme features may also be referred to and implemented, which will not be repeated here.

[0118] In some embodiments, step 120 specifically includes:

[0119] The target audio feature is used as a query parameter, and the target phoneme feature is used as a key parameter, which are input into the attention module. The attention module performs cross-attention calculation on the target audio feature and the target phoneme feature to obtain the phoneme alignment matrix.

[0120] Optionally, when performing a phoneme alignment matrix, the target audio feature can be used as a query parameter, and the target phoneme feature can be input into the attention module as a key parameter, so that the attention module can perform cross-attention calculations on the target audio feature and the target phoneme feature to obtain a phoneme alignment matrix between the initial audio feature and the initial phoneme feature, that is, a phoneme alignment matrix.

[0121] The cross attention calculation here can be a matrix multiplication calculation.

[0122] The method provided in this embodiment inputs the target audio features as query parameters and the target phoneme features as key parameters into the attention module for cross-attention calculation, so as to fuse the information of the target audio features and the target phoneme features through attention to predict the phoneme serial number label prediction value, thereby effectively realizing the alignment between the audio features and the phoneme features, and improving the accuracy, efficiency and generalization of the phoneme alignment.

[0123] In some embodiments, the phoneme serial number tag value corresponding to the sample text is obtained based on the following steps:

[0124] Obtaining the duration and sequence number information of each phoneme in the sample phoneme sequence corresponding to the sample text;

[0125] performing frame number conversion on the duration of each phoneme in the sample phoneme sequence according to the resolution value corresponding to the initial audio feature of the sample audio, to obtain the duration frame number corresponding to each phoneme in the sample phoneme sequence;

[0126] Each phoneme in the sample phoneme sequence is labeled according to the sequence number information and the number of continuous frames to obtain a phoneme sequence number label value corresponding to the sample text.

[0127] Figure 3 This is one of the flow charts of the phoneme serial number labeling step provided by the present invention; Figure 4 This is the second flow chart of the phoneme serial number labeling step provided by the present invention.

[0128] like Figure 3 As shown, when labeling the phoneme sequence corresponding to the sample text, a sample phoneme sequence corresponding to the sample text may be generated according to a pre-defined conversion rule. For example, the text of the sample text may be converted to pinyin and then split to form the sample phoneme sequence corresponding to the sample text. For example, if the sample text is "today's weather", the corresponding sample phoneme sequence may be represented as "j in t ian t ian qi".

[0129] After obtaining the sample phoneme sequence corresponding to the sample text, the start and end times of each phoneme in the sample phoneme sequence can be obtained to calculate the duration of each phoneme based on the start duration of each phoneme. The sequence information of each phoneme can also be obtained based on the position of each phoneme in the sample phoneme sequence. For example, the start time of "j" is 0s and the end time is 0.02s, the start time of "in" is 0.02s and the end time is 0.06s, and the start and end times of each phoneme can be obtained by analogy in this way.

[0130] The start and end times here may be achieved through manual annotation or through recognition by other models, such as a forced alignment model, which is not specifically limited in this embodiment.

[0131] like Figure 4 As shown, after obtaining the duration and sequence number information of each phoneme, the duration of each phoneme in the sample phoneme sequence can be converted into the corresponding duration frame number according to the resolution value corresponding to the initial audio feature (such as Mel spectrum) of the sample audio.

[0132] The conversion here can be achieved by dividing the duration by the duration corresponding to the resolution value. For example, when the duration corresponding to the resolution value of the initial audio feature is 20ms, since the duration of the phoneme "j" is 0.02s, its corresponding duration frame number is 1 frame, and the duration of the phoneme "in" is 0.04s, its corresponding duration frame number is 2 frames. By analogy in this way, the duration frame number corresponding to each phoneme can be obtained.

[0133] After obtaining the duration frame number corresponding to each phoneme, the duration frame number corresponding to each phoneme and the sequence information of each phoneme can be combined and converted to generate a phoneme sequence tag value that can be used to mark the duration and sequence information of the phoneme.

[0134] The combination conversion here may be implemented through logical calculation or generated through mapping through a label template, which is not specifically limited in this embodiment.

[0135] Exemplarily, in some embodiments, labeling each phoneme in the sample phoneme sequence according to the sequence information and the number of continuous frames to obtain a phoneme sequence label value corresponding to the sample text includes:

[0136] Determining the number of label dimensions corresponding to each phoneme in the sample phoneme sequence according to the number of continuous frames;

[0137] Determining, according to the sequence number information, the values ​​of the elements of each marking dimension corresponding to each phoneme in the sample phoneme sequence;

[0138] According to the value, each phoneme in the sample phoneme sequence is labeled to obtain the phoneme sequence label value corresponding to the sample text.

[0139] Optionally, when implementing the labeling of phoneme serial number labels through logical calculations, the number of label dimensions corresponding to each phoneme in the sample phoneme sequence can be first determined based on the number of continuous frames. For example, if the number of continuous frames corresponding to any phoneme is 1, then the number of label dimensions corresponding to the phoneme is 1. If the number of continuous frames corresponding to any phoneme is 3, then the number of label dimensions corresponding to the phoneme is 3.

[0140] In addition, it is also necessary to determine the values ​​of the elements of each marking dimension corresponding to each phoneme based on the serial number information. For example, if the serial number information corresponding to any phoneme is 1, the values ​​of the elements of each marking dimension corresponding to the phoneme are all 1. If the serial number information corresponding to any phoneme is 3, the values ​​of the elements of each marking dimension corresponding to the phoneme are all 3.

[0141] Subsequently, based on the values ​​of the elements of each marking dimension corresponding to each phoneme, the elements of each marking dimension of each phoneme in the sample phoneme sequence are filled with values ​​to realize the label marking of each phoneme, and the phoneme serial number labels of each phoneme are combined to obtain the phoneme serial number label marking value corresponding to the sample text.

[0142] For example, the duration of the phoneme "j" is 1 frame and the sequence number is 0, so the corresponding marking dimension is 1, the value of each marking dimension is 0, and the corresponding phoneme sequence number label is 0; the duration of the phoneme "in" is 2 frames and the sequence number is 1, so the corresponding marking dimension is 2, the value of each marking dimension is 1, and the corresponding phoneme sequence number label is 11. By analogy in this way, the phoneme sequence number label corresponding to each phoneme can be obtained.

[0143] The method provided in this embodiment marks the duration of each phoneme and its position number in the sequence, and then generates a phoneme sequence label value containing the phoneme duration and sequence number information, so as to simplify the phoneme duration-position prediction into a more intuitive and efficient label prediction, which not only reduces the complexity and difficulty of model calculation, but also increases the robustness of model prediction, improves the accuracy and stability of the prediction, and thus improves the phoneme alignment performance.

[0144] The phoneme alignment system provided by the present invention is described below. The phoneme alignment system described below and the phoneme alignment method described above can be referenced to each other.

[0145] Figure 5 Schematic diagram of the structure of the phoneme alignment system provided by the present invention. Figure 5 As shown, the phoneme alignment system includes:

[0146] The first feature extraction unit 510 is configured to input the initial audio features of the target audio and the initial phoneme features of the target text into the feature extraction module in the phoneme alignment model to obtain the target audio features of the target audio and the target phoneme features of the target text;

[0147] The second feature extraction unit 520 is used to input the target audio feature and the target phoneme feature into the attention module in the phoneme alignment model to obtain a phoneme alignment matrix;

[0148] The phoneme alignment unit 530 is configured to obtain the phoneme sequence number label prediction value corresponding to the initial audio feature according to the phoneme alignment matrix, and perform phoneme alignment processing on the target audio and the target text according to the phoneme sequence number label prediction value;

[0149] The phoneme alignment model is trained based on sample audio, sample text, and the phoneme sequence label values ​​corresponding to the sample text.

[0150] The system provided in this embodiment obtains more accurate target audio features and target phoneme features by inputting the initial audio features of the target audio and the initial phoneme features of the target text into the feature extraction module in the phoneme alignment model, and then inputs the target audio features and target phoneme features into the attention module to generate a phoneme alignment matrix, and obtains the phoneme serial number label prediction value corresponding to the initial audio features through the phoneme alignment matrix. In this way, by converting the phoneme duration and position prediction into the phoneme label prediction, the prediction difficulty of the model is reduced and the robustness of the model is increased. At the same time, the accurate phoneme alignment processing between the target audio and the target text is performed through the feature extraction module and the attention module, which increases the generalization of the model, thereby effectively improving the robustness and generalization of the phoneme alignment.

[0151] In some embodiments, the first feature extraction unit is specifically configured to:

[0152] Inputting the initial audio features of the target audio into the audio feature extraction branch unit in the feature extraction module to obtain the target audio features;

[0153] The initial phoneme features of the target text are input into the phoneme feature extraction branch unit in the feature extraction module to obtain the target phoneme features.

[0154] In some embodiments, the first feature extraction unit is further configured to:

[0155] When the inference mode corresponding to the target audio is a streaming inference mode, dividing the initial audio features of the target audio into a plurality of initial sub-features;

[0156] Inputting each of the initial sub-features into the audio feature extraction branch unit to obtain a target sub-feature corresponding to each of the initial sub-features;

[0157] determining the target audio feature according to the plurality of target sub-features;

[0158] The obtaining, according to the phoneme alignment matrix, a phoneme sequence number label prediction value corresponding to the initial audio feature, includes:

[0159] Obtaining a label prediction subvalue corresponding to each of the initial sub-features according to a sub-matrix corresponding to each of the initial sub-features in the phoneme alignment matrix;

[0160] The label prediction sub-values ​​corresponding to all initial sub-features in the initial audio feature of the target audio are combined to obtain the phoneme sequence number label prediction value corresponding to the initial audio feature.

[0161] In some embodiments, the first feature extraction unit is further configured to:

[0162] Inputting the initial phoneme features of the target text into the encoding layer in the phoneme feature extraction branch unit, and adding context-aware information to the initial phoneme features by the encoding layer to obtain phoneme encoding features;

[0163] The phoneme coding feature is input into the linear layer in the phoneme feature extraction branch unit, and the linear layer performs scale transformation on the phoneme coding feature to obtain the target phoneme feature.

[0164] In some embodiments, the second feature extraction unit is specifically configured to:

[0165] The target audio feature is used as a query parameter, and the target phoneme feature is used as a key parameter, which are input into the attention module. The attention module performs cross-attention calculation on the target audio feature and the target phoneme feature to obtain the phoneme alignment matrix.

[0166] In some embodiments, the phoneme alignment system further includes a sample construction unit, specifically configured to:

[0167] The phoneme sequence number tag value corresponding to the sample text is obtained based on the following steps:

[0168] Obtaining the duration and sequence number information of each phoneme in the sample phoneme sequence corresponding to the sample text;

[0169] performing frame number conversion on the duration of each phoneme in the sample phoneme sequence according to the resolution value corresponding to the initial audio feature of the sample audio, to obtain the duration frame number corresponding to each phoneme in the sample phoneme sequence;

[0170] Each phoneme in the sample phoneme sequence is labeled according to the sequence number information and the number of continuous frames to obtain a phoneme sequence number label value corresponding to the sample text.

[0171] In some embodiments, the sample building block is further configured to:

[0172] Determining the number of label dimensions corresponding to each phoneme in the sample phoneme sequence according to the number of continuous frames;

[0173] Determining, according to the sequence number information, the values ​​of the elements of each marking dimension corresponding to each phoneme in the sample phoneme sequence;

[0174] According to the value, each phoneme in the sample phoneme sequence is labeled to obtain the phoneme sequence label value corresponding to the sample text.

[0175] The system provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0176] Figure 6 An example of a physical structure diagram of an electronic device is shown below. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call logic instructions in the memory 630 to execute a phoneme alignment method, which includes: inputting initial audio features of target audio and initial phoneme features of target text into a feature extraction module in a phoneme alignment model to obtain target audio features of the target audio and target phoneme features of the target text; inputting the target audio features and the target phoneme features into an attention module in the phoneme alignment model to obtain a phoneme alignment matrix; obtaining predicted phoneme sequence labels corresponding to the initial audio features based on the phoneme alignment matrix, and performing phoneme alignment on the target audio and the target text based on the predicted phoneme sequence labels; wherein the phoneme alignment model is trained based on sample audio, sample text, and the phoneme sequence label values ​​corresponding to the sample text.

[0177] Furthermore, the logic instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0178] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the phoneme alignment method provided by the above methods, which includes: inputting the initial audio features of the target audio and the initial phoneme features of the target text into the feature extraction module in the phoneme alignment model to obtain the target audio features of the target audio and the target phoneme features of the target text; inputting the target audio features and the target phoneme features into the attention module in the phoneme alignment model to obtain a phoneme alignment matrix; according to the phoneme alignment matrix, obtaining the phoneme serial number label prediction value corresponding to the initial audio feature, and performing phoneme alignment processing on the target audio and the target text according to the phoneme serial number label prediction value; wherein, the phoneme alignment model is trained based on sample audio, sample text, and the phoneme serial number label mark value corresponding to the sample text.

[0179] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the phoneme alignment method provided by the above-mentioned methods, the method comprising: inputting the initial audio features of the target audio and the initial phoneme features of the target text into a feature extraction module in a phoneme alignment model to obtain the target audio features of the target audio and the target phoneme features of the target text; inputting the target audio features and the target phoneme features into an attention module in the phoneme alignment model to obtain a phoneme alignment matrix; according to the phoneme alignment matrix, obtaining the phoneme serial number label prediction value corresponding to the initial audio feature, and performing phoneme alignment processing on the target audio and the target text according to the phoneme serial number label prediction value; wherein, the phoneme alignment model is trained based on sample audio, sample text, and the phoneme serial number label mark value corresponding to the sample text.

[0180] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0181] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A phoneme alignment method, characterized in that: include: Inputting the initial audio features of the target audio and the initial phoneme features of the target text into a feature extraction module in the phoneme alignment model to obtain the target audio features of the target audio and the target phoneme features of the target text; Inputting the target audio feature and the target phoneme feature into the attention module in the phoneme alignment model to obtain a phoneme alignment matrix; According to the phoneme alignment matrix, obtaining the phoneme sequence number label prediction value corresponding to the initial audio feature, and performing phoneme alignment processing on the target audio and the target text according to the phoneme sequence number label prediction value; Among them, the phoneme alignment model is trained based on sample audio, sample text, and the phoneme serial number label tag value corresponding to the sample text; the phoneme serial number label tag value is a label formed by encoding the serial number and duration of each phoneme in the sample text; the phoneme serial number label prediction value is obtained by converting the phoneme alignment matrix according to the conversion process corresponding to the actual reasoning mode, and the conversion includes indexing the elements in each frame dimension of the phoneme alignment matrix with the maximum value through the Argmax function, or indexing the elements in each frame dimension of the combination matrix of the phoneme alignment matrix with the maximum value through the Argmax function and then combining them.

2. The phoneme alignment method according to claim 1, wherein: The step of inputting the initial audio features of the target audio and the initial phoneme features of the target text into a feature extraction module in the phoneme alignment model to obtain the target audio features of the target audio and the target phoneme features of the target text includes: Inputting the initial audio features of the target audio into the audio feature extraction branch unit in the feature extraction module to obtain the target audio features; The initial phoneme features of the target text are input into the phoneme feature extraction branch unit in the feature extraction module to obtain the target phoneme features.

3. The phoneme alignment method according to claim 2, wherein: Inputting the initial audio features of the target audio into the audio feature extraction branch unit in the feature extraction module to obtain the target audio features includes: When the inference mode corresponding to the target audio is a streaming inference mode, dividing the initial audio features of the target audio into a plurality of initial sub-features; Inputting each of the initial sub-features into the audio feature extraction branch unit to obtain a target sub-feature corresponding to each of the initial sub-features; determining the target audio feature according to the plurality of target sub-features; The obtaining, according to the phoneme alignment matrix, a phoneme sequence number label prediction value corresponding to the initial audio feature, includes: Obtaining a label prediction subvalue corresponding to each of the initial sub-features according to a sub-matrix corresponding to each of the initial sub-features in the phoneme alignment matrix; The label prediction sub-values ​​corresponding to all initial sub-features in the initial audio feature of the target audio are combined to obtain the phoneme sequence number label prediction value corresponding to the initial audio feature.

4. The phoneme alignment method according to claim 2, wherein: The step of inputting the initial phoneme features of the target text into the phoneme feature extraction branch unit in the feature extraction module to obtain the target phoneme features includes: Inputting the initial phoneme features of the target text into the encoding layer in the phoneme feature extraction branch unit, and adding context-aware information to the initial phoneme features by the encoding layer to obtain phoneme encoding features; The phoneme coding feature is input into the linear layer in the phoneme feature extraction branch unit, and the linear layer performs scale transformation on the phoneme coding feature to obtain the target phoneme feature.

5. The phoneme alignment method according to any one of claims 1 to 4, characterized in that: Inputting the target audio feature and the target phoneme feature into the attention module in the phoneme alignment model to obtain a phoneme alignment matrix includes: The target audio feature is used as a query parameter, and the target phoneme feature is used as a key parameter, which are input into the attention module. The attention module performs cross-attention calculation on the target audio feature and the target phoneme feature to obtain the phoneme alignment matrix.

6. The phoneme alignment method according to any one of claims 1 to 4, characterized in that: The phoneme sequence number tag value corresponding to the sample text is obtained based on the following steps: Obtaining the duration and sequence number information of each phoneme in the sample phoneme sequence corresponding to the sample text; performing frame number conversion on the duration of each phoneme in the sample phoneme sequence according to the resolution value corresponding to the initial audio feature of the sample audio, to obtain the duration frame number corresponding to each phoneme in the sample phoneme sequence; Each phoneme in the sample phoneme sequence is labeled according to the sequence number information and the number of continuous frames to obtain a phoneme sequence number label value corresponding to the sample text.

7. The phoneme alignment method according to claim 6, wherein: The step of labeling each phoneme in the sample phoneme sequence according to the sequence number information and the number of continuous frames to obtain a phoneme sequence number label value corresponding to the sample text includes: Determining the number of label dimensions corresponding to each phoneme in the sample phoneme sequence according to the number of continuous frames; Determining, according to the sequence number information, the values ​​of the elements of each marking dimension corresponding to each phoneme in the sample phoneme sequence; According to the value, each phoneme in the sample phoneme sequence is labeled to obtain the phoneme sequence label value corresponding to the sample text.

8. A phoneme alignment system, characterized in that: include: A first feature extraction unit is configured to input the initial audio features of the target audio and the initial phoneme features of the target text into a feature extraction module in the phoneme alignment model to obtain target audio features of the target audio and target phoneme features of the target text; A second feature extraction unit is configured to input the target audio feature and the target phoneme feature into an attention module in the phoneme alignment model to obtain a phoneme alignment matrix; A phoneme alignment unit, configured to obtain, based on the phoneme alignment matrix, a phoneme sequence number label prediction value corresponding to the initial audio feature, and perform phoneme alignment processing on the target audio and the target text based on the phoneme sequence number label prediction value; Among them, the phoneme alignment model is trained based on sample audio, sample text, and the phoneme serial number label tag value corresponding to the sample text; the phoneme serial number label tag value is a label formed by encoding the serial number and duration of each phoneme in the sample text; the phoneme serial number label prediction value is obtained by converting the phoneme alignment matrix according to the conversion process corresponding to the actual reasoning mode, and the conversion includes indexing the elements in each frame dimension of the phoneme alignment matrix with the maximum value through the Argmax function, or indexing the elements in each frame dimension of the combination matrix of the phoneme alignment matrix with the maximum value through the Argmax function and then combining them.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the phoneme alignment method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the phoneme alignment method according to any one of claims 1 to 7 is implemented.

Images