Phoneme alignment method and system, electronic equipment and storage medium

Abstract

The invention provides a phoneme alignment method and system, electronic equipment and a storage medium, and the method comprises the steps: inputting an initial audio feature of a target audio and an initial phoneme feature of a target text into a feature extraction module in a phoneme alignment model, obtaining a target audio feature of the target audio and a target phoneme feature of the target text; inputting the target audio features and the target phoneme features into an attention module in the phoneme alignment model to obtain a phoneme alignment matrix; and according to the phoneme alignment matrix, obtaining a phoneme serial number label prediction value corresponding to the initial audio feature, and according to the phoneme serial number label prediction value, carrying out phoneme alignment processing on the target audio and the target text. According to the method, stable and accurate prediction of the phoneme sequence number labels is realized through fine feature extraction and attention calculation, so that the robustness and generalization of phoneme alignment are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a phoneme alignment method, system, electronic device and storage medium. Background Art

[0002] Phoneme alignment is also known as phoneme duration prediction, which aims to predict the duration of phonemes in speech features. It can assist the acoustic model in outputting the phoneme sequence corresponding to the speech features while outputting the speech features, so as to realize functions such as breakpoint continuation and digital human lip syncing. Therefore, how to accurately and stably perform phoneme alignment is an important topic that needs to be studied urgently in the industry.

[0003] Current phoneme alignment models usually implicitly generate the duration of phonemes through a sequence generation mechanism, or explicitly predict the number of frames corresponding to the phonemes through a simple convolutional neural network. It is difficult to accurately capture the phoneme correspondence between audio and text, resulting in poor robustness and generalization of phoneme alignment. Summary of the invention

[0004] The present invention provides a phoneme alignment method, system, electronic device and storage medium, which are used to solve the defect that it is difficult to accurately capture the phoneme correspondence between audio and text in the prior art, resulting in poor robustness and generalization of phoneme alignment, and to improve the robustness and generalization of phoneme alignment.

[0005] The present invention provides a phoneme alignment method, comprising: Inputting the initial audio features of the target audio and the initial phoneme features of the target text into a feature extraction module in the phoneme alignment model to obtain the target audio features of the target audio and the target phoneme features of the target text; Inputting the target audio feature and the target phoneme feature into the attention module in the phoneme alignment model to obtain a phoneme alignment matrix; According to the phoneme alignment matrix, obtaining the phoneme sequence number label prediction value corresponding to the initial audio feature, and performing phoneme alignment processing on the target audio and the target text according to the phoneme sequence number label prediction value; The phoneme alignment model is trained based on sample audio, sample text, and phoneme sequence label values ​​corresponding to the sample text.

[0006] According to a phoneme alignment method provided by the present invention, the initial audio features of the target audio and the initial phoneme features of the target text are input into a feature extraction module in a phoneme alignment model to obtain the target audio features of the target audio and the target phoneme features of the target text, including: Inputting the initial audio features of the target audio into the audio feature extraction branch unit in the feature extraction module to obtain the target audio features; The initial phoneme features of the target text are input into the phoneme feature extraction branch unit in the feature extraction module to obtain the target phoneme features.

[0007] According to a phoneme alignment method provided by the present invention, the initial audio features of the target audio are input into the audio feature extraction branch unit in the feature extraction module to obtain the target audio features, including: When the inference mode corresponding to the target audio is a streaming inference mode, dividing the initial audio feature of the target audio into a plurality of initial sub-features; Inputting each of the initial sub-features into the audio feature extraction branch unit to obtain a target sub-feature corresponding to each of the initial sub-features; Determining the target audio feature according to the plurality of target sub-features; The obtaining, according to the phoneme alignment matrix, a phoneme sequence number label prediction value corresponding to the initial audio feature includes: According to the submatrix corresponding to each of the initial sub-features in the phoneme alignment matrix, obtaining a label prediction sub-value corresponding to each of the initial sub-features; The label prediction sub-values ​​corresponding to all initial sub-features in the initial audio feature of the target audio are combined to obtain the phoneme sequence number label prediction value corresponding to the initial audio feature.

[0008] According to a phoneme alignment method provided by the present invention, the initial phoneme feature of the target text is input into the phoneme feature extraction branch unit in the feature extraction module to obtain the target phoneme feature, including: Inputting the initial phoneme features of the target text into the encoding layer in the phoneme feature extraction branch unit, and adding context-aware information to the initial phoneme features by the encoding layer to obtain phoneme encoding features; The phoneme coding feature is input into the linear layer in the phoneme feature extraction branch unit, and the linear layer performs a scale transformation on the phoneme coding feature to obtain the target phoneme feature.

[0009] According to a phoneme alignment method provided by the present invention, the target audio feature and the target phoneme feature are input into an attention module in the phoneme alignment model to obtain a phoneme alignment matrix, including: The target audio feature is used as a query parameter, and the target phoneme feature is used as a key parameter, which are input into the attention module. The attention module performs cross-attention calculation on the target audio feature and the target phoneme feature to obtain the phoneme alignment matrix.

[0010] According to a phoneme alignment method provided by the present invention, the phoneme sequence number label mark value corresponding to the sample text is obtained based on the following steps: Obtaining the duration and sequence number information of each phoneme in the sample phoneme sequence corresponding to the sample text; According to the resolution value corresponding to the initial audio feature of the sample audio, performing frame number conversion on the duration of each phoneme in the sample phoneme sequence to obtain the duration frame number corresponding to each phoneme in the sample phoneme sequence; Each phoneme in the sample phoneme sequence is labeled according to the sequence number information and the continuous frame number to obtain a phoneme sequence number label value corresponding to the sample text.

[0011] According to a phoneme alignment method provided by the present invention, labeling each phoneme in the sample phoneme sequence according to the sequence information and the continuous frame number to obtain the phoneme sequence label value corresponding to the sample text includes: Determining the number of label dimensions corresponding to each phoneme in the sample phoneme sequence according to the number of continuous frames; Determine, according to the sequence number information, the value of the element of each marking dimension corresponding to each phoneme in the sample phoneme sequence; According to the value, each phoneme in the sample phoneme sequence is labeled to obtain the phoneme sequence number label value corresponding to the sample text.

[0012] The present invention also provides a phoneme alignment system, comprising: A first feature extraction unit, configured to input the initial audio feature of the target audio and the initial phoneme feature of the target text into a feature extraction module in the phoneme alignment model to obtain the target audio feature of the target audio and the target phoneme feature of the target text; A second feature extraction unit, configured to input the target audio feature and the target phoneme feature into an attention module in the phoneme alignment model to obtain a phoneme alignment matrix; A phoneme alignment unit, configured to obtain a phoneme sequence number label prediction value corresponding to the initial audio feature according to the phoneme alignment matrix, and perform phoneme alignment processing on the target audio and the target text according to the phoneme sequence number label prediction value; The phoneme alignment model is trained based on sample audio, sample text, and phoneme sequence label values ​​corresponding to the sample text.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned phoneme alignment methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the phoneme alignment method described above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the phoneme alignment method described above is implemented.

[0016] The phoneme alignment method, system, electronic device and storage medium provided by the present invention obtain more accurate target audio features and target phoneme features by inputting the initial audio features of the target audio and the initial phoneme features of the target text into the feature extraction module in the phoneme alignment model, and then inputting the target audio features and the target phoneme features into the attention module to generate a phoneme alignment matrix, and obtaining the phoneme sequence label prediction values ​​corresponding to the initial audio features through the phoneme alignment matrix, thereby reducing the prediction difficulty of the model and increasing the robustness of the model by converting the phoneme duration and position predictions into phoneme label predictions, while performing accurate phoneme alignment processing between the target audio and the target text through the feature extraction module and the attention module, thereby increasing the generalization of the model, thereby effectively improving the robustness and generalization of phoneme alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the phoneme alignment method provided by the present invention.

[0019] Figure 2 It is a flowchart of the training steps of the phoneme alignment model provided by the present invention.

[0020] Figure 3 This is one of the flow charts of the phoneme sequence number label marking step provided by the present invention.

[0021] Figure 4 This is the second flow chart of the phoneme sequence number label marking step provided by the present invention.

[0022] Figure 5It is a structural schematic diagram of the phoneme alignment system provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Phoneme alignment is generally used in acoustic models or non-autoregressive speech synthesis models. The acoustic model accepts speech feature input and generates speech features such as Mel spectrum. The input and output durations of this process are the same. The non-autoregressive speech synthesis model accepts phoneme input and outputs speech features such as Mel spectrum. Therefore, a phoneme duration prediction model is required to align the length of the phoneme feature with the length of the speech feature.

[0026] Existing phoneme alignment models are divided into two paradigms: autoregressive and non-autoregressive. In autoregressive models, such as the Towards End-to-End Speech Synthesis (tacotron) model, text and phoneme features are implicitly aligned through a sequence generation mechanism. This model does not require duration labels and supports streaming reasoning, but it is difficult to integrate into other models as a functional model. It is difficult to accurately capture the phoneme correspondence between audio and text, and it is easy to miss or repeat a phoneme, which leads to poor robustness and generalization of phoneme alignment.

[0027] In non-autoregressive models, for example, the duration prediction module in the Fast, Robust and Controllable Text to Speech (Fastspeech) model is used to predict the number of frames corresponding to the audio through a convolutional network, and the Variational Inference with Adversarial Learning for End-to-End Text-to-Speech (VITS) model is used to predict the duration of each phoneme through a dynamic programming algorithm. However, such methods are difficult to support streaming reasoning and are limited by the acquisition of global information. Therefore, it is difficult to accurately capture the phoneme correspondence between audio and text, resulting in poor robustness and generalization of phoneme alignment.

[0028] In order to solve the above problems, this embodiment provides a phoneme alignment method, which obtains more accurate target audio features and target phoneme features by inputting the initial audio features of the target audio and the initial phoneme features of the target text into the feature extraction module in the phoneme alignment model. Subsequently, the target audio features and the target phoneme features are input into the attention module to generate a phoneme alignment matrix, so as to obtain the phoneme sequence label prediction value corresponding to the initial audio features through the phoneme alignment matrix, thereby reducing the prediction difficulty of the model and increasing the robustness of the model by converting the phoneme duration and position prediction into the phoneme label prediction, while performing accurate phoneme alignment processing between the target audio and the target text through the feature extraction module and the attention module, thereby increasing the generalization of the model, thereby effectively improving the robustness and generalization of the phoneme alignment. Compared with existing autoregressive models, this method is easier to integrate into other models as a functional model, and may have better accuracy and robustness, avoiding the problem of missing or repeating a phoneme. Compared with non-autoregressive models, this method can more accurately capture the phoneme correspondence between audio and text while supporting streaming reasoning, thereby improving the robustness and generalization of phoneme alignment.

[0029] It should be noted that the phoneme alignment method provided in this application can be widely applied to various audio processing scenarios, such as phoneme alignment scenarios that can be adapted to acoustic models or vocoders that output audio features, and this embodiment does not specifically limit this. The method has strong robustness and generalization. By combining it with acoustic models or vocoders in different audio processing scenarios, the acoustic model or vocoder can output the phoneme number corresponding to the feature while outputting the speech feature, so as to realize breakpoint continuation, digital human lip syncing and other functions.

[0030] Figure 1 is a flow chart of the phoneme alignment method provided by the present invention. Figure 1 As shown, the method includes step 110 , step 120 and step 130 .

[0031] Step 110, inputting the initial audio features of the target audio and the initial phoneme features of the target text into a feature extraction module in the phoneme alignment model to obtain the target audio features of the target audio and the target phoneme features of the target text; Step 120, input the target audio features and the target phoneme features into the attention module in the phoneme alignment model to obtain a phoneme alignment matrix; wherein the phoneme alignment model is trained based on sample audio, sample text, and the phoneme serial number label mark value corresponding to the sample text.

[0032] The phoneme alignment model here supports the insertion of other acoustic models or vocoders, and supports streaming reasoning and non-streaming reasoning modes. The phoneme alignment model can accurately predict the phoneme sequence number label prediction value corresponding to the initial audio features based on the initial audio features and the initial phoneme features.

[0033] Figure 2 It is a flowchart of the training steps of the phoneme alignment model provided by the present invention.

[0034] Optionally, before executing step 110, a phoneme alignment model may be obtained through training. Figure 2 As shown, specifically, an initialization prediction model can be first constructed, and the initialization prediction model here can include an initial feature extraction module and an initial attention module.

[0035] The initial feature extraction module may be a model prepared for audio feature extraction and phoneme feature extraction after parameter initialization, or it may be a pre-trained model with audio feature extraction and phoneme feature extraction functions; the initial attention module may be a model prepared for cross-attention processing of audio features and phoneme features after parameter initialization, or it may be a pre-trained language model with cross-attention processing functions of audio features and phoneme features, and this embodiment does not make any specific limitation on this.

[0036] The initial feature extraction module may be an integrated feature extraction module that can further extract audio features and phoneme features, or it may be an initial audio feature extraction branch unit that has the function of further extracting audio features and an initial phoneme feature extraction branch unit that has the function of further extracting phoneme features. This embodiment does not specifically limit this.

[0037] Each branch unit in the initial audio feature extraction branch unit and the initial phoneme feature extraction branch unit may be a branch unit with the same or different structures formed by a network layer formed based on one or more network models, such as each branch unit is constructed based on an initial coding layer and an initial linear layer. The initial coding layer here may be initialized based on a transformer model (Transformer) or a convolution enhanced transformer model (Conformer), and this embodiment does not specifically limit this.

[0038] The following describes the training steps of the phoneme alignment model provided in this embodiment by taking the initial feature extraction module as a multi-task branch model including an initial audio feature extraction branch unit and an initial phoneme feature extraction branch unit, and each branch unit being a branch network layer constructed based on an initial coding layer and an initial linear layer as an example.

[0039] In addition, sample audio and sample text may be collected, and the sample text may be labeled with a phoneme sequence number to obtain a phoneme sequence number label value corresponding to the sample text. The phoneme sequence number label value here may be a label formed by encoding the sequence number and duration of each phoneme in the sample text.

[0040] Subsequently, the initialization prediction model can be iteratively trained based on the initial audio features of the sample audio, the initial phoneme features of the sample text, and the phoneme serial number label tag value corresponding to the sample text to obtain a phoneme alignment model that can accurately predict the phoneme serial number label. The iterative training here can be to train and reason the initialization prediction model as an independent model, or to deploy it into the initial acoustic model as a module to add a phoneme alignment function to the initial acoustic model, and to implement it through joint training and reasoning with the initial acoustic model. This embodiment does not specifically limit this.

[0041] For example, when the initialization prediction model is trained and inferred as an independent model, the initial audio features of the sample audio and the initial phoneme features of the sample text can be used as inputs of the initialization prediction model, and the initial audio feature extraction branch unit of the initial feature extraction module in the initialization prediction model performs feature extraction on the initial audio features of the sample audio to obtain sample audio features, and the initial phoneme feature extraction branch unit of the initial feature extraction module in the initialization prediction model performs feature extraction on the initial phoneme features of the sample text to obtain sample phoneme features.

[0042] The initial audio features of the sample audio here can be audio features obtained by extracting features from the sample audio based on an audio feature extraction algorithm, such as Mel Frequency Cepstrum Coefficient (MFCC) features, spectrum centroid, or Perceptual Linear Predictive (PLP) features, etc., or can be intermediate features obtained by extracting features from the sample audio based on a pre-trained deep learning model with audio feature extraction, such as an acoustic model, etc. This embodiment does not specifically limit this.

[0043] The target text here is the content description text corresponding to the target audio, and the initial phoneme features of the target text can be obtained by first converting the text into a phoneme sequence using a phoneme annotation tool and then extracting the phoneme features.

[0044] For example, for sample audio feature extraction, the initial encoding layer in the initial audio feature extraction branch unit may encode the initial audio feature in combination with the context perception information to obtain the sample audio coding feature, and then the initial linear layer in the initial audio feature extraction branch unit may further transform the sample audio coding feature with information such as scale to transform the sample audio coding feature to a certain feature dimension to obtain the sample audio feature. For example, the dimension of the sample audio coding feature A is After being processed by the initial linear layer in the initial audio feature extraction branch unit, the sample audio coding feature A can be transformed into a uniform dimension , get the sample audio features . Where N is the number of frames of the initial speech feature, and H are the dimensions before and after the transformation, respectively.

[0045] Similarly, for the sample phoneme feature extraction, the initial encoding layer in the initial phoneme feature extraction branch unit can encode the initial phoneme feature in combination with the context perception information to obtain the sample phoneme coding feature, and then the initial linear layer in the initial phoneme feature extraction branch unit can further transform the sample phoneme coding feature with scale and other information to transform the sample phoneme coding feature to a certain feature dimension to obtain the sample phoneme feature. For example, the dimension of the sample phoneme coding feature P is After the initial linear layer processing in the initial phoneme feature extraction branch unit, the sample phoneme encoding feature P can be transformed into a uniform dimension , get the sample phoneme features . Where M is the length of the initial phoneme feature, and H are the dimensions before and after the transformation, respectively.

[0046] After obtaining the sample audio features and sample phoneme features, the initial attention module in the initialization prediction model performs attention calculation on the sample audio features and sample phoneme features to obtain the sample phoneme alignment matrix. Finally, the sample phoneme alignment matrix V and the phoneme serial number label value corresponding to the sample text are input into the cross entropy loss function to calculate the target loss value, so as to iteratively train the initialization prediction model according to the target loss value, thereby obtaining a phoneme alignment model that can accurately predict the phoneme serial number label.

[0047] For example, the sample phoneme alignment matrix can be obtained by the initial attention module based on the sample audio features. is the query parameter (query), with sample phoneme features Perform cross-attention calculation for the key parameter (key) to obtain the value parameter (Value) as the sample phoneme alignment matrix; the specific calculation formula of the value parameter is as follows: ; in, is the sample phoneme alignment matrix, and the dimension of V is , where N is the number of frames of the initial speech feature and M is the length of the initial phoneme feature.

[0048] After the phoneme alignment model is obtained through training, since different reasoning modes correspond to different reasoning processes in actual applications, and thus the model input is also different, the initial audio features of the target audio and the initial phoneme features of the target text can be input information determined based on the actual reasoning mode corresponding to the actual application scenario to determine the first input information corresponding to the actual reasoning mode. The determined first input information is input into the feature extraction module in the phoneme alignment model, so that the feature extraction module further extracts audio features and phoneme features based on the first input information, thereby obtaining the target audio features of the target audio and the target phoneme features of the target text.

[0049] Exemplarily, for determining the first input information, if the actual reasoning mode is a non-streaming reasoning mode, the initial audio features of the target audio are directly combined with the initial phoneme features of the target text to obtain the first input information; if the actual reasoning mode is a streaming reasoning mode, the initial audio features of the target audio are split and combined with the initial phoneme features of the target text to obtain the first input information.

[0050] The initial audio features of the target audio here can also be audio features obtained by extracting features of the target audio based on an audio feature extraction algorithm, such as Mel Frequency Cepstrum Coefficient (MFCC) features, spectrum centroid, or perceptual linear prediction (PLP) features, etc., or can be obtained by extracting features based on a pre-trained deep learning model with audio feature extraction, such as an acoustic model, etc. This embodiment does not specifically limit this.

[0051] The target text here is the content description text corresponding to the target audio. The initial phoneme features of the target text may also be obtained by first converting the text into a phoneme sequence using a phoneme annotation tool and then extracting the phoneme features.

[0052] The feature extraction module here is constructed after training the initial feature extraction module, and its structure is the same as that of the initial feature extraction module. It can also be an integrated feature extraction module that can further extract audio features and phoneme features, or it can be an audio feature extraction branch unit with the function of further extracting audio features and a phoneme feature extraction branch unit with the function of further extracting phoneme features. This embodiment does not make specific limitations on this.

[0053] After the target audio features of the target audio and the target phoneme features of the target text are obtained through the feature extraction module, the target audio features of the target audio and the target phoneme features of the target text can also be input information determined based on the actual reasoning mode to determine the second input information corresponding to the actual reasoning mode. The determined second input information is input into the attention module in the phoneme alignment model, so that the attention module performs cross-attention calculation based on the second input information to obtain a phoneme alignment matrix between the initial audio features and the initial phoneme features, that is, a phoneme alignment matrix.

[0054] Exemplarily, for the determination of the second input information, if the actual reasoning mode is a non-streaming reasoning mode, the target audio feature of the target audio is an overall feature. At this time, the initial audio feature of the target audio can be directly combined with the initial phoneme feature of the target text to obtain the second input information; if the actual reasoning mode is a streaming reasoning mode, the target audio feature of the target audio contains multiple target sub-features. At this time, each target sub-feature can be combined with the initial phoneme feature of the target text to obtain the second input information.

[0055] Step 130, obtaining the phoneme sequence number label prediction value corresponding to the initial audio feature according to the phoneme alignment matrix, and performing phoneme alignment processing on the target audio and the target text according to the phoneme sequence number label prediction value.

[0056] Optionally, after the phoneme alignment matrix is ​​predicted by the phoneme alignment model, the phoneme alignment matrix can be transformed according to the conversion process corresponding to the actual reasoning mode to obtain the phoneme serial label prediction value corresponding to the initial audio feature. By converting the predicted phoneme duration and position into the serial label of the predicted phoneme, the prediction difficulty of the model can be reduced, and consecutive identical phonemes can be distinguished, thereby effectively increasing the robustness of the phoneme alignment.

[0057] After obtaining the predicted value of the phoneme serial number label corresponding to the initial audio feature, the target audio and target text may be phoneme aligned according to the serial number information and duration of the phoneme corresponding to the predicted value of the phoneme serial number label corresponding to the initial audio feature.

[0058] Among them, different inference modes correspond to different conversion processes. If the actual inference mode is a non-streaming inference mode, the phoneme alignment matrix can be directly converted, such as using the Argmax function to take the maximum index of the elements in each frame dimension in the phoneme alignment matrix to obtain the predicted value of the phoneme sequence label corresponding to the initial audio feature. , the specific calculation formula can be expressed as: ; If the actual reasoning mode is the streaming reasoning mode, each sub-matrix in the phoneme alignment matrix can be transformed, such as by using the Argmax function to perform index processing on the elements in each frame dimension in the combination matrix and then combining them to obtain the phoneme sequence number label prediction value corresponding to the initial audio feature.

[0059] The method provided in this embodiment obtains more accurate target audio features and target phoneme features by inputting the initial audio features of the target audio and the initial phoneme features of the target text into the feature extraction module in the phoneme alignment model, and then inputs the target audio features and the target phoneme features into the attention module to generate a phoneme alignment matrix, and obtains the phoneme sequence label prediction value corresponding to the initial audio features through the phoneme alignment matrix, thereby reducing the prediction difficulty of the model and increasing the robustness of the model by converting the phoneme duration and position prediction into the phoneme label prediction, while performing accurate phoneme alignment processing between the target audio and the target text through the feature extraction module and the attention module, thereby increasing the generalization of the model, thereby effectively improving the robustness and generalization of the phoneme alignment.

[0060] In some embodiments, step 110 specifically includes: Inputting the initial audio features of the target audio into the audio feature extraction branch unit in the feature extraction module to obtain the target audio features; The initial phoneme features of the target text are input into the phoneme feature extraction branch unit in the feature extraction module to obtain the target phoneme features.

[0061] Optionally, when the feature extraction module is a model with multiple tasks including an audio feature extraction branch unit and a phoneme feature extraction branch unit, the initial audio features of the target audio can be input into the audio feature extraction branch unit, so that the audio feature extraction branch unit can perform further feature extraction on the initial audio features of the target audio to obtain the target audio features of the target audio; and the initial phoneme features of the target text can be input into the phoneme feature extraction branch unit in the feature extraction module, so that the phoneme feature extraction branch unit can perform further feature extraction on the initial phoneme features of the target phonemes to obtain the target phoneme features of the target text.

[0062] Each feature extraction branch unit in the feature extraction module here is constructed based on the training of each initial feature extraction branch unit in the initial feature extraction module, and its structure is the same as that of each initial feature extraction branch unit in the initial feature extraction module, that is, each feature extraction branch unit in the feature extraction module can be a branch unit with the same or different structure formed by a network layer constructed based on one or more network models, such as each branch unit is constructed based on a coding layer and a linear layer. The coding layer here can be constructed based on a transformer model (Transformer) or a convolution enhanced transformer model (Conformer), and this embodiment does not specifically limit this.

[0063] The method provided in this embodiment inputs the initial audio features of the target audio and the initial phoneme features of the target text into different branch units in the feature extraction module, so that the different branch units can perform refined feature extraction on the corresponding input data to obtain more accurate target audio features and target phoneme features, thereby making the audio and text more matched at the feature level, thereby improving the accuracy of phoneme alignment and further improving the performance of speech recognition and speech synthesis.

[0064] In some embodiments, the initial audio features of the target audio are input into the audio feature extraction branch unit in the feature extraction module to obtain the target audio features, including: When the inference mode corresponding to the target audio is a streaming inference mode, dividing the initial audio feature of the target audio into a plurality of initial sub-features; Inputting each of the initial sub-features into the audio feature extraction branch unit to obtain a target sub-feature corresponding to each of the initial sub-features; Determining the target audio feature according to the plurality of target sub-features; The obtaining, according to the phoneme alignment matrix, a phoneme sequence number label prediction value corresponding to the initial audio feature includes: According to the submatrix corresponding to each of the initial sub-features in the phoneme alignment matrix, obtaining a label prediction sub-value corresponding to each of the initial sub-features; The label prediction sub-values ​​corresponding to all initial sub-features in the initial audio feature of the target audio are combined to obtain the phoneme sequence number label prediction value corresponding to the initial audio feature.

[0065] Optionally, since the phoneme alignment model supports streaming reasoning mode and non-streaming reasoning mode. Therefore, in practical applications, it can be determined whether the reasoning mode corresponding to the target audio is a streaming reasoning mode or a non-streaming reasoning mode. If it is determined that the reasoning mode corresponding to the target audio is a streaming reasoning mode, the initial audio features of the target audio are split into any size to obtain multiple initial sub-features, and each initial sub-feature is input into the audio feature extraction branch unit, so that the audio feature extraction branch unit extracts each initial sub-feature separately to obtain each target sub-feature, so as to form a target audio feature containing multiple sub-features, and in the subsequent reasoning process, each target sub-feature is combined with the target phoneme feature and input into the attention module for cross-attention calculation to obtain the sub-matrix corresponding to each initial sub-feature, so as to form a phoneme alignment matrix containing multiple sub-matrices, and each sub-matrix is ​​converted separately to obtain the label prediction sub-value corresponding to each sub-matrix, and then the label prediction sub-values ​​corresponding to all initial sub-features in the initial audio feature of the target audio are combined to obtain the phoneme serial number label prediction value corresponding to the initial audio feature.

[0066] If it is determined that the reasoning mode corresponding to the target audio is a non-streaming reasoning mode, the initial audio features of the target audio are directly input as a whole into the audio feature extraction branch unit, so that the audio feature extraction branch unit extracts features of the initial audio features of the target audio as a whole to obtain the target audio features. In the subsequent reasoning process, the target audio features as a whole are combined with the target phoneme features as input to the attention module for cross-attention calculation to obtain the phoneme alignment matrix, and the phoneme alignment matrix is ​​transformed as a whole to obtain the phoneme serial number label prediction value corresponding to the initial audio features.

[0067] The so-called cross-attention calculation can be a matrix multiplication calculation.

[0068] It should be noted that the phoneme alignment matrix obtained after the initial audio features of the target audio and the initial phoneme features of the target text enter the attention module through the feature extraction module It can be expressed as Therefore, according to the matrix block rule, after the initial audio features are segmented, the phoneme alignment matrix obtained after entering the attention module through the feature extraction module is It can be expressed as follows: ; in, The encoding features of each initial sub-feature corresponding to the initial audio feature of the target audio; The encoding features corresponding to the initial phoneme features of the target text; is the sub-matrix corresponding to each initial sub-feature corresponding to the initial audio feature; and is the weight coefficient.

[0069] Therefore, in the streaming inference mode, the initial audio features are divided into arbitrary sizes, and the results are concatenated after calculation. The resulting phoneme sequence label prediction values ​​are exactly the same as those in the non-streaming inference mode, thereby ensuring the consistency and stability of phoneme alignment in different inference modes, thereby improving the robustness, generalization, and flexibility of phoneme alignment.

[0070] The method provided in this embodiment realizes flexible processing of target audio features by supporting streaming reasoning mode and non-streaming reasoning mode. Regardless of whether the audio data is input in streaming or holistic form, it can accurately perform phoneme alignment and ensure the consistency and stability of phoneme alignment in different reasoning modes, thereby significantly improving the robustness, generalization and flexibility of phoneme alignment.

[0071] In some embodiments, the step of inputting the initial phoneme feature of the target text into the phoneme feature extraction branch unit in the feature extraction module to obtain the target phoneme feature includes: Inputting the initial phoneme features of the target text into the encoding layer in the phoneme feature extraction branch unit, and adding context-aware information to the initial phoneme features by the encoding layer to obtain phoneme encoding features; The phoneme coding feature is input into the linear layer in the phoneme feature extraction branch unit, and the linear layer performs a scale transformation on the phoneme coding feature to obtain the target phoneme feature.

[0072] The phoneme feature extraction branch unit here can be constructed by a coding layer and a linear layer, wherein the coding layer is used to perform feature encoding in combination with context-aware information; and the linear layer is used to perform feature dimension transformation.

[0073] Optionally, when extracting the target phoneme feature, the initial phoneme feature of the target text may be input into the encoding layer in the phoneme feature extraction branch unit, so that the encoding layer adds context-aware information to the initial phoneme feature for feature encoding to obtain the phoneme encoding feature. After obtaining the phoneme encoding feature, the phoneme encoding feature may be input into the linear layer in the phoneme feature extraction branch unit, so that the linear layer performs a scale transformation on the phoneme encoding feature to obtain the target phoneme feature with normalized feature dimensions.

[0074] The method provided in this embodiment, by constructing a phoneme feature extraction branch unit including a coding layer and a linear layer, realizes the addition of context-aware information and feature dimension transformation of the initial phoneme features of the target text, thereby extracting more accurate target phoneme features, which not only improves the accuracy and efficiency of phoneme alignment, but also realizes the main function of phoneme alignment by combining linear transformation and matrix multiplication of cross-attention calculation, so that the phoneme alignment model has the flexible switching between streaming mode reasoning and non-streaming mode reasoning, which effectively improves the flexibility and practicality of phoneme alignment reasoning.

[0075] Similarly, when extracting target audio features, the steps of extracting target phoneme features may also be referred to for implementation, which will not be repeated here.

[0076] In some embodiments, step 120 specifically includes: The target audio feature is used as a query parameter, and the target phoneme feature is used as a key parameter, which are input into the attention module. The attention module performs cross-attention calculation on the target audio feature and the target phoneme feature to obtain the phoneme alignment matrix.

[0077] Optionally, when performing a phoneme alignment matrix, the target audio feature can be used as a query parameter, and the target phoneme feature can be input into the attention module as a key parameter, so that the attention module can perform cross-attention calculations on the target audio feature and the target phoneme feature to obtain a phoneme alignment matrix between the initial audio feature and the initial phoneme feature, that is, the phoneme alignment matrix.

[0078] The cross attention calculation here can be a matrix multiplication calculation.

[0079] The method provided in this embodiment, by inputting the target audio features as query parameters and the target phoneme features as key parameters into the attention module for cross-attention calculation, so as to fuse the information of the target audio features and the target phoneme features through attention to predict the phoneme serial number label prediction value, can effectively achieve the alignment between the audio features and the phoneme features, and improve the accuracy, efficiency and generalization of phoneme alignment.

[0080] In some embodiments, the phoneme serial number tag value corresponding to the sample text is obtained based on the following steps: Obtaining the duration and sequence number information of each phoneme in the sample phoneme sequence corresponding to the sample text; According to the resolution value corresponding to the initial audio feature of the sample audio, performing frame number conversion on the duration of each phoneme in the sample phoneme sequence to obtain the duration frame number corresponding to each phoneme in the sample phoneme sequence; Each phoneme in the sample phoneme sequence is labeled according to the sequence number information and the continuous frame number to obtain a phoneme sequence number label value corresponding to the sample text.

[0081] Figure 3 It is one of the flow charts of the phoneme sequence number labeling step provided by the present invention; Figure 4 This is the second flow chart of the phoneme sequence number label marking step provided by the present invention.

[0082] like Figure 3 As shown, when labeling the phoneme sequence corresponding to the sample text, the sample phoneme sequence corresponding to the sample text may be generated according to a pre-conversion rule, such as converting the text of the sample text into pinyin and then splitting it to form the sample phoneme sequence corresponding to the sample text. For example, if the sample text is "today's weather", the corresponding sample phoneme sequence may be expressed as "j in t ian t ian qi".

[0083] After obtaining the sample phoneme sequence corresponding to the sample text, the start and end time of each phoneme in the sample phoneme sequence can be obtained to calculate the duration of each phoneme based on the start duration of each phoneme, and the sequence information of each phoneme can be obtained according to the position of each phoneme in the sample phoneme sequence. For example, the start time of "j" is 0s and the end time is 0.02s, the start time of "in" is 0.02s and the end time is 0.06s, and the start and end time of each phoneme can be obtained by analogy in this way.

[0084] The start and end time here may be realized through manual annotation, or may be realized through other models, such as a forced alignment model for identification, which is not specifically limited in this embodiment.

[0085] like Figure 4 As shown, after the duration and sequence number information of each phoneme is obtained, the duration of each phoneme in the sample phoneme sequence can be converted into the corresponding number of duration frames according to the resolution value corresponding to the initial audio features (such as Mel spectrum) of the sample audio.

[0086] The conversion here can be realized by dividing the duration by the duration corresponding to the resolution value. For example, when the duration corresponding to the resolution value of the initial audio feature is 20ms, since the duration of the phoneme "j" is 0.02s, its duration frame number is 1 frame, and the duration of the phoneme "in" is 0.04s, its duration frame number is 2 frames, and the duration frame number corresponding to each phoneme can be obtained by analogy in this way.

[0087] After obtaining the duration frame number corresponding to each phoneme, the duration frame number corresponding to each phoneme and the sequence information of each phoneme can be combined and converted to generate a phoneme sequence tag value that can be used to mark the duration and sequence information of the phoneme.

[0088] The combined conversion here may be implemented through logical calculation, or may be generated by mapping through a label template, which is not specifically limited in this embodiment.

[0089] Exemplarily, in some embodiments, labeling each phoneme in the sample phoneme sequence according to the sequence information and the continuous frame number to obtain the phoneme sequence label value corresponding to the sample text includes: Determining the number of label dimensions corresponding to each phoneme in the sample phoneme sequence according to the number of continuous frames; Determine, according to the sequence number information, the value of the element of each marking dimension corresponding to each phoneme in the sample phoneme sequence; According to the value, each phoneme in the sample phoneme sequence is labeled to obtain the phoneme sequence number label value corresponding to the sample text.

[0090] Optionally, when implementing the labeling of phoneme serial number labels through logical calculations, the number of labeling dimensions corresponding to each phoneme in the sample phoneme sequence can be first determined based on the number of continuous frames. For example, if the number of continuous frames corresponding to any phoneme is 1, then the number of labeling dimensions corresponding to the phoneme is 1; if the number of continuous frames corresponding to any phoneme is 3, then the number of labeling dimensions corresponding to the phoneme is 3.

[0091] In addition, it is also necessary to determine the values ​​of the elements of each marking dimension corresponding to each phoneme based on the serial number information. For example, if the serial number information corresponding to any phoneme is 1, the values ​​of the elements of each marking dimension corresponding to the phoneme are all 1. If the serial number information corresponding to any phoneme is 3, the values ​​of the elements of each marking dimension corresponding to the phoneme are all 3.

[0092] Subsequently, according to the values ​​of the elements of each marking dimension corresponding to each phoneme, the elements of each marking dimension of each phoneme in the sample phoneme sequence are filled with values ​​to realize the label marking of each phoneme, and the phoneme serial number label of each phoneme is combined to obtain the phoneme serial number label marking value corresponding to the sample text.

[0093] For example, the duration of the phoneme "j" is 1 frame and the sequence number is 0, so the corresponding marking dimension is 1, the value of each marking dimension is 0, and the corresponding phoneme sequence number label is 0; the duration of the phoneme "in" is 2 frames and the sequence number is 1, so the corresponding marking dimension is 2, the value of each marking dimension is 1, and the corresponding phoneme sequence number label is 11. By analogy in this way, the phoneme sequence number label corresponding to each phoneme can be obtained.

[0094] The method provided in this embodiment marks the duration of each phoneme and its position number in the sequence, and then generates a phoneme number label tag value containing the phoneme duration and sequence number information, so as to simplify the phoneme duration to position prediction into a more intuitive and efficient label prediction, which not only reduces the complexity and difficulty of model calculation, but also increases the robustness of model prediction, improves the accuracy and stability of prediction, and thus improves the phoneme alignment performance.

[0095] The phoneme alignment system provided by the present invention is described below. The phoneme alignment system described below and the phoneme alignment method described above can be referenced to each other.

[0096] Figure 5 Schematic diagram of the structure of the phoneme alignment system provided by the present invention. Figure 5 As shown, the phoneme alignment system includes: The first feature extraction unit 510 is used to input the initial audio features of the target audio and the initial phoneme features of the target text into the feature extraction module in the phoneme alignment model to obtain the target audio features of the target audio and the target phoneme features of the target text; The second feature extraction unit 520 is used to input the target audio feature and the target phoneme feature into the attention module in the phoneme alignment model to obtain a phoneme alignment matrix; The phoneme alignment unit 530 is used to obtain the phoneme sequence number label prediction value corresponding to the initial audio feature according to the phoneme alignment matrix, and perform phoneme alignment processing on the target audio and the target text according to the phoneme sequence number label prediction value; The phoneme alignment model is trained based on sample audio, sample text, and phoneme sequence label values ​​corresponding to the sample text.

[0097] The system provided in this embodiment obtains more accurate target audio features and target phoneme features by inputting the initial audio features of the target audio and the initial phoneme features of the target text into the feature extraction module in the phoneme alignment model, and then inputs the target audio features and the target phoneme features into the attention module to generate a phoneme alignment matrix, and obtains the phoneme sequence label prediction value corresponding to the initial audio features through the phoneme alignment matrix, thereby reducing the prediction difficulty of the model and increasing the robustness of the model by converting the phoneme duration and position prediction into the phoneme label prediction, while performing accurate phoneme alignment processing between the target audio and the target text through the feature extraction module and the attention module, thereby increasing the generalization of the model, thereby effectively improving the robustness and generalization of the phoneme alignment.

[0098] In some embodiments, the first feature extraction unit is specifically configured to: Inputting the initial audio features of the target audio into the audio feature extraction branch unit in the feature extraction module to obtain the target audio features; The initial phoneme features of the target text are input into the phoneme feature extraction branch unit in the feature extraction module to obtain the target phoneme features.

[0099] In some embodiments, the first feature extraction unit is further used to: When the inference mode corresponding to the target audio is a streaming inference mode, dividing the initial audio feature of the target audio into a plurality of initial sub-features; Inputting each of the initial sub-features into the audio feature extraction branch unit to obtain a target sub-feature corresponding to each of the initial sub-features; Determining the target audio feature according to the plurality of target sub-features; The obtaining, according to the phoneme alignment matrix, a phoneme sequence number label prediction value corresponding to the initial audio feature includes: According to the submatrix corresponding to each of the initial sub-features in the phoneme alignment matrix, obtaining a label prediction sub-value corresponding to each of the initial sub-features; The label prediction sub-values ​​corresponding to all initial sub-features in the initial audio feature of the target audio are combined to obtain the phoneme sequence number label prediction value corresponding to the initial audio feature.

[0100] In some embodiments, the first feature extraction unit is further used to: Inputting the initial phoneme features of the target text into the encoding layer in the phoneme feature extraction branch unit, and adding context-aware information to the initial phoneme features by the encoding layer to obtain phoneme encoding features; The phoneme coding feature is input into the linear layer in the phoneme feature extraction branch unit, and the linear layer performs a scale transformation on the phoneme coding feature to obtain the target phoneme feature.

[0101] In some embodiments, the second feature extraction unit is specifically configured to: The target audio feature is used as a query parameter, and the target phoneme feature is used as a key parameter, which are input into the attention module. The attention module performs cross-attention calculation on the target audio feature and the target phoneme feature to obtain the phoneme alignment matrix.

[0102] In some embodiments, the phoneme alignment system further includes a sample construction unit, specifically configured to: The phoneme sequence number label value corresponding to the sample text is obtained based on the following steps: Obtaining the duration and sequence number information of each phoneme in the sample phoneme sequence corresponding to the sample text; According to the resolution value corresponding to the initial audio feature of the sample audio, performing frame number conversion on the duration of each phoneme in the sample phoneme sequence to obtain the duration frame number corresponding to each phoneme in the sample phoneme sequence; Each phoneme in the sample phoneme sequence is labeled according to the sequence number information and the continuous frame number to obtain a phoneme sequence number label value corresponding to the sample text.

[0103] In some embodiments, the sample construction unit is further used to: Determining the number of label dimensions corresponding to each phoneme in the sample phoneme sequence according to the number of continuous frames; Determine, according to the sequence number information, the value of the element of each marking dimension corresponding to each phoneme in the sample phoneme sequence; According to the value, each phoneme in the sample phoneme sequence is labeled to obtain the phoneme sequence number label value corresponding to the sample text.

[0104] The system provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0105] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the phoneme alignment method, which includes: inputting the initial audio features of the target audio and the initial phoneme features of the target text into the feature extraction module in the phoneme alignment model to obtain the target audio features of the target audio and the target phoneme features of the target text; inputting the target audio features and the target phoneme features into the attention module in the phoneme alignment model to obtain the phoneme alignment matrix; according to the phoneme alignment matrix, obtaining the phoneme sequence number label prediction value corresponding to the initial audio features, and performing phoneme alignment processing on the target audio and the target text according to the phoneme sequence number label prediction value; wherein the phoneme alignment model is trained based on the sample audio, the sample text, and the phoneme sequence number label mark value corresponding to the sample text.

[0106] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0107] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the phoneme alignment method provided by the above-mentioned methods, which includes: inputting the initial audio features of the target audio and the initial phoneme features of the target text into a feature extraction module in a phoneme alignment model to obtain the target audio features of the target audio and the target phoneme features of the target text; inputting the target audio features and the target phoneme features into an attention module in the phoneme alignment model to obtain a phoneme alignment matrix; according to the phoneme alignment matrix, obtaining the phoneme sequence number label prediction value corresponding to the initial audio feature, and performing phoneme alignment processing on the target audio and the target text according to the phoneme sequence number label prediction value; wherein the phoneme alignment model is trained based on sample audio, sample text, and the phoneme sequence number label mark value corresponding to the sample text.

[0108] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the phoneme alignment method provided by the above-mentioned methods, the method comprising: inputting the initial audio features of the target audio and the initial phoneme features of the target text into a feature extraction module in a phoneme alignment model to obtain the target audio features of the target audio and the target phoneme features of the target text; inputting the target audio features and the target phoneme features into an attention module in the phoneme alignment model to obtain a phoneme alignment matrix; according to the phoneme alignment matrix, obtaining the phoneme sequence number label prediction value corresponding to the initial audio feature, and performing phoneme alignment processing on the target audio and the target text according to the phoneme sequence number label prediction value; wherein the phoneme alignment model is trained based on sample audio, sample text, and the phoneme sequence number label mark value corresponding to the sample text.

[0109] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0110] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A phoneme alignment method, characterized in that: include: Inputting the initial audio features of the target audio and the initial phoneme features of the target text into a feature extraction module in the phoneme alignment model to obtain the target audio features of the target audio and the target phoneme features of the target text; Inputting the target audio feature and the target phoneme feature into the attention module in the phoneme alignment model to obtain a phoneme alignment matrix; According to the phoneme alignment matrix, obtaining the phoneme sequence number label prediction value corresponding to the initial audio feature, and performing phoneme alignment processing on the target audio and the target text according to the phoneme sequence number label prediction value; The phoneme alignment model is trained based on sample audio, sample text, and phoneme sequence label values ​​corresponding to the sample text.

2. The phoneme alignment method according to claim 1, characterized in that: The step of inputting the initial audio features of the target audio and the initial phoneme features of the target text into a feature extraction module in the phoneme alignment model to obtain the target audio features of the target audio and the target phoneme features of the target text includes: Inputting the initial audio features of the target audio into the audio feature extraction branch unit in the feature extraction module to obtain the target audio features; The initial phoneme features of the target text are input into the phoneme feature extraction branch unit in the feature extraction module to obtain the target phoneme features.

3. The phoneme alignment method according to claim 2, characterized in that: The step of inputting the initial audio features of the target audio into the audio feature extraction branch unit in the feature extraction module to obtain the target audio features includes: When the inference mode corresponding to the target audio is a streaming inference mode, dividing the initial audio feature of the target audio into a plurality of initial sub-features; Inputting each of the initial sub-features into the audio feature extraction branch unit to obtain a target sub-feature corresponding to each of the initial sub-features; Determining the target audio feature according to the plurality of target sub-features; The obtaining, according to the phoneme alignment matrix, a phoneme sequence number label prediction value corresponding to the initial audio feature includes: According to the submatrix corresponding to each of the initial sub-features in the phoneme alignment matrix, obtaining a label prediction sub-value corresponding to each of the initial sub-features; The label prediction sub-values ​​corresponding to all initial sub-features in the initial audio feature of the target audio are combined to obtain the phoneme sequence number label prediction value corresponding to the initial audio feature.

4. The phoneme alignment method according to claim 2, characterized in that: The step of inputting the initial phoneme feature of the target text into the phoneme feature extraction branch unit in the feature extraction module to obtain the target phoneme feature includes: Inputting the initial phoneme features of the target text into the encoding layer in the phoneme feature extraction branch unit, and adding context-aware information to the initial phoneme features by the encoding layer to obtain phoneme encoding features; The phoneme coding feature is input into the linear layer in the phoneme feature extraction branch unit, and the linear layer performs a scale transformation on the phoneme coding feature to obtain the target phoneme feature.

5. The phoneme alignment method according to any one of claims 1 to 4, characterized in that: The step of inputting the target audio feature and the target phoneme feature into the attention module in the phoneme alignment model to obtain a phoneme alignment matrix includes: The target audio feature is used as a query parameter, and the target phoneme feature is used as a key parameter, which are input into the attention module. The attention module performs cross-attention calculation on the target audio feature and the target phoneme feature to obtain the phoneme alignment matrix.

6. The phoneme alignment method according to any one of claims 1 to 4, characterized in that: The phoneme sequence number label value corresponding to the sample text is obtained based on the following steps: Obtaining the duration and sequence number information of each phoneme in the sample phoneme sequence corresponding to the sample text; According to the resolution value corresponding to the initial audio feature of the sample audio, performing frame number conversion on the duration of each phoneme in the sample phoneme sequence to obtain the duration frame number corresponding to each phoneme in the sample phoneme sequence; Each phoneme in the sample phoneme sequence is labeled according to the sequence number information and the continuous frame number to obtain a phoneme sequence number label value corresponding to the sample text.

7. The phoneme alignment method according to claim 6, characterized in that: The step of labeling each phoneme in the sample phoneme sequence according to the sequence number information and the continuous frame number to obtain a phoneme sequence number label value corresponding to the sample text includes: Determining the number of label dimensions corresponding to each phoneme in the sample phoneme sequence according to the number of continuous frames; Determine, according to the sequence number information, the value of the element of each marking dimension corresponding to each phoneme in the sample phoneme sequence; According to the value, each phoneme in the sample phoneme sequence is labeled to obtain the phoneme sequence number label value corresponding to the sample text.

8. A phoneme alignment system, characterized in that: include: A first feature extraction unit, configured to input the initial audio feature of the target audio and the initial phoneme feature of the target text into a feature extraction module in the phoneme alignment model to obtain the target audio feature of the target audio and the target phoneme feature of the target text; A second feature extraction unit, configured to input the target audio feature and the target phoneme feature into an attention module in the phoneme alignment model to obtain a phoneme alignment matrix; A phoneme alignment unit, configured to obtain a phoneme sequence number label prediction value corresponding to the initial audio feature according to the phoneme alignment matrix, and perform phoneme alignment processing on the target audio and the target text according to the phoneme sequence number label prediction value; The phoneme alignment model is trained based on sample audio, sample text, and phoneme sequence label values ​​corresponding to the sample text.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the phoneme alignment method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the phoneme alignment method according to any one of claims 1 to 7 is implemented.

Images