Speech and text alignment method and device, electronic equipment, medium and program

Abstract

The present disclosure relates to a speech and text alignment method and device, electronic equipment, medium and program, the method constructs an end-to-end speech and text alignment model based on a bidirectional attention mechanism, expands the context information used for modeling to the entire speech and text range, improves the accuracy of boundary detection results, and thus improves the effect of speech and text alignment. In addition, the end-to-end speech and text alignment model provided by the present disclosure can avoid the error accumulation problem caused by the long pipeline framework.

Description

Technical Field

[0001] This disclosure relates to the field of speech processing technology, and in particular to a method, apparatus, electronic device, medium, and program for aligning speech and text. Background Technology

[0002] Speech-to-text forced alignment refers to determining the start and end boundaries of each element in the text within the speech for mutually matching speech and text. Currently, methods for speech-to-text forced alignment, such as Prosodylab-aligner and the automatic speech-to-phoneme alignment tool (Montreal Force Aligner, MFA), are typically developed and implemented based on hidden Markov models (HMMs). However, HMMs have insufficient ability to acquire contextual information, resulting in unsatisfactory accuracy of boundary prediction results during speech-to-text alignment. Therefore, improving the effectiveness of speech-to-text forced alignment is an urgent problem to be solved. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure provides a method, apparatus, electronic device, medium, and program for aligning speech and text.

[0004] Firstly, this disclosure provides a method for aligning speech and text, including:

[0005] The speech to be processed is input into the speech encoder included in the speech-text alignment model to obtain the speech encoding sequence output by the speech encoder;

[0006] The text to be processed is input into the text encoder included in the speech-text alignment model to obtain the text encoding sequence output by the text encoder; the content of the speech to be processed matches the content of the text to be processed.

[0007] The speech encoding sequence and the text encoding sequence are input into the bidirectional attention mechanism module included in the speech-text alignment model to obtain a first attention matrix and a second attention matrix output by the bidirectional attention mechanism module; the first attention matrix is ​​the attention matrix used for speech synthesis based on the text to be processed, and the second attention matrix is ​​the attention matrix used for speech recognition based on the speech to be processed;

[0008] The first attention matrix and the second attention matrix are input into the boundary detector included in the speech-text alignment model to obtain the boundary prediction results corresponding to each text element of the text to be processed output by the boundary detector. The boundary prediction results are used to indicate the start and end positions of the speech segments in the speech to be processed corresponding to the text elements.

[0009] As one possible implementation, the bidirectional attention mechanism module employs a bidirectional multiplicative attention mechanism or a bidirectional additive attention mechanism.

[0010] As one possible implementation, the step of inputting the first attention matrix and the second attention matrix into the boundary detector included in the speech-text alignment model to obtain the boundary prediction results corresponding to each text element of the text to be processed output by the boundary detector includes:

[0011] The boundary detector performs partial summation and accumulation processing on the transpose matrices corresponding to the first attention matrix and the second attention matrix along both the positive and negative directions of the input time sequence to obtain two correlation matrices corresponding to the first attention matrix and two correlation matrices corresponding to the second attention matrix.

[0012] Boundary features are extracted based on the two correlation matrices corresponding to the first attention matrix and the two correlation matrices corresponding to the second attention matrix.

[0013] Based on the boundary features, the starting position information corresponding to each text element is obtained;

[0014] Based on the starting position information corresponding to each of the text elements, the boundary prediction results corresponding to each of the text elements are obtained.

[0015] As one possible implementation, obtaining the starting position information corresponding to the text element based on the boundary features includes:

[0016] A linear transformation is performed based on the boundary features to obtain the first linear transformation result;

[0017] The first linear transformation result is mapped according to the first preset function to obtain the probability value that each position of the speech coding sequence is the starting position corresponding to the text element;

[0018] The probability values ​​of each position in the speech encoding sequence, which are the starting positions corresponding to the text elements, are partially summed and accumulated to obtain the corresponding partial sum accumulation result, which indicates the starting position information corresponding to the text elements.

[0019] As one possible implementation, the speech-text alignment model further includes: a positional encoding module; before inputting the speech encoding sequence and the text encoding sequence into the bidirectional attention mechanism module included in the speech-text alignment model to obtain the first attention weight matrix and the second attention weight matrix output by the bidirectional attention mechanism module, the method further includes:

[0020] The location encoding module adds location encoding to the speech encoding sequence and the text encoding sequence respectively to obtain a speech encoding sequence with added location information and a text encoding sequence with added location information; the location encoding includes: original location encoding and / or estimated location encoding.

[0021] As one possible implementation, the addition of positional encoding to the speech encoding sequence and the text encoding sequence respectively includes:

[0022] Perform a linear transformation on the first encoded sequence to obtain the second linear transformation result;

[0023] The second linear transformation result is mapped according to the second preset function, and then the mapping result is partially summed and accumulated to obtain the estimated position information.

[0024] The estimated location information and the second encoding sequence are concatenated in a preset manner to obtain a second encoding sequence with the estimated location information added.

[0025] Wherein, if the first encoding sequence is a text encoding sequence, then the second encoding sequence is the speech encoding sequence; if the first encoding sequence is a speech encoding sequence, then the second encoding sequence is the text encoding sequence.

[0026] As one possible implementation, the method further includes:

[0027] Based on the estimated location information, the actual location of each element in the second encoding sequence, and the loss function corresponding to the location encoding module, the loss information corresponding to the estimated location encoding is obtained.

[0028] The speech-text alignment model is optimized based on the loss information corresponding to the predicted position encoding.

[0029] As one possible implementation, the method further includes:

[0030] The target loss information is obtained based on the target loss function corresponding to the bidirectional attention mechanism module, the first attention matrix, and the second attention matrix; wherein, the target loss function is a loss function based on diagonalization constraints;

[0031] The speech-text alignment model is optimized based on the target loss information.

[0032] As one possible implementation, the target loss function is a weighted sum of the first attention matrix, the second attention matrix, and the diagonalization constraint matrix, wherein the diagonalization constraint matrix is ​​the matrix corresponding to the diagonalization constraint.

[0033] As one possible implementation, the diagonalization constraint matrix is ​​generated based on the first attention matrix and the second attention matrix.

[0034] As one possible implementation, the speech-text alignment model further includes: a speech decoder and / or a text decoder; the method further includes: inputting the text encoding sequence into the speech decoder to obtain a speech synthesis result; and / or, inputting the speech encoding sequence into the text decoder to obtain a speech recognition result.

[0035] As one possible implementation, the method further includes: optimizing the speech-text alignment model based on the speech synthesis result and / or the speech recognition result.

[0036] As one possible implementation, optimizing the speech-text alignment model based on the speech synthesis result and / or the speech recognition result includes:

[0037] Based on the mean square error between the speech sequence included in the speech synthesis result and the speech sequence corresponding to the speech to be processed, obtain the loss information corresponding to the speech synthesis; and / or,

[0038] Based on the cross-entropy between the predicted text element sequence included in the speech recognition result and the text element sequence corresponding to the text to be processed, the loss information corresponding to speech recognition is obtained.

[0039] The speech-text alignment model is optimized based on the loss information corresponding to speech synthesis and / or the loss information corresponding to speech recognition.

[0040] Secondly, this disclosure provides a speech-text alignment device, comprising:

[0041] The first encoding module is used to encode the input speech to be processed using a speech encoder with a speech alignment model to obtain a speech encoding sequence;

[0042] The second encoding module is used to encode the input text to be processed using the text encoder of the speech alignment model to obtain a text encoding sequence; the content of the speech to be processed matches the content of the text to be processed.

[0043] A bidirectional attention mechanism processing module is used to process the speech encoding sequence and the text encoding sequence using the bidirectional attention mechanism module of the speech alignment model to obtain a first attention matrix and a second attention matrix; the first attention matrix is ​​the attention matrix used for speech synthesis based on the text to be processed, and the second attention matrix is ​​the attention matrix used for speech recognition based on the speech to be processed.

[0044] The boundary detection module is used to employ a boundary detector with a speech alignment model. Based on a first attention matrix and a second attention matrix, it obtains the boundary prediction results corresponding to each text element in the text to be processed. The boundary prediction results are used to indicate the start and end positions of the speech segments in the speech to be processed corresponding to the text elements.

[0045] Thirdly, this disclosure provides an electronic device, including: a memory and a processor;

[0046] The memory is configured to store computer program instructions;

[0047] The processor is configured to execute the computer program instructions, causing the electronic device to implement the speech-text alignment method as described in any of the first aspects.

[0048] Fourthly, this disclosure provides a readable storage medium comprising: computer program instructions; at least one processor of an electronic device executes the computer program instructions, causing the electronic device to implement the speech-text alignment method as described in any of the first aspects.

[0049] Fifthly, this disclosure provides a computer program product that, when executed by a computer, implements the speech-text alignment method as described in any of the first aspects.

[0050] This disclosure provides a method, apparatus, electronic device, medium, and program for aligning speech and text. The method constructs an end-to-end speech-text alignment model based on a bidirectional attention mechanism, extending the contextual information used for modeling to the entire speech and text range, thereby improving the accuracy of boundary detection results and thus enhancing the speech-text alignment effect. Furthermore, the end-to-end speech-text alignment model provided in this disclosure avoids the error accumulation problem caused by long pipeline training frameworks. Attached Figure Description

[0051] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0052] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A schematic flowchart illustrating a method for aligning speech and text according to an embodiment of this disclosure;

[0054] Figure 2 This is a schematic diagram of the structure of a speech-text alignment model provided in an embodiment of the present disclosure;

[0055] Figure 3 A schematic diagram of the framework of a bidirectional attention mechanism provided in an embodiment of this disclosure;

[0056] Figure 4 A flowchart illustrating a method for aligning speech and text according to another embodiment of this disclosure;

[0057] Figure 5 This is a schematic diagram of the structure of a speech-text alignment model provided in another embodiment of the present disclosure;

[0058] Figure 6 A schematic diagram of the framework for additional positional encoding provided in another embodiment of this disclosure;

[0059] Figure 7 This is a schematic diagram of the structure of a speech-text alignment model provided in another embodiment of the present disclosure;

[0060] Figure 8 A schematic diagram of a diagonalized constraint matrix provided in an embodiment of this disclosure;

[0061] Figure 9 A schematic diagram of the structure of a speech and text alignment device provided in an embodiment of this disclosure;

[0062] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0063] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0064] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0065] Currently, when using Hidden Markov Models (HMMs) to achieve forced alignment of speech and text, the limitations of HMMs in capturing contextual information during the modeling process can lead to errors in the results. Furthermore, HMMs rely on multiple modules being trained sequentially, forming a long pipeline structure during training. This pipeline structure causes the prediction errors of previous modules to accumulate in subsequent modules, resulting in error accumulation and suboptimal predictions for the nearest boundary.

[0066] Based on this, this disclosure proposes a speech-text alignment method based on a bidirectional attention mechanism. This method incorporates a bidirectional attention mechanism and, based on this mechanism, constructs an end-to-end model framework. The bidirectional attention mechanism mentioned herein can be extended to the entire range of speech and text for modeling, and applying this end-to-end model framework can simultaneously address the problem of error accumulation. This significantly improves the forced speech-text alignment performance.

[0067] For example, the voice and text alignment method provided in this disclosure can be performed by an electronic device, which may be a tablet computer, a mobile phone (such as a foldable screen phone, a large screen phone, etc.), a wearable device, an in-vehicle device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a smart TV, a smart screen, a high-definition TV, a 4K TV, a smart speaker, a smart projector, and other Internet of Things (IoT) devices, a server, a server cluster, a cloud server, etc. This disclosure does not impose any restrictions on the specific type of electronic device.

[0068] This disclosure will use an electronic device as an example, along with accompanying drawings and application scenarios, to provide a detailed explanation of the voice and text alignment method provided in this disclosure.

[0069] Figure 1 This is a flowchart illustrating a method for aligning speech and text according to an embodiment of the present disclosure, with reference to... Figure 1 As shown, the method in this embodiment includes:

[0070] S101. Input the speech to be processed into the speech encoder included in the speech-text alignment model to obtain the speech encoding sequence output by the speech encoder.

[0071] The purpose of encoding the speech to be processed by the speech encoder is to convert the pulse code modulation (PCM) samples of the speech to be processed into a bit frame sequence. The speech encoding sequence obtained here is the converted bit frame sequence.

[0072] Electronic devices can employ a speech encoder to encode the speech to be processed. This disclosure does not limit the type of speech encoder, encoding method, etc. For example, the speech encoder can be a model encoder, which can be based on a deep neural network (DNN), or it can be based on a convolutional neural network (CNN), or it can be based on an attention mechanism.

[0073] S102. Input the text to be processed into the text encoder included in the speech-text alignment model to obtain the text encoding sequence output by the text encoder.

[0074] The text to be processed is matched with the content of the speech to be processed. The speech and text to be processed are usually used as paired input data for forced alignment processing. The text to be processed may include a sequence of symbolic representation elements (hereinafter referred to as the text element sequence) corresponding to the speech to be processed, such as a character sequence or a phoneme sequence.

[0075] The purpose of encoding the text to be processed is to use a markup language to mark the structure and features of the text so that electronic devices can process it.

[0076] Electronic devices can encode the text to be processed using a text encoder to obtain a text encoding sequence. This disclosure does not limit the type or encoding method of the text encoder. For example, the text encoder can be a model encoder, which can be based on a deep neural network (DNN), a convolutional neural network (CNN), or an attention mechanism.

[0077] S103. Input the speech encoding sequence and the text encoding sequence into the bidirectional attention mechanism module included in the speech-text alignment model to obtain the first attention matrix and the second attention matrix output by the bidirectional attention mechanism module.

[0078] The bidirectional attention mechanism can include an attention mechanism along the speech recognition direction and an attention mechanism along the speech synthesis direction. The electronic device can analyze contextual information based on the speech encoding sequence and the text encoding sequence to obtain attention matrices along the speech recognition direction and the speech synthesis direction, respectively. These two attention matrices can then be used to characterize the regions where attention needs to be focused during speech-text alignment.

[0079] The attention matrix along the speech recognition direction can be understood as the attention matrix used for speech recognition based on the speech to be processed; the attention matrix along the speech synthesis direction can be understood as the attention matrix used for speech synthesis based on the text to be processed.

[0080] In one possible implementation, the electronic device can use the speech coding sequence and the text coding sequence as inputs to a bidirectional attention mechanism, and then model the relationship between the speech coding sequence and the text coding sequence through a compatibility function to obtain a shared attention matrix for the speech coding sequence and the text coding sequence. The first attention matrix and the second attention matrix mentioned above can be obtained by processing the shared attention matrix.

[0081] The first attention matrix is ​​the attention matrix along the speech synthesis direction, and the second attention matrix is ​​the attention matrix along the speech recognition direction.

[0082] Depending on the implementation of the compatibility function, bidirectional attention mechanisms can be further divided into bidirectional multiplicative attention mechanisms or bidirectional additive attention mechanisms. These mechanisms will be discussed in detail later in the text.

[0083] In other possible implementations, the bidirectional attention mechanism can also be implemented through two independent unidirectional attention mechanisms. That is, the bidirectional attention mechanism module of the speech-text alignment model may include two independent unidirectional attention mechanism sub-modules. Each unidirectional attention mechanism sub-module is used to output an attention matrix in a specified direction (speech recognition direction or speech synthesis direction). The input of the unidirectional attention mechanism sub-module may be related to the associated speech recognition direction or speech synthesis direction, and this disclosure does not limit the attention processing method of the two unidirectional attention mechanism sub-modules.

[0084] S104. Input the first attention matrix and the second attention matrix into the boundary detector included in the speech-text alignment model to obtain the boundary prediction results corresponding to each text element of the text to be processed in the output of the boundary detector.

[0085] Electronic devices, through the boundary detector included in the speech-text alignment model, can extract the boundary features of the currently located text element in the text encoding sequence from the first attention matrix and the second attention matrix. Then, by analyzing the boundary features, they can obtain the probability value of whether the current bit frame position in the speech encoding sequence is the corresponding starting position. Based on the probability values ​​of whether each bit frame position in the speech encoding sequence is the starting position of the current text element, the starting position of the current text element in the speech to be processed is determined by accumulating the probability values.

[0086] The method in this embodiment constructs an end-to-end speech-text alignment model based on a bidirectional attention mechanism, extending the contextual information used for modeling to the entire range of the speech and text to be processed, thereby improving the accuracy of boundary detection results and thus enhancing the speech-text alignment effect. Furthermore, the end-to-end speech-text alignment model provided in this disclosure can avoid the error accumulation problem caused by long pipeline training frameworks.

[0087] Figure 2 This is a schematic diagram of the structure of a speech-text alignment model provided in an embodiment of this disclosure. (Refer to...) Figure 2 As shown, the speech-text alignment model 200 includes: a speech encoder 201, a text encoder 202, a bidirectional attention mechanism module 203, and a boundary detector 204. The bidirectional attention mechanism module 203 is positioned between the speech encoder 201 and the boundary detector 204, and also between the text encoder 202 and the boundary detector 204.

[0088] As one possible implementation, the speech-text alignment model provided in this embodiment uses the one-hot encoded sequence of the phonemes corresponding to the text to be processed (hereinafter referred to as the phoneme sequence) as the text input and the normalized Mel-Frequency Cepstral Coefficients (MFCC) sequence as the speech input.

[0089] The speech encoder 201 is mainly used to receive the normalized MFCC sequence corresponding to the speech to be processed as the speech input, and to obtain the speech encoded sequence after processing the speech input by convolution and other methods, and output the speech encoded sequence to the bidirectional attention mechanism module 203. This disclosure does not limit the structure of the speech encoder 201. For example, the speech encoder 201 may include a 3-layer convolutional neural network and a 2-layer bidirectional gated recurrent unit (GRU) network. The convolutional kernel of the convolutional neural network can be 1*17. It should be understood that the size of the convolutional kernel can be set according to requirements; this is only an example.

[0090] The text encoder 202 is mainly used to receive the phoneme sequence corresponding to the text to be processed as text input, and to perform convolution and other processing on the phoneme sequence to obtain the text encoding sequence, and output the obtained text encoding sequence to the bidirectional attention mechanism module 203. This disclosure does not limit the structure of the text encoder 202. For example, the text encoder in the Tacotron2 model can be used as the text encoder 202 included in the speech-text alignment model mentioned in this disclosure. Its structure is one embedding layer, two convolutional neural network layers and one bidirectional GRU network layer. Of course, other text encoder structures can also be used.

[0091] The bidirectional attention mechanism module 203 is mainly used to receive speech encoded sequences and text encoded sequences as input from the speech encoder 201 and the text encoder 202 respectively, and to process the speech encoded sequences and text encoded sequences through the attention mechanism to obtain an attention matrix along the speech synthesis direction. Figure 2 China and Israel W TTS (for illustration) and attention matrix along the speech recognition direction ( Figure 2 China and Israel W ASR (Illustration)

[0092] Where W TTS When W is the first attention matrix, then ASR Let W be the second attention matrix; if W ASR When W is the first attention matrix, then TTS This is the second attention matrix. Subsequent embodiments will use the former as an example for illustration.

[0093] in, Figure 3 This is a schematic diagram illustrating the framework of a bidirectional attention mechanism provided in one embodiment of this disclosure. (Refer to...) Figure 3 As shown, the bidirectional attention mechanism takes two sets of key-value pairs as input. Figure 3 The diagram uses K1, K2, V1, and V2 as illustrations, where K1 corresponds to V1, and K2 corresponds to V2. The relationship between K1 and K2 is modeled using a compatibility function, resulting in a shared attention matrix for K2 and K2. Figure 3(Illustrated by the letter A). Matrix A is processed along its original direction and the direction after transpose, for example using the softmax function, to obtain matrix W. 12 Sum matrix W 21 .

[0094] Combination Figure 2 as well as Figure 3 As shown, Figure 3 The matrix W shown 12 Equivalent to the aforementioned Figure 2 The W shown TTS , Figure 3 The matrix W shown 21 Equivalent to the aforementioned Figure 2 The W shown ASR .

[0095] The bidirectional attention mechanism can be represented by formulas (1) to (3) as follows:

[0096] A = f(K1, K2) Formula (1)

[0097] W 12 W 21 =soft max(A, A T ) Formula (2)

[0098] O1, O 21 =W 12 T V1, W 21 T V2 formula (3)

[0099] In formula (1), f represents the compatibility function, and A represents the shared attention matrix. (This is in conjunction with the preceding text.) Figure 2 As shown, V1 and V2 represent the outputs of the text encoder and speech encoder, respectively, and O1 and O2 represent the inputs of the decoders used for speech synthesis and speech recognition, respectively.

[0100] Please refer to formula (1). Depending on the implementation of the compatibility function f, the bidirectional attention mechanism can be further divided into bidirectional multiplicative attention mechanism and bidirectional additive attention mechanism.

[0101] In the bidirectional multiplicative attention mechanism, the compatibility function is formalized as follows:

[0102] A = f1(K1) × f2(K2) T Formula (4)

[0103] Here, f1 and f2 are two linear transformations.

[0104] In the bidirectional additive attention mechanism, the compatibility function is formalized as follows:

[0105] A = f a (dup1(f1(K1))+dup1(f2(K2))) Formula (5)

[0106] Here, f1 and f2 are two linear transformations. dup1 and dup2 adjust the outputs of f1 and f2 to matrices of the same dimension by adding a dimension and copying them. a This is a linear transformation to reduce the dimension of A so that its dimension meets the requirements of the bidirectional attention mechanism.

[0107] It should be noted that the linear transformations f1 and f2 in formula (4) and f1 and f2 in formula (5) can be different linear transformations.

[0108] Please continue reading. Figure 2 As shown, the boundary detector 204 is mainly used to obtain the attention matrices corresponding to the two directions of speech recognition and speech synthesis from the bidirectional attention mechanism module 203 to predict the start and end boundaries of each text element in the text to be processed, that is, to obtain the boundary prediction results corresponding to each text element. The boundary prediction results corresponding to each text element can be obtained through the boundary signal. The boundary signal is a time-series transition signal in which the signal before the boundary (i.e., the starting position) is 0 and the signal after the boundary is 1.

[0109] The boundary detector 204 can first perform partial summation and accumulation calculations on the first attention matrix and the second attention matrix respectively, and then concatenate them with the original first attention matrix and the original second attention matrix to obtain a 6-dimensional feature matrix. For example, this can be expressed by the following formula:

[0110]

[0111] In formula (6), F represents the 6-dimensional feature matrix obtained by splicing; r represents the reverse of its input in the temporal direction.

[0112] It should be noted that the "partial sum accumulation" calculation method mentioned here means that the accumulation calculation is performed according to the order of the elements in the matrix. For each element's position, the element at that position is updated based on the corresponding accumulation calculation result. For example, given the original vector S = [s1, s2, s3, ..., sN], performing a partial sum accumulation calculation on vector S, the first element s1' = s1 remains unchanged, the second element s2' = s1 + s2, the third element s3' = s1 + s2 + s3, and so on, until sN' = s1 + s2 + ... + sN. Therefore, the vector obtained after performing the partial sum accumulation calculation is S' = [s1', s2', s3', ..., sN'].

[0113] After obtaining the feature matrix F, boundary features can be extracted from it. For example, a 3-layer convolutional neural network (CNN) can be used to extract boundary features from the feature matrix F, with a kernel size of 17*17. The extracted boundary features are then mapped to the range of 0 to 1 using the sigmoid function. Next, partial summation is performed, and the region is further restricted to the range of 0 to 1 using the tanh function to obtain the predicted boundary signal. This can be expressed by the following formula:

[0114] B′=tanh(cumsum(sigmoid(f(convs(F))))) Formula (7)

[0115] In formula (7), convs represents a convolutional neural network; f represents a linear transformation; B′ represents the predicted boundary signal; and cumsum represents partial sum accumulation.

[0116] The boundary detector 204 can output the predicted boundary signals corresponding to each text element in the text to be processed. By combining the predicted boundary signals corresponding to the adjacent text elements, the start and end positions of the speech segments corresponding to each text element in the speech to be processed can be determined.

[0117] Figure 4 This is a schematic flowchart illustrating a method for aligning speech and text according to another embodiment of this disclosure. (Refer to...) Figure 4 As shown, the method provided in this embodiment includes:

[0118] S401. Input the speech to be processed into the speech encoder included in the speech-text alignment model to obtain the speech encoding sequence output by the speech encoder.

[0119] S402. Input the text to be processed into the text encoder included in the speech-text alignment model to obtain the text encoding sequence output by the text encoder.

[0120] In this embodiment, steps S401 and S402 are respectively related to Figure 1 Steps S101 and S102 in the illustrated embodiment are similar and can be referred to Figure 1 as well as Figure 2 The specific implementation methods of the speech encoding sequence and text encoding sequence in the illustrated embodiments are not repeated here for the sake of brevity.

[0121] S403. Append positional codes to the speech coding sequence and the text coding sequence respectively to obtain the speech coding sequence with added positional information and the text coding sequence with added positional information.

[0122] Among them, positional encoding refers to the processing method of encoding positional information into the sequence. Positional encoding can add corresponding positional information to each bit frame in the speech encoding sequence and to each element in the text encoding sequence. After positional information is added to both the speech and text encoding sequences, boundary detection can be performed. This can effectively distinguish the same elements at different positions in the text to be processed and the same audio frames at different positions in the speech to be processed, thereby avoiding boundary detection confusion.

[0123] As one possible implementation, additional location coding may include original location coding and / or estimated location coding.

[0124] Among them, original position encoding refers to the processing method of appending the position information of each text element in the text to be processed to the text encoding sequence; and the processing method of appending the position of each audio frame in the corresponding audio frame sequence to the speech encoding sequence.

[0125] Estimated position encoding refers to a processing method that estimates the position of each text element in a text element sequence by processing the speech to be processed, and then appends the estimated text element position information (i.e., estimated position information) to the text encoding sequence; and a processing method that estimates the position of audio frames in an audio frame sequence by processing the text to be processed, and then appends the estimated audio frame position information to the speech encoding sequence.

[0126] As one possible implementation, the electronic device can perform a linear transformation on the first encoded sequence to obtain the corresponding linear transformation result; then map the linear transformation result obtained in the previous step according to the second preset function, and obtain the estimated position information by performing partial sum accumulation processing on the mapping result; and then concatenate the estimated position information with the second encoded sequence in a preset manner to obtain a second encoded sequence with the estimated position information added.

[0127] In accordance with the preceding text, if the first encoding sequence is a text encoding sequence, the second encoding sequence is a speech encoding sequence; if the first encoding sequence is a speech encoding sequence, the second encoding sequence is a text encoding sequence.

[0128] The specific implementation methods of the original position coding and the estimated position coding can be referred to below. Figure 5 Detailed description of the embodiments.

[0129] S404. Input the speech encoding sequence with additional location information and the text encoding sequence with additional location information into the bidirectional attention mechanism module included in the speech-text alignment model to obtain the first attention matrix and the second attention matrix output by the bidirectional attention mechanism module.

[0130] Electronic devices can adopt the methods described above. Figure 2 The bidirectional attention mechanism module 203 shown takes the speech encoding sequence and the text encoding sequence with additional location information as inputs for bidirectional attention mechanism processing. The specific processing procedure can be found in [reference needed]. Figure 2 As shown, for the sake of simplicity, further details will not be elaborated here.

[0131] S405. Input the first attention matrix and the second attention matrix into the boundary detector included in the speech-text alignment model to obtain the boundary prediction results corresponding to each text element of the text to be processed in the output of the boundary detector.

[0132] In this embodiment, step S405 and Figure 1 Step S104 in the illustrated embodiment is similar and can be referred to. Figure 1 as well as Figure 2 The specific implementation details of boundary prediction in the illustrated embodiment are omitted here for the sake of brevity.

[0133] The method provided in this embodiment adds positional information to the speech encoding sequence and the text encoding sequence to distinguish the same text elements at different positions in the text to be processed and the same bit frames in the speech to be processed, thereby effectively avoiding the problem of boundary prediction confusion for the same text elements at different positions in the text to be processed.

[0134] Figure 5 This is a schematic diagram of the structure of a speech-text alignment model provided in an embodiment of this disclosure. First, please refer to... Figure 5 The speech-text alignment model shown is in Figure 3 Based on the embodiment shown, a position encoding module 205 is set between the speech encoder 201 and the bidirectional attention mechanism module 203, and between the text encoder 202 and the bidirectional attention mechanism module 203.

[0135] The location encoding module 205 is mainly used to receive a speech encoding sequence from the speech encoder 201 as input and a text encoding sequence from the text encoder 202 as another input. It adds location encoding to the two inputs respectively and outputs a speech encoding sequence with added location information and a text encoding sequence with added location information.

[0136] As mentioned earlier, positional information can be added to speech and text encoded sequences using two methods: appending original positional codes and appending estimated positional codes. Below, we will combine... Figure 6 The schematic diagram of the additional positional encoding shown below illustrates the specific implementation methods of the two encoding methods:

[0137] The original position encoding can be achieved using trigonometric functions, and can be expressed by the following formula:

[0138] PE (pos，2i) =sin(pos / 10000) 2i / d ) Formula (8)

[0139] PE (pos，2i+1) =cos(pos / 10000) 2i / d ) Formula (9)

[0140] In formulas (8) and (9), for text encoding sequences, pos represents the position number of an element in the element sequence; for speech encoding sequences, pos represents the position number of an audio frame in the audio frame sequence; d represents the dimension of the input text feature or speech feature; and i represents the circular index on the feature dimension.

[0141] By calculating each dimension of the input text features or speech features, the original text location coding features or original speech location coding features can be obtained.

[0142] The predicted location encoding involves predicting possible speech locations from the original text encoding sequence and predicting possible text locations from the original speech encoding sequence. First, a linear transformation is performed on the output of the speech encoder or text encoder to obtain the corresponding linear transformation result. Then, an activation function (such as the ReLU function) is used to map the linear transformation result, and finally, the predicted text location or predicted speech location is obtained through partial sum accumulation. This can be expressed by the following formula:

[0143] pos s ′=cumsum(ReLU(f t (E t ))) Formula (10)

[0144] pos t ′=cumsum(ReLU(f s (E s ))) Formula (11)

[0145] Among them, E t Represents the text encoding sequence (i.e., the output of the text encoder); E s This represents the speech coding sequence (i.e., the output of the speech encoder); ft and f s Represents a linear transformation; cumsum represents partial sum accumulation; pos represents... s ′ represents the estimated speech location, pos t ′ indicates the estimated text position.

[0146] Next, the estimated text positions and estimated speech positions can be converted into estimated text position coding features and estimated speech position coding features in a similar manner to the original position coding. The original text position coding features and the original speech position coding features are then appended to two copies of the output of the text encoder or speech encoder, respectively. Finally, the two copies are concatenated to obtain the output that has both the original position coding and the estimated position coding appended to it.

[0147] This can be expressed by the following formula:

[0148] E t ′=[E t +PE t E t +PE s ′] Formula (12)

[0149] E s ′=[E s +PE t ′;E s +PE s ] Formula (13)

[0150] Among them, PE t PE s These represent the original text location coding features and the original speech location coding features, respectively; PE t '、PE s ′ represent the estimated text location coding features and the estimated speech location coding features, respectively; E t ′ represents a text encoding sequence with the original positional code and the estimated positional code appended; E s ′ represents a speech coding sequence with the original position code and the estimated position code appended.

[0151] It should be noted that other position coding calculation methods can also be used, or the original position coding or the estimated position coding can be added separately, as long as position information can be added to the output of the text encoder and the speech encoder.

[0152] It should be noted that in some cases, monotonicity-related constraints can be added to the bidirectional attention mechanism to resolve the boundary prediction confusion of the same elements at different positions. By adopting this approach, the position encoding module can be eliminated in the speech-text alignment model, which can further simplify the model structure and ensure the alignment effect between speech and text.

[0153] Figure 7 This is a schematic diagram of the structure of a speech-text alignment model provided in another embodiment of this disclosure. (Refer to...) Figure 7 The model structure shown is in Figure 5 Based on the illustrated embodiment, it also includes: a text decoder 206 and / or a speech decoder 207.

[0154] The speech encoder 201, the bidirectional attention mechanism module 203, and the text decoder 206 constitute a speech recognition model for performing speech recognition tasks. Specifically, the attention matrix for the speech recognition direction output by the bidirectional attention mechanism module 203 and the speech encoding sequence output by the speech encoder 201 can be weighted to obtain a weighted calculation result. This weighted calculation result can be used as input to the text decoder 206 to output the speech recognition result.

[0155] The text decoder 206 can be implemented using a structure of a two-layer bidirectional long short-term memory (LSTM) network and a one-layer linear transform layer. For example, the LSTM network can be a 128-dimensional network structure.

[0156] It should be noted that the text decoder 206 can also be implemented using other structures, and is not limited to the structure shown in this example. For example, the number of LSTM networks, their dimensions, and the number of linear transformation layers can be different.

[0157] The text encoder 202, the bidirectional attention mechanism module 203, and the speech decoder 207 constitute a speech synthesis model for performing speech synthesis tasks. Specifically, the attention matrix for the speech synthesis direction output by the bidirectional attention mechanism module 203 and the text encoding sequence output by the text encoder 202 can be weighted to obtain a weighted calculation result. This weighted calculation result can be used as input to the speech decoder 207 to output the speech synthesis result.

[0158] The speech decoder 207 can be implemented using a structure of a two-layer bidirectional LSTM network and a one-layer linear transform layer. For example, the LSTM network can be a 256-dimensional network structure.

[0159] It should be noted that the speech decoder 207 can also be implemented using other structures, and is not limited to the structure shown in this example. For example, the number of LSTM networks, their dimensions, and the number of linear transformation layers can be different.

[0160] pass Figure 7 In the embodiment shown, this disclosure constructs a multi-task joint end-to-end model for forced semantic text alignment, speech recognition, and speech synthesis based on a bidirectional attention mechanism.

[0161] It should be noted that in some possible implementations, the position encoding module 205 may not be included between the speech encoder 201, the text encoder 202, and the bidirectional attention mechanism module 203. The implementation of this can be found in [reference needed]. Figure 2 Detailed description of the illustrated embodiment.

[0162] Please continue reading. Figure 7 ,exist Figure 7 Based on the illustrated embodiment, the loss function of this model can be composed of a weighted sum of multiple loss functions, including: a speech recognition loss function, a speech synthesis loss function, a forced alignment loss function, a predicted position loss function, and a loss function corresponding to the attention mechanism.

[0163] In the weighted summation, the aforementioned loss functions can correspond to different weights. For example, the weight of the speech recognition loss function is 0.1, the weight of the speech synthesis loss function is 1, the weight of the forced alignment loss function is 100, the weight of the position estimation loss function is 10, and the weight of the loss function corresponding to the attention mechanism is 0. In the unsupervised training process, the loss function corresponding to the attention mechanism can replace the forced alignment loss function, with a weight of 100.

[0164] The following section provides a detailed introduction to each of the aforementioned loss functions:

[0165] The speech recognition loss function can be composed of the cross entropy between the predicted speech recognition result and the labeled speech recognition result. Taking the speech input to be processed as an example in the previous text, the predicted speech recognition result can be the predicted phoneme sequence, and the labeled speech recognition result can be the phoneme sequence included in the text to be processed.

[0166] The speech synthesis loss function can be composed of the mean squared error (MSE) between the predicted speech synthesis result and the labeled speech synthesis result. Taking the text to be processed as the text input for speech synthesis in the previous text as an example, the predicted speech synthesis result can be the predicted MFCC sequence, and the labeled speech synthesis result can be the MFCC sequence corresponding to the speech to be processed.

[0167] The forced alignment loss function is the loss function corresponding to the boundary detector, which can be composed of the average absolute error (MSE) between the predicted boundary information and the labeled boundary information corresponding to each text element in the text to be processed. The labeled boundary information corresponding to each text element in the text to be processed can be obtained through multiple corrections.

[0168] The prediction location loss function can include a text location loss function and a speech location loss function. The last element of the predicted text location should be the same as the actual text length; therefore, the MSE (Mean Estimate) normalized to the actual text length from the last element of the predicted text location constitutes the text location loss function. Similarly, the MSE normalized to the actual speech length from the last element of the predicted speech location constitutes the speech location loss function. The text location loss function and the speech location loss function can be expressed by the following formulas:

[0169]

[0170]

[0171] in, This indicates the estimated text position loss; Indicates the estimated speech location loss; L t L s These represent the actual text length (i.e., the actual text length of the text to be processed) and the actual speech length (i.e., the actual speech length of the speech to be processed), respectively; last indicates taking the last element of the input.

[0172] The attention mechanism loss function can be a diagonalized attention weight loss function, which is a weighted sum of the attention matrix and the matrix corresponding to the diagonalized constraints (hereinafter referred to as the diagonalized constraint matrix). This can be expressed by the following formula:

[0173]

[0174] Where, loss a represents the attention mechanism loss; D represents the diagonalization constraint matrix; ⊙ represents positional multiplication.

[0175] It should be noted that the diagonalization constraint matrix D can be dynamically generated based on the input first attention matrix and second attention matrix. The generation method can satisfy the following formula:

[0176]

[0177] Where i and j are dimension-related subscripts, and p and q are the relative positions of i and j in their corresponding dimensions. For example, a generated diagonalized constraint matrix can be referenced... Figure 8 As shown.

[0178] Of course, other calculation methods can also be used to generate the diagonalized constraint matrix D, and this disclosure does not limit this.

[0179] In addition, it should be noted that during model training, the calculation method or weights of the loss function can be changed, or one or more of the aforementioned loss functions can be omitted, and can be flexibly set according to needs.

[0180] Figure 9 This is a schematic diagram of the structure of a speech-text alignment device provided according to an embodiment of this disclosure. (Refer to...) Figure 9 As shown, the speech-text alignment device 900 provided in this embodiment may include:

[0181] The first encoding module 901 is used to encode the input speech to be processed using a speech encoder with a speech alignment model, so as to obtain the speech encoding sequence output by the speech encoder.

[0182] The second encoding module 902 is used to encode the input text to be processed using the text encoder of the speech alignment model to obtain the text encoding sequence output by the text encoder; the content of the speech to be processed matches the content of the text to be processed.

[0183] The bidirectional attention mechanism processing module 903 is used to process the speech encoding sequence and the text encoding sequence using the bidirectional attention mechanism module of the speech alignment model to obtain a first attention matrix and a second attention matrix; the first attention matrix is ​​the attention matrix used for speech synthesis based on the text to be processed, and the second attention matrix is ​​the attention matrix used for speech recognition based on the speech to be processed.

[0184] The boundary detection module 904 is used to employ the boundary detector of the speech alignment model to obtain the boundary prediction results corresponding to each text element of the text to be processed according to the first attention matrix and the second attention matrix. The boundary prediction results are used to indicate the start and end positions of the speech segments in the speech to be processed corresponding to the text elements.

[0185] As one possible implementation, the bidirectional attention mechanism module employs a bidirectional multiplicative attention mechanism or a bidirectional additive attention mechanism.

[0186] As one possible implementation, the boundary detection module 904 is specifically used to perform partial summation and accumulation processing on the transpose matrices corresponding to the first attention matrix and the second attention matrix along both the positive and negative directions of the input time sequence to obtain two correlation matrices corresponding to the first attention matrix and two correlation matrices corresponding to the second attention matrix; to extract boundary features based on the two correlation matrices corresponding to the first attention matrix and the two correlation matrices corresponding to the second attention matrix to obtain boundary features; to obtain the starting position information corresponding to each text element based on the boundary features; and to obtain the boundary prediction results corresponding to each text element included in the text to be processed based on the starting position information corresponding to each text element.

[0187] As one possible implementation, the boundary detection module 904 is specifically used to perform a linear transformation based on the boundary features to obtain a first linear transformation result; map the first linear transformation result according to a first preset function to obtain the probability value that each position of the speech coding sequence is the starting position corresponding to the current element; perform partial sum accumulation processing on the probability values ​​that each position of the speech coding sequence is the starting position corresponding to the current element to obtain the corresponding partial sum accumulation result, which indicates the starting position information corresponding to the element.

[0188] As one possible implementation, the speech-text alignment device 900 further includes: a position information encoding module 905; correspondingly, the speech-text alignment model also includes a position encoding module; the position information encoding module 905 is used to add position encoding to the speech encoding sequence and the text encoding sequence respectively using the position encoding module to obtain a speech encoding sequence with added position information and a text encoding sequence with added position information; and inputs the speech encoding sequence with added position information and the text encoding sequence with added position information to the bidirectional attention mechanism processing module 903; wherein, the position encoding includes: original position encoding and / or estimated position encoding.

[0189] As one possible implementation, the location information encoding module 905 is specifically used to perform a linear transformation on the first encoding sequence to obtain a second linear transformation result; map the second linear transformation result according to a second preset function, and then perform partial summation processing on the mapping result to obtain estimated location information; superimpose the estimated location information with the second encoding sequence to obtain a second encoding sequence with additional location information; wherein, if the first encoding sequence is a text encoding sequence, then the second encoding sequence is the speech encoding sequence; if the first encoding sequence is a speech encoding sequence, then the second encoding sequence is the text encoding sequence.

[0190] As one possible implementation, the speech-text alignment device 900 further includes: a model optimization module ( Figure 9 (Not shown in the image), used to obtain target loss information based on the target loss function corresponding to the bidirectional attention mechanism, the first attention matrix, and the second attention matrix; wherein, the target loss function is a loss function based on diagonalization constraints; and to optimize the speech-text alignment model based on the target loss information.

[0191] As one possible implementation, the target loss function is a weighted sum of the first attention matrix, the second attention matrix, and the diagonalization constraint matrix, wherein the diagonalization constraint matrix is ​​the matrix corresponding to the diagonalization constraint.

[0192] As one possible implementation, the diagonalization constraint matrix is ​​generated based on the first attention matrix and the second attention matrix.

[0193] As one possible implementation, the speech-to-text alignment device 900 further includes: a decoding module ( Figure 9 (not shown in the image); correspondingly, the speech-text alignment module further includes a speech decoder and / or a text decoder; a decoding module is used to process the text-encoded sequence using the speech decoder to obtain the speech synthesis result; and / or, to process the speech-encoded sequence using the text decoder to obtain the speech recognition result.

[0194] Specifically, the text encoding sequence and the first attention matrix can be input into the speech decoder to obtain the speech synthesis result output by the speech decoder; the speech encoding sequence and the second attention matrix can be input into the text decoder to obtain the speech recognition result output by the text decoder.

[0195] As one possible implementation, the model optimization module is further configured to optimize the speech-text alignment model based on the speech synthesis result and / or the speech recognition result.

[0196] The model optimization module can obtain the corresponding loss information based on the speech recognition loss function and speech synthesis loss function introduced above, and then optimize the speech-text alignment model based on the loss information.

[0197] The speech and text alignment device provided in this embodiment can be used to execute the technical solutions of any of the foregoing method embodiments. Its implementation principle and technical effect are similar, and can be referred to the detailed description of the foregoing method embodiments. For the sake of brevity, it will not be repeated here.

[0198] Figure 10 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present disclosure. (Refer to...) Figure 9As shown, the electronic device 1000 provided in this embodiment may include a memory 1001 and a processor 1002.

[0199] The memory 1001 can be a separate physical unit, connected to the processor 1002 via a bus 1003. Alternatively, the memory 1001 and processor 1002 can be integrated and implemented in hardware. The memory 1001 stores program instructions, which the processor 1002 calls to execute the operations of any of the above method embodiments.

[0200] Optionally, when some or all of the methods in the above embodiments are implemented by software, the electronic device 1000 may also include only the processor 1002. The memory 1001 for storing programs is located outside the electronic device 1000, and the processor 1002 is connected to the memory via circuits / wires for reading and executing the programs stored in the memory.

[0201] The processor 1002 can be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP.

[0202] The processor 1002 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0203] The memory 1001 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory may also include a combination of the above types of memory.

[0204] This disclosure also provides a readable storage medium, including: computer program instructions; when executed by at least one processor of an electronic device, the computer program instructions implement the speech and text alignment method shown in any of the above method embodiments.

[0205] This disclosure also provides a computer program product, the program product including a computer program stored in a readable storage medium, at least one processor of the electronic device can read the computer program from the readable storage medium, the at least one processor executes the computer program to enable the electronic device to implement the speech and text alignment method as shown in any of the above method embodiments.

[0206] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0207] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of aligning speech and text, characterized by, The method comprises: inputting the to-be-processed speech into a speech encoder included in a speech-text alignment model to obtain a speech encoding sequence output by the speech encoder; inputting the to-be-processed text into a text encoder included in the speech-text alignment model to obtain a text encoding sequence output by the text encoder; the content of the to-be-processed speech and the content of the to-be-processed text match each other; inputting the speech encoding sequence and the text encoding sequence into a bidirectional attention mechanism module included in the speech-text alignment model to obtain a first attention matrix and a second attention matrix output by the bidirectional attention mechanism module; the first attention matrix is an attention matrix used for speech synthesis according to the to-be-processed text, and the second attention matrix is an attention matrix used for speech recognition according to the to-be-processed speech; inputting the first attention matrix and the second attention matrix into a boundary detector included in the speech-text alignment model to obtain boundary prediction results corresponding to each text element included in the to-be-processed text, which are output by the boundary detector and are used to indicate the start position and the end position of a speech segment corresponding to the text element in the to-be-processed speech.

2. The method of claim 1, wherein, The bidirectional attention mechanism module adopts a bidirectional multiplicative attention mechanism or a bidirectional additive attention mechanism.

3. The method of claim 1, wherein, The inputting the first attention matrix and the second attention matrix into the boundary detector included in the speech-text alignment model to obtain the boundary prediction results corresponding to each text element included in the to-be-processed text, which are output by the boundary detector, comprises: The boundary detector respectively performs partial sum accumulation processing on the first attention matrix and a transposed matrix corresponding to the second attention matrix along the forward and reverse directions of the input time sequence to obtain two associated matrices corresponding to the first attention matrix and two associated matrices corresponding to the second attention matrix; boundary feature extraction is performed on the two associated matrices corresponding to the first attention matrix and the two associated matrices corresponding to the second attention matrix to obtain boundary features; start position information corresponding to each text element is obtained according to the boundary features; boundary prediction results corresponding to each text element are obtained according to the start position information corresponding to each text element.

4. The method of claim 3, wherein, The obtaining the start position information corresponding to each text element according to the boundary features comprises: linear transformation is performed on the boundary features to obtain a first linear transformation result; a first preset function is used to map the first linear transformation result to obtain a probability value of each position of the speech encoding sequence being the start position of the text element; partial sum accumulation processing is performed on the probability value of each position of the speech encoding sequence being the start position of the text element to obtain a corresponding partial sum accumulation result, which indicates the start position information of the text element.

5. The method according to claim 1 or 2, characterized in that, The speech-text alignment model further comprises a position encoding module; before the speech encoding sequence and the text encoding sequence are input into the bidirectional attention mechanism module comprised in the speech-text alignment model, the method further comprises: The position encoding module is used to respectively attach position encoding to the speech encoding sequence and the text encoding sequence, so as to obtain a speech encoding sequence with additional position information and a text encoding sequence with additional position information; the position encoding comprises original position encoding and / or estimated position encoding.

6. The method of claim 5, wherein, The position encoding module is used to respectively attach position encoding to the speech encoding sequence and the text encoding sequence, so as to obtain a speech encoding sequence with additional position information and a text encoding sequence with additional position information; the position encoding comprises original position encoding and / or estimated position encoding. The method further comprises: linearly transforming the first encoding sequence to obtain a second linear transformation result; mapping the second linear transformation result according to a second preset function, and then performing partial sum accumulation processing on the mapping result to obtain estimated position information; splicing the estimated position information and a second encoding sequence according to a preset mode to obtain a second encoding sequence with the estimated position information attached thereto; 7. The method of claim 1, wherein, wherein, if the first encoding sequence is a text encoding sequence, the second encoding sequence is the speech encoding sequence; if the first encoding sequence is a speech encoding sequence, the second encoding sequence is the text encoding sequence. The method further comprises: obtaining target loss information according to a target loss function corresponding to the bidirectional attention mechanism module, the first attention matrix and the second attention matrix; wherein, the target loss function is a loss function based on diagonalization constraint; 8. The method of claim 7, wherein, optimizing the speech-text alignment model according to the target loss information.

9. The method of claim 8, wherein, The target loss function is a weighted sum of the first attention matrix, the second attention matrix and a diagonalization constraint matrix, wherein the diagonalization constraint matrix is a matrix corresponding to the diagonalization constraint.

10. The method of claim 1, wherein, The diagonalization constraint matrix is generated according to the first attention matrix and the second attention matrix. The speech-text alignment model further comprises a speech decoder and / or a text decoder; the method further comprises:

11. The method of claim 10, wherein, inputting the text encoding sequence into the speech decoder to obtain a speech synthesis result; and / or, inputting the speech encoding sequence into the text decoder to obtain a speech recognition result. The method further comprises:

12. An apparatus for aligning speech and text, characterized by optimizing the speech-text alignment model according to the speech synthesis result and / or the speech recognition result. comprises: a first encoding module configured to encode inputted to-be-processed speech by using a speech encoder of a speech alignment model to obtain a speech encoding sequence; a second encoding module configured to encode inputted to-be-processed text by using a text encoder of the speech alignment model to obtain a text encoding sequence; the content of the to-be-processed speech and the content of the to-be-processed text match each other; The bidirectional attention mechanism processing module is configured to process the speech coding sequence and the text coding sequence by using a bidirectional attention mechanism module of the speech-text alignment model to obtain a first attention matrix and a second attention matrix; the first attention matrix is an attention matrix used for speech synthesis according to the text to be processed, and the second attention matrix is an attention matrix used for speech recognition according to the speech to be processed; The boundary detection module is configured to obtain boundary prediction results corresponding to respective text elements included in the text to be processed by using a boundary detector of the speech-text alignment model according to the first attention matrix and the second attention matrix, the boundary prediction results being used to indicate starting positions and ending positions of speech segments corresponding to the text elements in the speech to be processed.

13. An electronic device, comprising: The method comprises: a memory and a processor; the memory is configured to store computer program instructions; the processor is configured to execute the computer program instructions, so that the electronic device implements the speech-text alignment method according to any one of claims 1 to 11.

14. A readable storage medium, characterized by, The method comprises: computer program instructions; the computer program instructions are executed by at least one processor of an electronic device, so that the electronic device implements the speech-text alignment method according to any one of claims 1 to 11.

15. A computer program product, characterised in that, When a computer executes the computer program product, so that the computer implements the speech-text alignment method according to any one of claims 1 to 11.

Images