Training Method of Speech Alignment Network, Speech Alignment Method and Electronic Device

Abstract

The present application discloses a method for training a speech alignment network, comprising: obtaining a labeled text sequence and an audio feature sequence; inputting the audio feature sequence into an encoding network to obtain an audio feature encoding sequence; obtaining the value of an alignment loss function; inputting the labeled text sequence and the audio feature encoding sequence into a decoder network based on an attention mechanism to obtain an audio feature decoding sequence; obtaining the value of an attention loss function based on the audio feature decoding sequence and the labeled text sequence; and cyclically training the encoding network and the decoding network based on the value of the alignment loss function and the value of the attention loss function until a condition is met, and outputting the encoding network. The trained network model can be deployed on a mobile phone to achieve speech alignment with good timeliness. A method for speech alignment, a corresponding electronic device, and a computer-readable storage medium are also provided.

Description

Technical Field

[0001] The present application relates to the technical field of speech processing, and more particularly to a training method for a speech alignment network and a method for speech alignment. In addition, the present application also relates to related electronic devices and computer-readable storage media. Background Art

[0002] In the digital music field, lyrics timestamp technology can be used to mark the start and end times of each word in the lyrics, as well as the start and end times of each line, ultimately producing a lyrics text file with a QRC (lyrics file type) timestamp. Existing techniques, based on the differences between singing and speaking pronunciation (primarily reflected in speech rate, which determines the length of each phoneme), modify pronunciation dictionaries and, based on information about different song genres, propose a genre-aware acoustic alignment model. This model directly aligns the lyrics to the accompanying music audio, obtaining the start and end times of each word. Existing acoustic alignment models are difficult to deploy on mobile clients due to their numerous modules, complex training and inference, and high computational resource requirements. Most are deployed on the server backend. This deployment achieves high alignment accuracy but results in extended latency. Users upload their audio and lyrics files to a backend server, which then generates the alignment file and transmits it back to the user asynchronously. This processing method offers high accuracy but suffers from poor timeliness, limiting its applicability to limited scenarios.

[0003] The content of this background technology description is only for facilitating understanding of the relevant technology in this field and is not regarded as an admission of the prior art. Summary of the Invention

[0004] Therefore, the embodiment of the present invention intends to provide a solution that can quickly achieve voice alignment on the mobile phone side. It uses a lightweight end-to-end neural network and can be directly deployed on the mobile phone side. The user only needs to select the audio and its corresponding lyrics to obtain the corresponding aligned lyrics timestamp text in real time. This processing method saves traffic costs and time costs.

[0005] An embodiment of the present invention provides a method for training a speech alignment network, comprising the following steps:

[0006] Obtain annotated text sequence and audio feature sequence;

[0007] Inputting the audio feature sequence into an encoder network to obtain an audio feature encoding sequence;

[0008] Obtaining a value of an alignment loss function based on the audio feature encoding sequence and the annotated text sequence;

[0009] Inputting the annotated text sequence and the audio feature encoding sequence into a decoder network based on an attention mechanism to obtain an audio feature decoding sequence;

[0010] Obtaining a value of an attention loss function based on the audio feature decoding sequence and the annotated text sequence;

[0011] If the value of the alignment loss function is greater than the first loss threshold or the value of the attention loss function is greater than the second loss threshold, iteratively updating the encoder network and the decoder network based on the value of the alignment loss function and the value of the attention loss function, and returning to execute the step of obtaining the labeled text sequence and the audio feature sequence until the value of the alignment loss function is less than or equal to the first loss threshold and the value of the attention loss function is less than or equal to the second loss threshold;

[0012] The encoder network after the last update is used as the speech alignment network.

[0013] In some embodiments of the present invention, obtaining the value of the alignment loss function includes:

[0014] Summing the probability of each alignment distribution of the audio feature coding sequence and the annotated text sequence to obtain a probability sum value;

[0015] The negative logarithm of the probability sum is taken, and the result is used as the value of the alignment loss function.

[0016] In some embodiments of the present invention, the encoder network is a lightweight end-to-end network.

[0017] In some embodiments of the present invention, the annotated text sequence includes an annotated text sequence in units of characters or an annotated text sequence in units of phonemes.

[0018] An embodiment of the present invention further provides a method for speech alignment, comprising the following steps:

[0019] Obtain a target text sequence and a target audio, wherein the target text sequence is a lyrics text sequence of the target audio;

[0020] Inputting the audio feature sequence of the target audio into the speech alignment network generated by any of the above training methods to obtain the target audio feature encoding sequence;

[0021] Based on the target audio feature encoding sequence, the target text sequence is aligned with the target audio.

[0022] In some embodiments of the present invention, the target audio feature encoding sequence is the posterior probability distribution of each word or each phoneme in the target text sequence at each moment.

[0023] In some embodiments of the present invention, aligning the target text sequence with the target audio based on the target audio feature encoding sequence includes:

[0024] Based on the posterior probability distribution, an optimal time distribution of each word or each phoneme in the target text sequence to the time frame sequence of the target audio is generated, so that the target text sequence is aligned with the target audio.

[0025] In some embodiments of the present invention, generating, based on the posterior probability distribution, an optimal time distribution of each word or each phoneme in the target text sequence to the time frame sequence of the target audio includes:

[0026] Arbitrarily distributing each word or each phoneme in the target text sequence to the time frame sequence of the target audio to generate a full distribution sequence;

[0027] Calculate the sum of the posterior probabilities of each word or each phoneme in each distribution sequence in the full distribution sequence at the corresponding time;

[0028] The aligned distribution sequence with the largest sum of posterior probabilities is taken as the optimal time distribution.

[0029] In some embodiments of the present invention, generating, based on the posterior probability distribution, an optimal time distribution of each word or each phoneme in the target text sequence to the time frame sequence of the target audio includes:

[0030] Initialize the time frame sequence of the aligned distribution according to the time frame sequence of the target audio;

[0031] Inserting a placeholder before the first character or the first phoneme in the target text sequence, inserting a placeholder after the last character or the last phoneme in the target text sequence, and inserting a placeholder between adjacent characters or phonemes to form a state sequence;

[0032] Setting the state of the first time frame in the time frame sequence of the aligned distribution to a placeholder or the first character or phoneme in the state sequence;

[0033] Set the state of the last time frame in the time frame sequence of the aligned distribution to be a placeholder or the last character or phoneme in the state sequence

[0034] Obtaining an aligned distribution sequence based on the state sequence, the state of the first time frame in the aligned distribution time frame sequence, and the state of the last time frame;

[0035] Based on the posterior probabilities of each state in the aligned distribution sequence, calculate the sum of the posterior probabilities of each aligned distribution;

[0036] The alignment distribution with the largest sum of posterior probabilities is taken as the optimal time distribution.

[0037] In some embodiments of the present invention, obtaining an aligned distribution sequence based on the state sequence, the state of the first time frame, and the state of the last time frame in the aligned distribution time frame sequence includes:

[0038] Based on the state sequence, the state of the previous time frame in the aligned distribution is recursively set according to the current time frame state of its distribution, until the first time frame state of its distribution is set to a placeholder or the first character or first phoneme in the state sequence, thereby obtaining an aligned distribution sequence.

[0039] In some embodiments of the present invention, the step of recursively setting the state of the previous time frame in the aligned distribution according to the current time frame state of the aligned distribution based on the state sequence until the first time frame state of the aligned distribution is set to a placeholder or the first character or first phoneme in the state sequence, and obtaining the aligned distribution sequence includes:

[0040] If the state of the current time frame of the aligned distribution is a placeholder, then based on the state sequence, the state corresponding to the previous time frame of the current time frame is set to be the same as the state of the current time frame, or the state corresponding to the previous time frame is set to be the state before the state of the current time frame;

[0041] If the state of the current time frame is not a placeholder, and the state of the current time frame is the same as the two previous states of the state of the current time frame, then based on the state sequence, the state corresponding to the previous time frame is set to be the same as the state of the current time frame, or the state corresponding to the previous time frame is set to be the previous state of the current time frame;

[0042] If the current time frame state is not a placeholder, and the state of the current time frame is different from the two previous states of the current time frame state, set the state corresponding to the previous time frame of the current time frame to be the same as the state of the current time frame, or set the state corresponding to the previous time frame of the current time frame to be the previous state of the current time frame, or set the state corresponding to the previous time frame of the current time frame to be the two previous states of the current time frame state;

[0043] The above steps are recursively executed until the first time frame state is set to a placeholder or the first character or the first phoneme in the state sequence, and an aligned distribution sequence is obtained.

[0044] In some embodiments of the present invention, generating, based on the target audio feature code sequence, an optimal time distribution of each word or each phoneme in the target text sequence to the time frame sequence of the target audio, includes:

[0045] Initialize the time frame sequence of the aligned distribution according to the time frame sequence of the target audio;

[0046] Inserting a placeholder before the first character or the first phoneme in the target text sequence, inserting a placeholder after the last character or the last phoneme in the target text sequence, and inserting a placeholder between adjacent characters or phonemes to form a state sequence;

[0047] Setting the state of the first time frame in the time frame sequence of the aligned distribution to a placeholder or the first character or phoneme in the state sequence;

[0048] Set the state of the last time frame in the time frame sequence of the aligned distribution to be a placeholder or the last character or phoneme in the state sequence

[0049] Based on the state sequence and the current time frame of the aligned distribution, recursively setting the state of the current time frame in the aligned distribution according to the state of the previous time frame of the aligned distribution, until the state of the last time frame is set to a placeholder or the last character or the first phoneme in the state sequence, thereby obtaining an aligned distribution sequence;

[0050] Calculating a sum of the posterior probabilities of the aligned distributions based on the posterior probabilities of the states in the aligned distributions;

[0051] The alignment distribution with the largest sum of posterior probabilities is taken as the optimal time distribution.

[0052] In some embodiments of the present invention, recursively setting the state of the current time frame in the alignment distribution according to the state of the previous time frame based on the state sequence until the state of the last time frame is set to a placeholder or the last character or the first phoneme in the state sequence, to obtain the alignment distribution sequence, includes:

[0053] Based on the state sequence, arbitrarily set the state of the current time frame;

[0054] When determining that the current time frame state is a placeholder, verifying that the state corresponding to the previous time frame is the same as the state of the current time frame, or that the state corresponding to the previous time frame is a state previous to the state of the current time frame, determining that the state of the current time frame is within the target range;

[0055] If the status of the current time frame is not a placeholder, and the status of the current time frame is the same as the two previous statuses of the current time frame, and the status corresponding to the previous time frame is verified to be the same as the status of the current time frame, or the status corresponding to the previous time frame is the previous status of the current time frame, then the status of the current time frame is determined to be within the target range;

[0056] If the current time frame status is not a placeholder, and the current time frame status is different from the two previous statuses of the current time frame status, and the status corresponding to the previous time frame is verified to be the same as the current time frame status, or the status corresponding to the previous time frame is the previous status of the current time frame status, or the status corresponding to the previous time frame is the two previous statuses of the current time frame status, then the current time frame status is determined to be within the target range;

[0057] Based on the state of the previous time frame, the state with the highest posterior probability among the states within the target range is obtained as the target state, and the state of the current time frame is set according to the target state;

[0058] The above steps are recursively executed until the state of the last time frame is set to a placeholder or the last character or phoneme in the state sequence, and an aligned distribution sequence is obtained.

[0059] In an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the program is executed by a processor, the training method of the speech alignment network or the speech alignment method of any embodiment of the present invention is implemented.

[0060] In an embodiment of the present invention, an electronic device is provided, comprising: a processor and a memory storing a computer program, wherein the processor is configured to execute the training method of the speech alignment network or the speech alignment method of any embodiment of the present invention when running the computer program.

[0061] The trained end-to-end speech alignment network used in the embodiment of the present invention is particularly suitable for deployment on mobile phones. In addition, in the speech alignment method in the embodiment of the present invention, the audio feature sequence and the corresponding text lyrics sequence are input, and the target audio feature coding sequence can be obtained by using the speech alignment network, and the CTC (Connectionist temporal classification) Alignment alignment algorithm can be used to solve the optimal path, and then obtain which corresponding modeling unit (word or phoneme) each frame belongs to. In this way, the starting frame of each specific word can be obtained, and finally the starting time of the word can be obtained, and the timestamp of the word can be generated. Save traffic cost and time cost.

[0062] Other optional features and technical effects of the embodiments of the present invention are partially described below, and partially can be understood by reading this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The elements shown are not limited to the scale shown in the drawings. The same or similar reference numerals in the drawings represent the same or similar elements, wherein:

[0064] Figure 1 A schematic diagram showing an example of alignment;

[0065] Figure 2a An example flow chart of a method for training a speech alignment network according to an embodiment of the present invention is shown;

[0066] Figure 2b An example diagram of data flow of a method for training a speech alignment network according to an embodiment of the present invention is shown;

[0067] Figure 3a An example flow chart of a speech alignment method according to an embodiment of the present invention is shown;

[0068] Figure 3b An example diagram of data flow of a speech alignment method according to an embodiment of the present invention is shown;

[0069] Figure 4a An exemplary flow chart of an alignment step in a speech alignment method according to an embodiment of the present invention is shown;

[0070] Figure 4b Another exemplary flow chart of the alignment step in the speech alignment method according to an embodiment of the present invention is shown;

[0071] Figure 4c Another exemplary flow chart of the alignment step in the speech alignment method according to an embodiment of the present invention is shown;

[0072] Figure 5a An exemplary flow chart of a recursive method for alignment steps in a speech alignment method according to an embodiment of the present invention is shown;

[0073] Figure 5b An example diagram of a recursive process of an alignment step in a speech alignment method according to an embodiment of the present invention is shown;

[0074] Figure 5c An example diagram of a recursive process of an alignment step in a speech alignment method according to an embodiment of the present invention is shown;

[0075] Figure 6a Another exemplary flow chart of a recursive method for the alignment step in the speech alignment method according to an embodiment of the present invention is shown;

[0076] Figure 6b Another example diagram of a recursive process of an alignment step in a speech alignment method according to an embodiment of the present invention is shown;

[0077] Figure 7a shows example graphs of outputs based on word and phoneme modeling;

[0078] Figure 7bshows example graphs of outputs based on word and phoneme modeling;

[0079] Figure 7c shows example graphs of outputs based on word and phoneme modeling;

[0080] Figure 8 An exemplary structural diagram of a training device for a speech alignment network according to an embodiment of the present invention is shown;

[0081] Figure 9 An exemplary structural diagram of a speech alignment device according to an embodiment of the present invention is shown;

[0082] Figure 10 A schematic diagram of an exemplary structure of an electronic device capable of implementing the method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with specific embodiments and accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0084] In the embodiment of the present invention, a speech recognition model can be defined, a speech sequence can be input, and the corresponding recognition text can be output. In the embodiment of the present invention, "alignment" means that a speech recognition model is defined, a speech sequence and a correct annotated text (which can be a text sequence or a phoneme sequence) are input, and the corresponding relationship between the speech frame and the text is given, such as the 0th frame to the 10th frame corresponds to the phoneme 'm', such as Figure 1 shown.

[0085] An acoustic alignment model includes an acoustic model, a pronunciation dictionary, and an appropriate alignment algorithm for obtaining aligned sequences. Some known solutions employ an HMM-based hybrid acoustic alignment model. This model has numerous modules, complex training and inference processes, and requires significant computing resources, making it difficult to deploy on mobile clients. Most deployments are done on server backends, achieving high alignment accuracy but with significant latency.

[0086] In an embodiment of the present invention, an end-to-end lyrics to audio alignment method is proposed, which is mainly used in clients such as mobile phones. It uses a pure neural network training method and adopts the CTC Viterbi algorithm to solve the optimal path to obtain the alignment of the phoneme level (training a phone-level acoustic model) or the word level (training a word-level acoustic model), thereby training a neural network model for audio alignment, and conveniently and quickly deploying it to the mobile phone end for lyrics to audio alignment. The neural network model adopts a Transformer model based on the Attention-Encoder-Decoder (AED) mechanism, uses Joint CTC / AED joint training and learning, and finally uses the CTC Decoder part to complete the alignment function: for a trained model, given a piece of audio and the corresponding annotated lyrics text, the alignment result of the speech and text is obtained by the score information of the CTCDecoder part. Specifically, a neural network is used to train an end-to-end speech recognition network, and then only the Encoder module is used to output the probability score of the modeling unit for each frame. The CTC Viterbi Alignment algorithm is used to obtain the start and end time of the modeling unit, and then the timestamp of the modeling unit is obtained to achieve the alignment function, where the modeling unit is a word or a phoneme.

[0087] In this embodiment of the present invention, the speech alignment network can be applied to a Transformer network. The network is trained using a hybrid CTC / attention joint training mechanism. The modeling units used in the network output can be multiple. The model uses a word-level end-to-end approach, resulting in word-level alignment. If phone-level alignment is required, a phone-level model must be trained.

[0088] Specifically, if Figure 2a 、 2b As shown, an embodiment of the present invention provides a method for training a speech alignment network, comprising the following steps:

[0089] S110: Obtain annotated text sequence and audio feature sequence.

[0090] The annotated text is converted into an annotated text sequence through dictionary conversion. After the audio data is collected and calculated in the time domain or frequency domain, an audio feature is generated for each time frame. The audio features are arranged in the order of the time frames to generate an audio feature sequence.

[0091] S120: Input the audio feature sequence into the encoder network to obtain an audio feature coding sequence.

[0092] In some embodiments of the present invention, an encoder-decoder network architecture is used, and the encoder network that can be used includes a Tansformer network. In some embodiments of the present invention, the encoder network is a lightweight end-to-end network, which facilitates subsequent network deployment on the mobile phone, achieves localized recognition alignment, and reduces network transmission time.

[0093] In an embodiment of the present invention, the audio feature coding sequence may include a coding sequence representing audio features such as fbank (FilterBank) or mfcc (Mel-frequency cepstral coefficients). In a more specific embodiment, the audio feature coding sequence may be the posterior probability (distribution) of each modeling unit, such as a word or phoneme (as described above, depending on the modeling unit level to be modeled), corresponding to audio features (frames) such as fbank (FilterBank) or mfcc (Mel-frequency cepstral coefficients) obtained after audio processing.

[0094] S130. Obtain a value of an alignment loss function based on the audio feature coding sequence and the annotated text sequence.

[0095] The value of the alignment loss function is generated according to the audio feature coding sequence and the annotated text sequence. In an embodiment of the present invention, the alignment loss function can use CTC loss. The definition of CTC loss is as follows: L(S) = -ln∑p(z|x) where p(z|x) represents the probability of outputting sequence z given an input x, and S is the training set. The loss function can be interpreted as: the sum of the probabilities of outputting the correct sequence (i.e., the annotated text sequence) after a given sample, and then taking the negative logarithm is the CTC loss function. In some embodiments of the present invention, obtaining the value of the alignment loss function includes: summing the probabilities of each alignment distribution of the audio feature coding sequence and the annotated text sequence to obtain the total probability value;

[0096] Take the negative logarithm of the sum of probabilities and use the result as the value of the alignment loss function.

[0097] S140: Input the annotated text sequence and the audio feature encoding sequence into a first decoding network based on the attention mechanism to obtain an audio feature decoding sequence.

[0098] This invention uses an attention mechanism to train the encoder and decoder networks, resulting in faster training convergence. The attention mechanism, a solution that mimics human attention, aims to quickly filter out high-value information from a large amount of information. This approach addresses the difficulty of obtaining a reasonable final vector representation for long input sequences in the first decoding network. By retaining the output of the first encoding network, this output is then learned using a model incorporating the attention mechanism for information screening.

[0099] S150. Obtain a value of an attention loss function based on the audio feature decoding sequence and the annotated text sequence.

[0100] The attention loss function is calculated based on the decoded audio feature sequence and the annotated text sequence. The attention loss function can use a softmax loss function or a variant of softmax loss. In this embodiment of the present invention, the decoder network output is the result of speech recognition. By introducing the attention loss function, the encoder network is trained to have speech recognition characteristics, ensuring the alignment effect.

[0101] S160. If the value of the alignment loss function is greater than the first loss threshold and / or the value of the attention loss function is greater than the second loss threshold, based on the value of the alignment loss function and the value of the attention loss function, iteratively update the encoder network and the decoder network, and return to execute step S110 until the value of the alignment loss function is less than or equal to the first loss threshold and the value of the attention loss function is less than or equal to the second loss threshold.

[0102] When updating the encoder network and decoder network, the network parameters are updated by calculating the gradient of the loss.

[0103] S170: Use the encoder network after the last update as the speech alignment network.

[0104] The training method in the embodiment of the present invention adopts a dual loss function of alignment loss function and attention loss function for training, so that the trained speech alignment network has both alignment characteristics and speech recognition characteristics.

[0105] In addition, in an embodiment of the present invention, although the dual loss functions of alignment loss function and attention loss function are used for training, surprisingly, only the encoder network is used as the speech alignment network. Therefore, the training method in the present invention uses a lightweight end-to-end network to train the speech alignment network, which can be deployed to the front end of the mobile phone to achieve localized alignment, reduce network transmission time, reduce latency, and have richer applicable scenarios.

[0106] In some embodiments, the trained decoder network may also be used to output speech recognition results, for example, for other usage scenarios.

[0107] In some embodiments of the present invention, the annotated text sequence includes an annotated text sequence in units of characters or an annotated text sequence in units of phonemes. Accordingly, the trained speech alignment network can align character-level sequences or phoneme-level sequences.

[0108] Based on the speech recognition network obtained by the training method of the present invention, an embodiment of the present invention also provides a speech alignment method, which can, on the one hand, utilize the speech alignment network trained according to the embodiment of the present invention, and on the other hand, can optionally perform CTC Alignment (CTC alignment method) processing based on the features output by the speech recognition network, and select the alignment result by scoring.

[0109] Specifically, if Figure 3a 、 Figure 3b As shown, an embodiment of the present invention provides a method for speech alignment, comprising the following steps:

[0110] S210: Obtain a target text sequence and a target audio, where the target text sequence is a lyrics text sequence of the target audio.

[0111] The target text sequence is generated by performing dictionary conversion on the target text.

[0112] S220. Input the audio feature sequence of the target audio into the speech alignment network generated by any of the above training methods to obtain the target audio feature coding sequence.

[0113] A corresponding audio feature sequence is generated based on the target audio. For example, the target audio can be converted into an audio feature sequence through FFT (Fast Fourier Transform).

[0114] S230: Align the target text sequence with the target audio based on the target audio feature encoding sequence.

[0115] The alignment process is to map each character or phoneme in the target text sequence to each time frame of the target audio.

[0116] The target audio feature encoding sequence X = [x1, x2, x3..., xt..., xT] (usually represented by audio features such as fbank or mfcc, where T is the number of target audio time frames), and the target text sequence Y = [y1, y2, y3..., yu..., yU]. The alignment task is to map the characters or phonemes in the target text sequence to the target audio time frames, where the length of X is generally greater than that of Y. If the correspondence between yu and xt is known, this task can be transformed into a classification task at the speech frame level, that is, classifying each time instant xt to obtain yu.

[0117] In some preferred embodiments of the present invention, considering that in speech recognition tasks where the length of the input sequence is greater than the length of the output sequence, the units of the target text sequence are copied and a placeholder blank is inserted.

[0118] Specifically, in the CTC model of the embodiment of the present invention, the following algorithm is used to generate all possible extended sequences of the target text: where N is the output sequence length and T is the input sequence length.

[0119] The expansion process is as follows:

[0120] Generate b0 blank placeholders;

[0121] Loop execution:

[0122] n=1toN;

[0123] Generate the nth token tn times;

[0124] Generate blank placeholder bn times;

[0125] Where tn and bn need to satisfy the restriction: b0+t1+b1+t2+b2+...tn+bn=T.

[0126] Different modeling units in the embodiments of the present invention have different dictionaries. In multiple embodiments of the present invention, modeling units can be used interchangeably with tokens.

[0127] In an embodiment of the present invention, a speech alignment relationship can be obtained by finding the CTC alignment distribution with the highest probability. Specifically, a trained model is used to generate a target audio feature encoding sequence, and all possible CTC alignment distribution probabilities obtained thereby are scored, and the CTC alignment distribution with the highest score is selected. The inventors note that in order to avoid the exponential time complexity caused by exhaustive enumeration, a Viterbi algorithm can be used in another embodiment to reduce the time complexity and solve this problem, as further described below.

[0128] In some embodiments, each character or phoneme in the target text sequence has a different correspondence with each time frame, forming different character / phoneme-time frame orderings. Therefore, a large number of orderings are possible, but not every ordering is optimal. Therefore, in this embodiment, the scores of each ordering can be exhaustively calculated one by one, with the ordering with the highest score being the optimal ordering. For example, the time frame sequence includes t1, t2, t3, t4, and t5, and the text sequence is cat. One ordering method is to have c correspond to t1, t2, a correspond to t3, t4, and t correspond to t5. Another ordering method is to have c correspond to t1, a correspond to t2, t3, t4, and t correspond to t5, and so on. In some embodiments, the ordering method is to have c correspond to t1, a correspond to t2, t3, and t correspond to t4, t5. The last ordering method has the highest score, and this ordering method is the optimal ordering method.

[0129] As previously mentioned, in some embodiments of the present invention, a CTC Viterbi method can be used to quickly select the sort order, reducing alignment time. This CTC Viterbi method can be implemented using the known Viterbi optimal path algorithm. For CTC speech recognition tasks, the path selection at each step is equivalent to the selectable state for each time frame. For example, if word modeling is used, it represents the selectable word or blank placeholder for each time frame; if phoneme modeling is used, it represents the selectable phoneme or blank placeholder for each time frame.

[0130] In some embodiments of the present invention, as previously described, the target audio feature encoding sequence may be the posterior probability distribution of each word or phoneme in the target text sequence at each time instant. The calculated scores may be correlated with these posterior probabilities. Thus, in some embodiments, the scores of the words or phonemes corresponding to each time frame may be expressed using posterior probabilities.

[0131] In some embodiments of the present invention, based on the target audio feature coding sequence, the target text sequence is aligned with the target audio, including: based on the posterior probability distribution, generating the optimal time distribution of each word or each phoneme in the target text sequence distributed to the time frame sequence of the target audio, wherein the optimal time distribution refers to the sequence formed by the distribution of each word or each phoneme in the target text sequence in each time frame, when the overall score or overall posterior probability is the highest, the distribution at this time is the optimal time distribution. More specifically, as Figure 4a As shown, generating the optimal time distribution of each word or each phoneme in the target text sequence to the time frame sequence of the target audio includes:

[0132] S231a, arbitrarily distributing each word or each phoneme in the target text sequence to the time frame sequence of the target audio to generate a full distribution sequence;

[0133] S232a. Calculate the sum of the posterior probabilities of each word or each phoneme in each distribution sequence in the full distribution sequence at the corresponding time based on the target audio feature coding sequence;

[0134] S233a. The distribution sequence with the largest sum of posterior probabilities is taken as the optimal time distribution.

[0135] In some examples, the time frame sequence includes t1, t2, t3, t4, and t5, and the text sequence is cat. One distribution method is that c corresponds to t1 and t2 (the posterior probability is 0.3), a corresponds to t3 and t4 (the posterior probability is 0.1), and t corresponds to t5 (the posterior probability is 0.1). This distribution method results in a total posterior probability of 0.5. Another distribution method is that c corresponds to t1 (the posterior probability is 0.8), a corresponds to t2, t3, and t4 (the posterior probability is 0.1). The posterior probability is 0.1), t corresponds to t5 (posterior probability is 0.1), and the sum of the posterior probabilities of this distribution is 1. In some embodiments, the sorting method is c corresponding to t1 (0.8), a corresponding to t2, t3 (posterior probability is 0.8), t corresponding to t4, t5 (posterior probability is 0.8), the sum of the posterior probabilities is 2.4, the sum of the posterior probabilities is the largest, and the distribution method in which c corresponds to t1, a corresponds to t2, t3, and t corresponds to t4 and t5 is the optimal time distribution method.

[0136] As mentioned above, in other embodiments of the present invention, alignment selection can be performed based on the CTC Viterbi method, specifically, as Figure 4b As shown, based on the target audio feature coding sequence, the optimal time distribution of each word or each phoneme in the target text sequence to the time frame sequence of the target audio is generated, including:

[0137] S231b: Initialize the time frame sequence of the aligned distribution according to the time frame sequence of the target audio.

[0138] S232b. Insert a placeholder before the first character or the first phoneme in the target text sequence, insert a placeholder after the last character or the last phoneme in the target text sequence, and insert placeholders between adjacent characters or phonemes to form a state sequence.

[0139] S233b: Set the state of the first time frame in the aligned distribution to a placeholder or the first character or phoneme in the state sequence.

[0140] S234b: Set the state of the last time frame in the aligned distribution to a placeholder or a last character or phoneme in the state sequence.

[0141] S235b. Based on the state sequence, recursively set the state of the previous time frame in the aligned distribution according to the current time frame state of the aligned distribution, until the first time frame state is set to a placeholder or the first character or first phoneme in the state sequence, and obtain the aligned distribution sequence.

[0142] The current time frame is any time frame before the last time frame in the aligned distribution. Recursively set the state from the last time frame forward until the current time frame is the first time frame, and then output the state corresponding to each time frame.

[0143] In some embodiments of the present invention, Figure 5a As shown, based on the state sequence, the state of the previous time frame in the alignment distribution is recursively set according to the current time frame state of the alignment distribution, until the first time frame state is set to a placeholder or the first character or first phoneme in the state sequence, and the alignment distribution sequence is obtained, including:

[0144] S2351b. If the state S of the current time frame is a placeholder, then based on the state sequence, set the state corresponding to the previous time frame to be the same as the state S of the current time frame, or set the state corresponding to the previous time frame to be the state S-1 before the state of the current time frame;

[0145] S2352b. If the state S of the current time frame is not a placeholder, and the state S of the current time frame is the same as the two states S-2 before the state of the current time frame, then based on the state sequence, set the state corresponding to the previous time frame to be the same as the state S of the current time frame, or set the state corresponding to the previous time frame to be the state S-1 before the state of the current time frame;

[0146] S2353b. If the current time frame state S is not a placeholder, and the current time frame state S is different from the two states S-2 preceding the current time frame state, then based on the state sequence, the state corresponding to the previous time frame is set to be the same as the state S of the current time frame, or the state corresponding to the previous time frame is set to be the state S-1 preceding the state of the current time frame, or the state corresponding to the previous time frame is set to be the two states S-2 preceding the state of the current time frame;

[0147] S2354b, recursively execute the above steps until the first time frame state is set to a placeholder or the first character or the first phoneme in the state sequence, and obtain an aligned distribution sequence.

[0148] like Figure 5b As shown, assuming that the labeled text sequence y is cat, after inserting the placeholder ∈ in the labeled text sequence, a state sequence ∈c∈a∈t∈ is formed, the state of the first time frame x1 is ∈ or c, and the state of the corresponding last time frame x6 is t or ∈. According to the state sequence, when the state S of the current time frame (x4) is a (i.e., not a placeholder), the first two states S-2 are c, and the states are different, then the state of the previous time frame (x3) can be the same as the state S (i.e., a) of the current time frame, or the previous state S-1 (i.e., ∈) of the current time frame state, or the previous two states S-2 (i.e., c) of the current time frame. In some embodiments, the state of the previous frame can be selected by the posterior probability, for example, the state with the largest posterior probability can be selected from a, c, ∈ by the posterior probability. In some embodiments, the selection can also be made uniformly in subsequent calculations.

[0149] In subsequent recursion, such as Figure 5cAs shown, the state S of the current time frame (x3) is ∈ placeholder, then the state of the previous time frame (x2) is ∈ or c, and the state of each time frame is recursively set until the state of the first time frame is encountered and the recursion is terminated when the initial setting is reached. Output all possible alignment distributions. The recursive process in this embodiment is to gradually set the state of the previous time frame, but each time it is set, it is necessary to repeat the step of setting the state according to the state of the current time frame until the previous time frame is the first time frame, and when the state is the same as the initial setting, the state from the first time frame to the current time frame is output in reverse.

[0150] S236b, calculating the sum of the posterior probabilities of the aligned distributions based on the posterior probabilities of the states in the aligned distributions;

[0151] S237b. The alignment distribution with the largest sum of posterior probabilities is taken as the optimal time distribution.

[0152] In some preferred embodiments, step S235b and its substeps can be implemented in combination with S236b and S237b. For example, as before, during recursion, the state of the previous frame can be selected using the posterior probability. For example, the state with the largest posterior probability can be selected from a, c, ∈ using the posterior probability. When the recursion is completed, the alignment distribution with the largest sum of posterior probabilities is obtained as the optimal time distribution.

[0153] In the embodiment of the present invention, the alignment distribution can be calculated by using the above-mentioned reverse recursive method, or by using the forward recursive method, that is, selecting the current time frame according to the state of the previous time frame. Figure 4c As shown, based on the target audio feature coding sequence, the optimal time distribution of each word or each phoneme in the target text sequence to the time frame sequence of the target audio is generated, including:

[0154] S231c, initialize the time frame sequence of the aligned distribution according to the time frame sequence of the target audio.

[0155] S232c. Insert a placeholder before the first character or the first phoneme in the target text sequence, insert a placeholder after the last character or the last phoneme in the target text sequence, and insert placeholders between adjacent characters or phonemes to form a state sequence.

[0156] S233c: Set the state of the first time frame in the aligned distribution to a placeholder or the first character or phoneme in the state sequence.

[0157] S234c. Set the state of the last time frame in the aligned distribution to a placeholder or a last character or phoneme in the state sequence.

[0158] S235c. Based on the state sequence and the current time frame of the aligned distribution, recursively set the state of the current time frame in the aligned distribution according to the state of the previous time frame of the aligned distribution, until the state of the last time frame is set to a placeholder or the last character or the first phoneme in the state sequence, and obtain the aligned distribution sequence.

[0159] In some embodiments of the present invention, Figure 6a As shown, based on the state sequence and the current time frame of the aligned distribution, the state of the current time frame in the aligned distribution is recursively set according to the state of the previous time frame, until the state of the last time frame is set to a placeholder or the last character or the first phoneme in the state sequence, and the aligned distribution sequence is obtained, including:

[0160] S2351c. Based on the state sequence, arbitrarily set the state S of the current time frame of the alignment distribution.

[0161] S2352c: If the current time frame state S is a placeholder, verify that the state corresponding to the previous time frame is the same as the state S of the current time frame, or if the state corresponding to the previous time frame is the state S-1 before the state of the current time frame, determine that the state S of the current time frame is within the target range.

[0162] S2353c. If the state S of the current time frame is not a placeholder, and the state S of the current time frame is the same as the two previous states S-2 of the state of the current time frame, and it is verified that the state corresponding to the previous time frame is the same as the state S of the current time frame, or the state corresponding to the previous time frame is the previous state S-1 of the state of the current time frame, it is determined that the state S of the current time frame is within the target range.

[0163] S2354c. If the current time frame state S is not a placeholder, and the state S of the current time frame is different from the two states S-2 before the state of the current time frame, verify that the state corresponding to the previous time frame is the same as the state S of the current time frame, or the state corresponding to the previous time frame is the state S-1 before the state of the current time frame, or the state corresponding to the previous time frame is the two states S-2 before the state of the current time frame, then determine that the state S of the current time frame is within the target range.

[0164] S2355c. Based on the state of the previous time frame, the state with the highest posterior probability among the states within the target range is obtained as the target state, and the state of the current time frame is set according to the target state.

[0165] S2356c. Recursively execute the above steps until the state of the last time frame is set to a placeholder or the last character or phoneme in the state sequence, and obtain an aligned distribution sequence.

[0166] like Figure 6bAs shown, the state of the previous time frame (x1) is c or ∈. Assume that the state S of the current time frame x2 is set to the second ∈ in the state sequence ∈c∈a∈t∈, and verify that the state of the previous time frame x1 is ∈, then the verification passes, or verify that the previous time frame x1 is c, which is the previous state S-1 of the current time frame state, and the verification passes, so the state S of the current time frame can be ∈.

[0167] Assume that the state S of the current time frame x2 is set to c in the state sequence ∈c∈a∈t∈, the first two states S-2 of the current time frame state do not exist, the current time frame state and the first two states of the current time frame state are different, but the state of the previous time frame x1 can be c, the state of the previous time frame and the state of the current time frame are the same, the verification is passed, so the state S of the current time frame x2 can be c.

[0168] Assume that the state S of the current time frame x2 is a in ∈c∈a∈t∈, the two previous states S-2 of the current time frame state are c, the current time frame state and the two previous states of the current time frame state are different, verify that the state of the previous time frame x1 can be c, and the previous time frame state is the two previous states of the current time frame state, and the verification passes.

[0169] Assume that the current time frame state S is t in ∈c∈a∈t∈, and the two previous states S-2 of the current time frame state are a. The state of the previous time frame x1 should be the current time frame state S, the previous state S-1 of the current time frame state, and the two previous states S-2 of the current time frame state, which are t, ∈, and a respectively. However, if the previous time frame state is c or ∈, the verification fails, and the current frame state is not t. From the verified states (c, ∈, a), the state with the highest posterior probability is selected as the state of the current time frame, and run recursively until the final state is consistent with the initial setting.

[0170] S236c. Based on the posterior probability of each state in the aligned distribution, calculate the sum of the posterior probabilities of each aligned distribution.

[0171] Since the last state may be a word (phoneme) or a placeholder, the score of the alignment distribution ending with the placeholder or the score of the alignment distribution ending with the last word (phoneme) will be calculated respectively, and the alignment distribution with the maximum score will be the optimal distribution.

[0172] S237c. The alignment distribution with the largest sum of posterior probabilities is taken as the optimal time distribution.

[0173] Similarly, in some preferred embodiments, step S235c and its sub-steps may be implemented in combination with S236c and S237c.

[0174] In an embodiment of the present invention, after determining the optimal distribution, backtracking can be performed, that is, the state of the time frame is determined according to the optimal distribution based on the time frame sequence, that is, the time frame is bound to the state, so as to output the timestamp of the word or phoneme in the annotated text.

[0175] Figure 7a 、 Figure 7b 、 Figure 7c Examples of output based on word and phoneme modeling are given. Figure 7a As shown, the text sequence is "Let me see that you are reluctant to leave", and each text or phoneme is aligned with the time frame sequence. In the output optimal distribution, the time frames corresponding to "let" are more than the time frames corresponding to "I".

[0176] This paper proposes an end-to-end lyrics-to-audio speech alignment method, specifically optimized for mobile phones. It eliminates the need for backend server deployment and can be deployed directly on the phone. Users can select audio and corresponding lyrics to use this method for lyrics-to-audio alignment. This solution automatically generates timestamps for lyrics locally, saving labor and time and improving the user experience. The alignment method in this embodiment uses the CTC Viterbi alignment method, significantly improving alignment efficiency.

[0177] In the embodiment of the present invention, Figure 8 As shown, a training device 500 for a speech alignment network is shown, comprising:

[0178] An acquisition module 510 is configured to acquire an annotated text sequence and an audio feature sequence;

[0179] The encoding module 520 is configured to input the audio feature sequence into an encoder network to obtain an audio feature encoding sequence;

[0180] An alignment loss function calculation module 530 is configured to obtain a value of an alignment loss function based on the audio feature encoding sequence and the annotated text sequence;

[0181] A decoding module 540 is configured to input the annotated text sequence and the audio feature encoding sequence into a decoder network based on an attention mechanism to obtain an audio feature decoding sequence;

[0182] an attention loss function calculation module 550 , further configured to obtain a value of an attention loss function based on the audio feature decoding sequence and the annotated text sequence;

[0183] An updating module 560 is configured to iteratively update the encoder network and the decoder network based on the value of the alignment loss function and the value of the attention loss function if the value of the alignment loss function is greater than a first loss threshold and / or the value of the attention loss function is greater than a second loss threshold;

[0184] The loop training module 570 returns to the step of obtaining the labeled text sequence and the audio feature sequence until the value of the alignment loss function is less than or equal to the first loss threshold and the value of the attention loss function is less than or equal to the second loss threshold;

[0185] The output module 580 is configured to use the last updated encoder network as the speech alignment network.

[0186] In some embodiments, the alignment loss function calculation module 530 is specifically configured to sum the probabilities of each alignment distribution of the audio feature coding sequence and the annotated text sequence to obtain a probability sum value; take the negative logarithm of the probability sum value, and use the result as the value of the alignment loss function.

[0187] In some embodiments, the first encoding network is a lightweight end-to-end network.

[0188] In some embodiments, the annotated text sequence includes an annotated text sequence in units of characters or an annotated text sequence in units of phonemes.

[0189] In some embodiments, the training device may combine features of the training method of the speech alignment network of any embodiment, and vice versa, which will not be elaborated here.

[0190] In the embodiment of the present invention, Figure 9 As shown, a speech alignment device 600 is shown, comprising:

[0191] An acquisition module 610 is configured to acquire a target text sequence and a target audio, wherein the target text sequence is a lyrics text sequence of the target audio;

[0192] The encoding module 620 is configured to input the audio feature sequence of the target audio into the speech alignment network generated by any of the above training methods to obtain a target audio feature encoding sequence;

[0193] The alignment module 630 is configured to align the target text sequence with the target audio based on the target audio feature encoding sequence.

[0194] In some embodiments, the target audio feature encoding sequence is the posterior probability distribution of each word or each phoneme in the target text sequence at each time.

[0195] In some embodiments, the alignment module 630 is specifically configured to generate an optimal time distribution of each word or each phoneme in the target text sequence based on the target audio feature coding sequence.

[0196] In some embodiments, the alignment module 630 is further configured to:

[0197] Arbitrarily distributing each word or phoneme in the target text sequence to the time frame sequence of the target audio to generate a full distribution sequence; calculating, based on the target audio feature coding sequence, the sum of the posterior probabilities of each word or phoneme in the to-be-processed text sequence distributed to the corresponding time instants in each distribution sequence in the full distribution alignment distribution sequence;

[0198] The distribution sequence with the largest sum of posterior probabilities is aligned with the distribution sequence as the optimal time distribution.

[0199] In some embodiments, the alignment module 630 is further configured to:

[0200] Initialize the time frame sequence of the aligned distribution according to the time frame sequence of the target audio;

[0201] Inserting a placeholder before the first character or the first phoneme in the target text sequence, inserting a placeholder after the last character or the last phoneme in the target text sequence, and inserting placeholders between adjacent characters or phonemes to form a state sequence;

[0202] Setting the state of the first time frame in the aligned distribution to a placeholder or the first character or phoneme in the state sequence;

[0203] Set the state of the last time frame in the alignment distribution to a placeholder or the last character or phoneme in the state sequence

[0204] Based on the state sequence, recursively setting the state of the previous time frame in the aligned distribution according to the current time frame state of the distribution thereof, until the first time frame state is set to a placeholder or the first character or the first phoneme in the state sequence, thereby obtaining an aligned distribution sequence;

[0205] Calculating a sum of the posterior probabilities of the aligned distributions based on the posterior probabilities of the states in the aligned distributions;

[0206] The alignment distribution with the largest sum of posterior probabilities is taken as the optimal time distribution.

[0207] In some embodiments, the alignment module 630 is further configured to:

[0208] If the state S of the current time frame is a placeholder, then based on the state sequence, the state corresponding to the previous time frame of the current time frame is set to be the same as the state S of the current time frame, or the state corresponding to the previous time frame is set to be the state S-1 before the state of the current time frame;

[0209] If the state S of the current time frame is not a placeholder, and the state S of the current time frame is the same as the two states S-2 before the state of the current time frame, then based on the state sequence, the state corresponding to the previous time frame is set to be the same as the state S of the current time frame, or the state corresponding to the previous time frame is set to be the state S-1 before the state of the current time frame;

[0210] If the current time frame state S is not a placeholder, and the current time frame state S is different from the two states S-2 before the current time frame state, then based on the state sequence, the state corresponding to the previous time frame is set to be the same as the state S of the current time frame, or the state corresponding to the previous time frame is set to be the state S-1 before the state of the current time frame, or the state corresponding to the previous time frame is set to be the two states S-2 before the state of the current time frame;

[0211] The above steps are recursively executed until the first time frame state is set to a placeholder or the first character or the first phoneme in the state sequence, and an aligned distribution sequence is obtained.

[0212] In some embodiments, the alignment module 630 is further configured to:

[0213] Initialize the time frame sequence of the aligned distribution according to the time frame sequence of the target audio;

[0214] Inserting a placeholder before the first character or the first phoneme in the target text sequence, inserting a placeholder after the last character or the last phoneme in the target text sequence, and inserting placeholders between adjacent characters or phonemes to form a state sequence;

[0215] Setting the state of the first time frame in the aligned distribution to a placeholder or the first character or phoneme in the state sequence;

[0216] Set the state of the last time frame in the alignment distribution to a placeholder or the last character or phoneme in the state sequence

[0217] Based on the state sequence, recursively setting the state of the current time frame in the alignment distribution according to the state of the previous time frame, until the state of the last time frame is set to a placeholder or the last character or the first phoneme in the state sequence, to obtain an alignment distribution sequence;

[0218] Calculating the sum of the posterior probabilities of the aligned distributions based on the posterior probabilities of the states in the aligned distributions in the aligned distribution sequence;

[0219] The alignment distribution with the largest sum of posterior probabilities is taken as the optimal time distribution.

[0220] In some embodiments, the alignment module 630 is further configured to:

[0221] Based on the state sequence, arbitrarily set the state S of the current time frame;

[0222] If the current time frame state S is a placeholder, verify that the state corresponding to the previous time frame is the same as the state S of the current time frame, or if the state of the previous time frame is the state S-1 before the state of the current time frame, determine that the state S of the current time frame is within the target range;

[0223] If the state S of the current time frame is not a placeholder, and the state S of the current time frame is the same as the two states S-2 before the state of the current time frame, and the state corresponding to the previous time frame is verified to be the same as the state S of the current time frame, or the state corresponding to the previous time frame is the state S-1 before the state of the current time frame, then the state S of the current time frame is determined to be within the target range;

[0224] If the current time frame state S is not a placeholder, and the current time frame state S is different from the two states S-2 before the current time frame state, verify that the state corresponding to the previous time frame is the same as the current time frame state S, or the state corresponding to the previous time frame is the state S-1 before the current time frame state, or the state corresponding to the previous time frame is the two states S-2 before the current time frame state, then determine that the current time frame state S is within the target range;

[0225] Based on the state of the previous time frame, the state with the highest posterior probability among the states within the target range is obtained as the target state, and the state of the current time frame is set according to the target state;

[0226] The above steps are recursively executed until the state of the last time frame is set to a placeholder or the last character or phoneme in the state sequence, and an aligned distribution sequence is obtained.

[0227] In some embodiments, the apparatus 600 may be combined with the method features of speech alignment of any embodiment, and vice versa, which will not be elaborated here.

[0228] In an embodiment of the present invention, an electronic device is provided, comprising: a processor and a memory storing a computer program, wherein the processor is configured to execute any method for training a speech alignment network or a method for speech alignment of the present invention when running the computer program.

[0229] Figure 10 A schematic diagram of an electronic device 1000 that can implement a method or implement an embodiment of the present invention is shown. In some embodiments, the method may include more or fewer electronic devices than shown. In some embodiments, the method may be implemented using a single electronic device or multiple electronic devices. In some embodiments, the method may be implemented using cloud-based or distributed electronic devices.

[0230] like Figure 10 As shown, electronic device 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate operations and processes according to the programs and / or data stored in read-only memory (ROM) 1002 or the programs and / or data loaded from storage portion 1008 into random access memory (RAM) 1003. CPU 1001 can be a multi-core processor or can include multiple processors. In some embodiments, CPU 1001 can include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM 1003, various programs and data required for the operation of electronic device 1000 are also stored. CPU 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0231] The above-mentioned processor and memory are used together to execute the program stored in the memory. When the program is executed by the computer, it can implement the steps or functions of the high-resolution audio generation model training method, high-resolution audio generation method, and sound effect switching method described in the above-mentioned embodiments.

[0232] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, a touch screen, and the like; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1008 including a hard disk and the like; and a communication section 1009 including a network interface card such as a LAN card or a modem. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1010 as needed, so that computer programs read therefrom can be installed into the storage section 1008 as needed. Figure 10 Only some components are shown schematically, which does not mean that the computer system 1000 only includes Figure 10 Components shown.

[0233] In some embodiments, the electronic device 1000 refers to a mobile terminal, including a mobile phone, a car terminal, a smart TV, etc. Taking a mobile phone as an example, the electronic device 1000 also includes a display screen with a touch function, an external speaker, a gyroscope, a camera, a 4G / 5G antenna and other device modules.

[0234] The systems, devices, modules, or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smartphone, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server, or a combination thereof.

[0235] Although not shown, in an embodiment of the present invention, a storage medium is provided, wherein the storage medium stores a computer program, and the computer program is configured to execute any file difference-based compilation method according to any embodiment of the present invention when executed.

[0236] Storage media in embodiments of the present invention include permanent and non-permanent, removable and non-removable items that can be used to store information using any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0237] The methods, programs, systems, and apparatuses of the embodiments of the present invention may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.

[0238] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, those skilled in the art will appreciate that the functional modules / units or controllers and related method steps described in the above embodiments may be implemented using software, hardware, or a combination of software / hardware.

[0239] Unless explicitly stated, the actions or steps of the methods, procedures, and methods described in accordance with the embodiments of the present invention do not have to be performed in a specific order and can still achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0240] In this document, multiple embodiments of the present invention are described, but for the sake of brevity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the embodiments may be omitted. In this document, "one embodiment", "some embodiments", "example", "specific example", or "some examples" are intended to apply to at least one embodiment or example according to the present invention, but not all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. Those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are mutually contradictory.

[0241] While the exemplary systems and methods of the present invention have been specifically shown and described with reference to the foregoing embodiments, these are merely examples of the best modes for implementing the present systems and methods. Those skilled in the art will appreciate that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A method for training a speech alignment network, characterized in that: The steps include: Obtain annotated text sequence and audio feature sequence; Inputting the audio feature sequence into an encoder network to obtain an audio feature encoding sequence; Obtaining a value of an alignment loss function based on the audio feature encoding sequence and the annotated text sequence; Inputting the annotated text sequence and the audio feature encoding sequence into a decoder network based on an attention mechanism to obtain an audio feature decoding sequence; Obtaining a value of an attention loss function based on the audio feature decoding sequence and the annotated text sequence; If the value of the alignment loss function is greater than the first loss threshold or the value of the attention loss function is greater than the second loss threshold, based on the value of the alignment loss function and the value of the attention loss function, iteratively update the encoder network and the decoder network, and return to execute the steps of obtaining the labeled text sequence and the audio feature sequence, obtaining the audio feature encoding sequence, obtaining the value of the alignment loss function, obtaining the audio feature decoding sequence, obtaining the value of the attention loss function, and iteratively updating the encoder network and the decoder network until the value of the alignment loss function is less than or equal to the first loss threshold and the value of the attention loss function is less than or equal to the second loss threshold; The encoder network after the last update is used as the speech alignment network.

2. The training method according to claim 1, characterized in that Obtaining the value of the alignment loss function includes: Summing the probability of each alignment distribution of the audio feature coding sequence and the annotated text sequence to obtain a probability sum value; The negative logarithm of the probability sum is taken, and the result is used as the value of the alignment loss function.

3. The training method according to claim 1, characterized in that The encoder network is a lightweight end-to-end network.

4. The training method according to claim 1, characterized in that The annotated text sequence includes an annotated text sequence in units of characters or an annotated text sequence in units of phonemes.

5. A method for speech alignment, characterized in that: include: Obtain a target text sequence and a target audio, wherein the target text sequence is a lyrics text sequence of the target audio; Inputting the audio feature sequence of the target audio into the speech alignment network generated by any training method of claims 1 to 4 to obtain a target audio feature coding sequence; Based on the target audio feature encoding sequence, the target text sequence is aligned with the target audio.

6. The method according to claim 5, characterized in that The target audio feature coding sequence is the posterior probability distribution of each word or each phoneme in the target text sequence at each moment.

7. The method according to claim 6, characterized in that The aligning the target text sequence with the target audio based on the target audio feature encoding sequence includes: Based on the posterior probability distribution, an optimal time distribution of each word or each phoneme in the target text sequence to the time frame sequence of the target audio is generated, so that the target text sequence is aligned with the target audio.

8. The method according to claim 7, characterized in that The step of generating an optimal time distribution of each word or each phoneme in the target text sequence to the time frame sequence of the target audio based on the posterior probability distribution includes: Arbitrarily distributing each word or each phoneme in the target text sequence to the time frame sequence of the target audio to generate a full distribution sequence; Calculate the sum of the posterior probabilities of each word or each phoneme in each distribution sequence in the full distribution sequence at the corresponding time; The distribution sequence with the largest sum of posterior probabilities is taken as the optimal time distribution.

9. The method according to claim 7, characterized in that The step of generating an optimal time distribution of each word or each phoneme in the target text sequence to the time frame sequence of the target audio based on the posterior probability distribution includes: Initialize the time frame sequence of the aligned distribution according to the time frame sequence of the target audio; Inserting a placeholder before the first character or the first phoneme in the target text sequence, inserting a placeholder after the last character or the last phoneme in the target text sequence, and inserting a placeholder between adjacent characters or phonemes to form a state sequence; Setting the state of the first time frame in the time frame sequence of the aligned distribution to a placeholder or the first character or phoneme in the state sequence; Setting the state of the last time frame in the time frame sequence of the aligned distribution to a placeholder or a last character or a last phoneme in the state sequence; Obtaining an aligned distribution sequence based on the state sequence, the state of the first time frame in the aligned distribution time frame sequence, and the state of the last time frame; Calculating the sum of the posterior probabilities of the aligned distributions based on the posterior probabilities of the states in the aligned distributions in the aligned distribution sequence; The alignment distribution with the largest sum of posterior probabilities is taken as the optimal time distribution.

10. The method according to claim 9, characterized in that The obtaining of an aligned distribution sequence based on the state sequence, the state of the first time frame in the aligned distribution time frame sequence, and the state of the last time frame includes: Based on the state sequence, the state of the previous time frame in the aligned distribution is recursively set according to the state of the current time frame of the aligned distribution, until the state of the first time frame of the aligned distribution is set to a placeholder or the first character or first phoneme in the state sequence, thereby obtaining an aligned distribution sequence.

11. The method according to claim 10, characterized in that The step of recursively setting the state of the previous time frame in the aligned distribution according to the state of the current time frame of the aligned distribution based on the state sequence until the state of the first time frame of the aligned distribution is set to be a placeholder or the first character or the first phoneme in the state sequence, and obtaining the aligned distribution sequence includes: If the state of the current time frame of the aligned distribution is a placeholder, then based on the state sequence, setting the state of the previous time frame of the current time frame as the state of the current time frame, or setting the state of the previous time frame as the state before the state of the current time frame; If the state of the current time frame is not a placeholder, and the state of the current time frame is the same as the two previous states of the state of the current time frame, then set the state of the previous time frame of the current time frame as the state of the current time frame, or set the state corresponding to the previous time frame as the state before the state of the current time frame; If the current time frame state is not a placeholder, and the state of the current time frame is different from the two states before the state of the current time frame, then the state of the previous time frame of the current time frame is set as the state of the current time frame, or the state of the previous time frame is set to the state before the state of the current time frame, or the state of the previous time frame is set to the two states before the state of the current time frame; The above steps are recursively executed until the first time frame state is set to a placeholder or the first character or the first phoneme in the state sequence, and an aligned distribution sequence is obtained.

12. The method according to claim 9, characterized in that The obtaining of an aligned distribution sequence based on the state sequence, the state of the first time frame in the aligned distribution time frame sequence, and the state of the last time frame includes: Based on the state sequence and the current time frame of the aligned distribution, the state of the current time frame in the aligned distribution is recursively set according to the state of the previous time frame of the aligned distribution, until the state of the last time frame is set to a placeholder or the last character or the first phoneme in the state sequence, and the aligned distribution sequence is obtained.

13. The method according to claim 12, characterized in that The step of recursively setting the state of the current time frame in the aligned distribution based on the state of the previous time frame of the aligned distribution and the state of the current time frame based on the state of the aligned distribution until the state of the last time frame is set to a placeholder or the last character or the first phoneme in the state sequence, and obtaining the aligned distribution sequence includes: Based on the state sequence, arbitrarily set the state of the current time frame; If the current time frame state is a placeholder, it is verified that the state of the previous time frame is the same as the state of the current time frame, or the state of the previous time frame is the state before the state of the current time frame, and the state of the current time frame is determined to be within the target range; If the status of the current time frame is not a placeholder, and the status of the current time frame is the same as the two previous statuses of the current time frame, and the status corresponding to the previous time frame is verified to be the same as the status of the current time frame, or the status corresponding to the previous time frame is the previous status of the current time frame, then the status of the current time frame is determined to be within the target range; If the current time frame status is not a placeholder, and the current time frame status is different from the two previous statuses of the current time frame status, and the status corresponding to the previous time frame is verified to be the same as the current time frame status, or the status corresponding to the previous time frame is the previous status of the current time frame status, or the status corresponding to the previous time frame is the two previous statuses of the current time frame status, then the current time frame status is determined to be within the target range; Based on the state of the previous time frame, the state with the highest posterior probability among the states within the target range is obtained as the target state, and the state of the current time frame is set according to the target state; The above steps are recursively executed until the state of the last time frame is set to a placeholder or the last character or phoneme in the state sequence, and an aligned distribution sequence is obtained.

14. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

15. An electronic device, characterized in that: include: A processor and a memory storing a computer program, wherein the processor is configured to perform the method according to any one of claims 1 to 13 when running the computer program.

Images