An unsupervised method and apparatus for automatic audio-text alignment.

By employing an unsupervised audio-text automatic alignment method, phoneme sequences are converted into syllable type and speech attribute sequences. Hidden Markov models and recurrent neural networks are used for syllable and phoneme alignment, solving the problem of low efficiency in existing technologies and achieving automatic alignment across languages.

CN117012201BActive Publication Date: 2026-11-10BEIJING LANGUAGE AND CULTURE UNIVERSITY
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202310855904.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-07-12
Publication Date
2026-11-10
Estimated Expiration
2043-07-12

AI Technical Summary

Technical Problem

Existing methods for automatic audio and text alignment require manual alignment of data to train the model, resulting in low efficiency and limitations to specific languages, failing to achieve unsupervised automatic alignment.

Method used

An unsupervised approach is adopted, which converts phoneme sequences into syllable type and speech attribute sequences, and uses hidden Markov models and recurrent neural networks to perform syllable and phoneme alignment. Combined with the best alignment path retrieval algorithm, self-learning and iterative operations are performed directly on the data that needs to be aligned, avoiding the step of manually aligning training samples.

Benefits of technology

It improves the efficiency of audio-text alignment, saves training and time costs, and can be generalized to alignment tasks of any language, achieving unsupervised automatic alignment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117012201B_ABST
    Figure CN117012201B_ABST
Patent Text Reader

Abstract

This invention relates to the field of speech signal processing technology, specifically to an unsupervised method and apparatus for automatic audio-text alignment. The method includes: acquiring audio and transcribed text; obtaining a phoneme sequence corresponding to the transcribed text; converting the phoneme sequence into a syllable type sequence and a speech attribute sequence; performing syllable-level alignment based on the audio and the audio syllable type sequence; and determining the phoneme-level alignment result based on the syllable-level alignment result, the speech attribute sequence, a speech attribute sequence prediction model, and an optimal alignment path retrieval algorithm, thus completing the audio-text alignment operation. Using this invention can save training costs and time costs, significantly improving the efficiency of audio-text alignment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, and in particular to an unsupervised method and apparatus for automatic audio-text alignment. Background Technology

[0002] Speech is one of the most direct, convenient, and natural ways for humans to exchange information. Scientific research on human speech has a long history and yields fruitful results. With continuous improvements in methods of data collection, recording, encoding, storage, and processing, the development of speech resources has made significant progress. Speech resources play a crucial role in intelligent speech technologies, such as automatic speech recognition, text transcription, speech synthesis, and spoken document retrieval. Simultaneously, speech resources have received high attention from linguists, becoming an important component of linguistic research. Speech resources at least include the collected audio and the corresponding transcribed text. Since language is recursively organized into different levels of units, an ideal speech resource should include transcribed content from different language levels.

[0003] Currently, phoneme transcription is the most common method. After professional investigation, the phonological system of the investigated language is obtained. Then, professionals use the phonemes in the phonological system as the transcription text of the listening results. In this way, the entire audio is transcribed as a phoneme sequence. Further, it is necessary to determine the start and end times of each phoneme in the audio, which is the alignment problem between the audio and the transcribed text. Compared to collecting a large amount of audio and transcribed text, obtaining accurate phoneme alignment is relatively difficult. Statistics show that professionals using manual alignment tools such as Praat need to listen to the audio while viewing the spectrogram, segment the audio fragments while adding the transcription content, and the time spent is at least 100 times the length of the audio to be processed. Existing automatic alignment tools mainly rely on the known phonological system of a language to build an acoustic model of each phoneme in the phonological system using a Hidden Markov Model (HMM), and then use the acoustic model of the phonemes to determine the start and end times of the phonemes in the audio. However, the acoustic model of the phonemes needs to be trained on already aligned speech resources. To obtain the training data for alignment, it is necessary to return to the route of using manual alignment tools.

[0004] Therefore, avoiding the supervised solution approach of manually aligning data first and then training the model, and instead directly operating on the data that needs alignment to form an unsupervised automatic alignment method has become a current focus of demand. Summary of the Invention

[0005] This invention provides an unsupervised method and apparatus for automatic audio-text alignment. The technical solution is as follows:

[0006] On the one hand, an unsupervised method for automatic audio-text alignment is provided, implemented by an electronic device, which includes:

[0007] S1. Acquire audio and transcribed text, and obtain the phoneme sequence corresponding to the transcribed text;

[0008] S2. Convert the phoneme sequence into a syllable type sequence and a speech attribute sequence;

[0009] S3. Align the audio at the syllable level according to the audio and the audio syllable type sequence;

[0010] S4. Based on the syllable-level alignment results, the speech attribute sequence, the speech attribute sequence prediction model, and the optimal alignment path retrieval algorithm, determine the phoneme-level alignment results, and then complete the alignment operation between the audio and the transcribed text.

[0011] Optionally, the syllable types in the syllable type sequence include four types: V, VC, CV, and CVC, and each syllable type includes one or more phonemes;

[0012] The speech attribute sequence includes nine attributes: low vowel, mid vowel, high vowel, glissando, lisping, nasal, fricative, affricate, and plosive. Each attribute corresponds to one phoneme.

[0013] Optionally, step S3, which aligns syllables at the syllable level based on the audio and the audio syllable type sequence, includes:

[0014] S31. The audio is processed using a speech activity detection algorithm to distinguish between silent segments and non-silent segments. The non-silent segments are converted into loudness envelopes that change over time. Based on the loudness, syllable segments to be adjusted are obtained, and each syllable segment to be adjusted is divided into 3 sub-segments.

[0015] S32. Train the syllable model using the syllable segment to be adjusted and the corresponding sub-segment, and train the silence model using the silence segment to obtain a new alignment result; the syllable model is a hidden Markov model with 12 states, and the silence model is a hidden Markov model with 5 states.

[0016] S33. Compare the new alignment result with the previous alignment result. If the new alignment result is different from the previous alignment result, set the syllable segment in the new alignment result as the syllable segment to be adjusted and proceed to S32. If the new alignment result is the same as the previous alignment result, determine the new alignment result as the syllable-level alignment result.

[0017] Optionally, the hidden Markov model with 12 states includes five parts: start, syllable beginning, syllable core, syllable end, and end. The start and end parts are each set with one state, the syllable beginning is set with four states, and the syllable core and syllable end are each set with three states.

[0018] The hidden Markov model with five states includes five parts: start, syllable beginning, syllable core, syllable end, and syllable ending, with one state set for each part.

[0019] Optionally, step S32 uses the syllable segment to be adjusted and the corresponding sub-segment to train the syllable model, and uses the silence segment to train the silence model to obtain a new alignment result, including:

[0020] The syllable model is fitted using the syllable segment to be adjusted, the silence model is fitted using the silence segment to be adjusted, the first version of the syllable model and silence model is trained using the HTK toolkit, the model parameters are learned using the Welch algorithm, and the new alignment result is solved using the Viterbi algorithm.

[0021] Optionally, step S4 determines the phoneme-level alignment result based on the syllable-level alignment result, the speech attribute sequence, the speech attribute sequence prediction model, and the optimal alignment path retrieval algorithm, thereby completing the alignment operation between the audio and the transcribed text, including:

[0022] S41. Build an initial speech attribute sequence prediction model, wherein the speech attribute sequence prediction model is based on a recurrent neural network (RNN).

[0023] S42. Using connectionist temporal classification (CTC) cost and reconstruction cost as training objectives, train the initialized speech attribute sequence prediction model using syllable-level alignment results and the speech attribute sequence.

[0024] S43. After training is completed, the audio is input into the trained speech attribute sequence prediction model to obtain the probability matrix of each frame belonging to each speech attribute.

[0025] S44. Using the best alignment path retrieval algorithm, select the best alignment path in the probability matrix, determine the phoneme-level alignment result using the best alignment path, and then complete the alignment operation between the audio and the transcribed text.

[0026] Optionally, the main body of the network architecture of the speech attribute sequence prediction model consists of an encoder and a decoder. Both the encoder and the decoder are composed of two bidirectional long short time memory (Bi-LSTM), and each Bi-LSTM layer has 512 units.

[0027] For CTC cost, a dense layer with softmax activation is used after the decoder. The dense layer contains 10 network units, corresponding to 9 speech attribute classes and a blank class, respectively.

[0028] For reconstruction costs, a dense layer with tanh activation is used after the decoder.

[0029] Optionally, step S44 uses an optimal alignment path retrieval algorithm to select the optimal alignment path from the probability matrix, determines the phoneme-level alignment result using the optimal alignment path, and thus completes the alignment operation between the audio and the transcribed text, including:

[0030] S441. Using Viterbi's dynamic programming algorithm, calculate the cumulative score matrix of the probability matrix;

[0031] S442. Remove blank labels from the cumulative score matrix, use beam search decoding to retrieve the best path in the cumulative score matrix, and determine the speech attribute sequence with time points based on the best path.

[0032] S443. Determine the phoneme sequence with time points based on the speech attribute sequence with time points, and complete the alignment of the transcribed text and audio based on the phoneme sequence with time points.

[0033] On the other hand, an unsupervised audio-text automatic alignment device is provided, which is applied to an unsupervised audio-text automatic alignment method. The device includes:

[0034] The acquisition module is used to acquire audio and transcribe text, and to acquire the phoneme sequence corresponding to the transcribed text.

[0035] The conversion module is used to convert the phoneme sequence into a syllable type sequence and a speech attribute sequence;

[0036] The first alignment module is used to perform syllable-level alignment based on the audio and the audio syllable type sequence;

[0037] The second alignment module is used to determine the phoneme-level alignment result based on the syllable-level alignment result, the speech attribute sequence, the speech attribute sequence prediction model, and the optimal alignment path retrieval algorithm, and to complete the alignment operation between the audio and the transcribed text.

[0038] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described unsupervised automatic audio and text alignment method.

[0039] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the above-described unsupervised audio-text automatic alignment method.

[0040] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0041] In this invention, audio and transcribed text are collected to obtain the phoneme sequence corresponding to the transcribed text. The phoneme sequence is converted into a syllable type sequence and a speech attribute sequence. Syllable-level alignment is performed based on the audio and the audio syllable type sequence. Based on the syllable-level alignment result, the speech attribute sequence, the speech attribute sequence prediction model, and the optimal alignment path retrieval algorithm, the phoneme-level alignment result is determined, completing the audio-to-transcribed text alignment operation. The alignment method of this invention eliminates the need for manual alignment to obtain training samples and for training the model using training samples. Instead, the model is trained while being used, saving training costs and time, and greatly improving the efficiency of audio-to-text alignment. Furthermore, by utilizing common syllable types and speech attributes as prior knowledge, this method can be extended to alignment tasks of any language, making it an unsupervised automatic alignment method, thus expanding the scope and range of its application. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of an unsupervised audio and text automatic alignment method provided in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of the audio and the corresponding transcription at different levels provided in the embodiments of the present invention;

[0045] Figure 3 This is a schematic diagram of an alignment process provided in an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of a syllable model provided in an embodiment of the present invention;

[0047] Figure 5 This is a block diagram of an unsupervised audio and text automatic alignment device provided in an embodiment of the present invention;

[0048] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0049] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0050] This invention provides an unsupervised method for automatic audio-text alignment, which can be implemented by an electronic device, such as a terminal or a server. Figure 1 The flowchart shown is for an unsupervised audio-text automatic alignment method. The processing flow of this method may include the following steps:

[0051] S1. Acquire audio and transcribed text, and obtain the phoneme sequence corresponding to the transcribed text.

[0052] In one feasible implementation, the speech resource includes at least the acquired audio and the corresponding transcribed text. Through professional investigation, the phonological system of the investigated language can be obtained. Then, professionals use the phonemes from the phonological system as the transcribed text of the listening results. In this way, the entire audio is often transcribed as a phoneme sequence.

[0053] S2. Convert the phoneme sequence into a syllable type sequence and a speech attribute sequence.

[0054] Optionally, the syllable types in the syllable type sequence may include four types: V, VC, CV, and CVC, with each syllable type including one or more phonemes.

[0055] The speech attributes in the speech attribute sequence can include nine attributes: low vowel, mid vowel, high vowel, glissando, nasal consonant, fricative, affricate, and plosive. Each attribute corresponds to one phoneme.

[0056] In one feasible implementation, after the collected language is saved in audio format, professionals record the phonemes, syllables, vocabulary, and prosody—at different levels related to linguistics and phonetics—in transcribed text based on the listening results. This provides important data for intelligent speech technology and linguistic research. This invention can provide a universal transcribed text format for different languages ​​using syllable types and speech attributes. At the syllable level, each syllable segment is classified into one of four syllable types. At the phoneme level, each phoneme segment is classified into one of nine speech attributes. Ultimately, two transcribed sequences are formed in the transcribed text: a syllable type sequence and a speech attribute sequence.

[0057] Based on research findings in linguistics and phonetics, this invention summarizes four syllable types and nine phonetic attributes, providing a universal transcription method for audio content in different languages, including syllable-level transcription and phoneme-level transcription. Linguistic research shows that language is recursively organized into units at different levels. Among these, the syllable is a crucial first-level unit common to all languages. Based on the number of vowels (V) or consonants (C) in syllable components, and statistical analysis of numerous languages, it has been found that four syllable types—V, VC, CV, and CVC—account for over 80% of all syllable types, and many complex syllable types can be categorized into these four types. Furthermore, phonetic research indicates that the smallest unit of speech that humans can produce can be summarized into nine phonetic attributes: low vowels, mid vowels, high vowels, glissando, liquids, nasals, fricatives, affricates, and plosives. Therefore, based on the aforementioned summarized types and attributes, such as... Figure 2 As shown, the phoneme sequence is converted into a syllable type sequence and a speech attribute sequence.

[0058] S3. Align the audio at the syllable level based on the audio and the sequence of audio syllable types.

[0059] In one feasible implementation, the transcribed content in the text should be aligned temporally with the corresponding audio segment, i.e., the start and end times of the transcribed content in the audio must be determined. The most common audio-text alignment involves determining the start and end times of each phoneme at the phoneme level in the audio. Obtaining accurate phoneme alignment is relatively difficult compared to collecting large amounts of audio and transcribed text. Statistics show that professionals using manual alignment tools need to listen to the audio while viewing the spectrogram, segmenting the audio segments while adding the transcribed content, which takes at least 100 times the length of the audio to be processed. Therefore, this invention decomposes the phoneme alignment problem at the phoneme level into two steps. The first step performs alignment of a longer syllable level to obtain syllable segments and their start and end times; based on obtaining the syllable segments, the second step performs phoneme alignment of a shorter phoneme level to obtain phoneme segments and their start and end times, such as... Figure 3 As shown.

[0060] The main idea behind existing automatic alignment tools is to use Hidden Markov Models (HMMs) to build an acoustic model for each phoneme in a given language's phonological system, and then use these models to determine the start and end times of phonemes in the audio. Therefore, the phoneme acoustic models need to be trained on already aligned speech resources. This type of supervised method leads to a chicken-and-egg problem: training the acoustic model requires the aligned corpus, while obtaining the alignment result requires a well-trained acoustic model. Furthermore, the phoneme acoustic models can only be used for the speech resources currently requiring alignment. For new alignments, retraining is necessary.

[0061] This invention employs an iterative solution process of pre-alignment—model learning—re-alignment when performing syllable layer alignment. It operates directly on the data that needs alignment, eliminating the need to train the model using already aligned data. Instead, it performs self-learning and iterative operations directly on the data that needs alignment, without being constrained by any specific language. This is an unsupervised, universal, and automatic alignment method.

[0062] Optionally, the specific operation of S3 may include the following steps S31-S33:

[0063] S31. Use a speech activity detection algorithm to process the audio, distinguish between silent segments and non-silent segments, convert the non-silent segments into loudness envelopes that change over time, obtain the syllable segments to be adjusted based on the loudness, and divide each syllable segment to be adjusted into 3 sub-segments.

[0064] In one feasible implementation, the voice activity detection algorithm can be the VAD (Voice Activity Detection) algorithm. The operation of using this algorithm to identify silent segments can refer to the methods commonly used in the prior art, which will not be described in detail in this embodiment of the invention.

[0065] In the loudness envelope, the maximum value is determined as the syllable kernel, and the local minimum value between the syllable kernels is determined as the syllable boundary, thereby obtaining the syllable segment to be adjusted.

[0066] Furthermore, each syllable segment to be adjusted is divided into 3 sub-segments. The segment from the left minimum to half of the maximum value is the first segment of the syllable, the segment from half of the maximum value to the right minimum value is the last segment of the syllable, and the middle segment is the nucleus segment.

[0067] S32. Train the syllable model using the syllable segment to be adjusted and the corresponding sub-segment, and train the silence model using the silence segment to obtain a new alignment result.

[0068] The scale model is a 12-state Hidden Markov Model (HMM), and the silence model is a 5-state HMM. The 12-state HMM can include five parts: start, syllable beginning, syllable core, syllable end, and syllable coda. Figure 4 As shown, the beginning and end parts each have 1 state, the syllable beginning has 4 states, and the syllable core and syllable end each have 3 states; a hidden Markov model with 5 states can include 5 parts: beginning, syllable beginning, syllable core, syllable end, and end, with each part having 1 state.

[0069] Alternatively, the specific operation of S32 can be as follows:

[0070] The syllable model is fitted using the syllable segment to be adjusted, the silence model is fitted using the silence segment to be adjusted, the first version of the syllable model and silence model is trained using the HTK toolkit, the model parameters are learned using the Welch algorithm, and the new alignment result is solved using the Viterbi algorithm.

[0071] In one feasible implementation, the Hidden Markov Model used in this embodiment of the invention is an unsupervised fitting HMM algorithm, namely the Baum-Welch algorithm. Using the HTK toolkit, the Welch algorithm, and the Viterbi algorithm are all common methods of this algorithm, which will not be elaborated here in this embodiment of the invention.

[0072] S33. Compare the new alignment result with the previous alignment result. If the new alignment result is different from the previous alignment result, set the syllable segment in the new alignment result as the syllable segment to be adjusted and proceed to S32. If the new alignment result is the same as the previous alignment result, determine the new alignment result as the syllable-level alignment result.

[0073] In one feasible implementation, once a new alignment result is obtained, the syllable and silence models can be updated using the aligned syllable and silence segments. New alignments can then be obtained using the new versions of the syllable and silence models. This iterative process of model learning and realignment is repeated until the resulting alignment no longer changes.

[0074] S4. Based on the syllable-level alignment results, speech attribute sequences, speech attribute sequence prediction models, and the optimal alignment path retrieval algorithm, determine the phoneme-level alignment results, and then complete the alignment operation between the audio and the transcribed text.

[0075] In one feasible implementation, after syllable-level alignment is achieved, phoneme-level phoneme alignment is performed based on the syllable-level alignment results. Phoneme-level phoneme alignment is decomposed into two steps: predicting a sequence of speech attributes from the audio and retrieving the optimal alignment path from the predicted sequence.

[0076] Optionally, the specific operation of S4 may include the following steps S41-S44:

[0077] S41. Build an initial speech attribute sequence prediction model, which is based on a recurrent neural network (RNN).

[0078] S42. Using connectionist temporal classification CTC cost and reconstruction cost as training objectives, the initial speech attribute sequence prediction model is trained using syllable-level alignment results and speech attribute sequences.

[0079] Optionally, the network architecture of the speech attribute sequence prediction model consists of an encoder and a decoder. Both the encoder and decoder are composed of two bidirectional long short-time memory (Bi-LSTM) layers, each with 512 units. For CTC cost, a dense layer with softmax activation is used after the decoder. The dense layer contains 10 network units, corresponding to the nine speech attribute classes and the blank class, respectively. For reconstruction cost, a dense layer with tanh activation is used after the decoder.

[0080] In one feasible implementation, CTC cost can ensure that the speech attribute sequence predicted by the network is as close as possible to the given speech attribute sequence, while reconstruction cost helps the predicted content appear at the accurate time location.

[0081] S43. After training is completed, input the audio into the trained speech attribute sequence prediction model to obtain the probability matrix of each frame belonging to each speech attribute.

[0082] In one feasible implementation, the output of the speech attribute sequence prediction model is a 9×m matrix representing the probability that each frame of the audio belongs to each speech attribute, where m represents the number of frames in the audio. Training is complete when the sum of the CTC cost and the reconstruction cost is minimized.

[0083] S44. Using the best alignment path retrieval algorithm, select the best alignment path in the probability matrix, determine the phoneme-level alignment result using the best alignment path, and then complete the alignment operation between the audio and the transcribed text.

[0084] Optionally, the specific operation of S44 may include the following steps S441-S443:

[0085] S441. Using Viterbi's dynamic programming algorithm, calculate the cumulative score matrix of the probability matrix.

[0086] In one feasible implementation, the method for calculating the cumulative score matrix can be a commonly used calculation method in the prior art, which will not be elaborated here in the embodiments of the present invention.

[0087] S442. Remove blank labels from the cumulative score matrix, use beam search decoding to retrieve the best path in the cumulative score matrix, and determine the speech attribute sequence with time points based on the best path.

[0088] In one feasible implementation, after successful training, the neural network can be used to calculate the probability that each frame in the input audio belongs to each speech attribute class, thereby predicting speech attribute sequences from the audio. To obtain the specific position of the speech attribute sequence in each frame, the optimal alignment path for that speech attribute sequence needs to be retrieved from the probabilities. For this purpose, a cumulative score matrix is ​​first calculated. This matrix can be efficiently calculated using dynamic programming similar to the Viterbi algorithm. Then, blank labels in the matrix are removed, and beam search decoding is used to retrieve the optimal path in the matrix to obtain the phoneme-level alignment result, which is the speech attribute sequence with time points.

[0089] S443. Determine the phoneme sequence with time points based on the speech attribute sequence with time points, and complete the alignment of the transcribed text and audio based on the phoneme sequence with time points.

[0090] In this embodiment of the invention, the time points of the speech attribute sequence are aligned to the phoneme sequence according to the correspondence between the speech attribute sequence and the phoneme sequence, and then aligned to the transcribed text. The transcribed text and audio are aligned according to the time points.

[0091] In this embodiment of the invention, audio and transcribed text are collected to obtain the phoneme sequence corresponding to the transcribed text; the phoneme sequence is converted into a syllable type sequence and a speech attribute sequence; syllable-level alignment is performed based on the audio and the audio syllable type sequence; and phoneme-level alignment is determined based on the syllable-level alignment result, the speech attribute sequence, the speech attribute sequence prediction model, and the optimal alignment path retrieval algorithm, thus completing the audio-to-transcribed text alignment operation. The alignment method of this invention does not require manual alignment to obtain training samples or training the model using training samples; instead, it trains the model while using it, saving training costs and time costs and greatly improving the efficiency of audio-to-text alignment. Furthermore, by utilizing common syllable types and speech attributes as prior knowledge, this method can be extended to alignment tasks of any language, making it an unsupervised automatic alignment method, thus expanding the scope and range of its application.

[0092] Figure 5 This is a block diagram illustrating an unsupervised audio-text automatic alignment apparatus according to an exemplary embodiment, the apparatus being used in an unsupervised audio-text automatic alignment method. (Refer to...) Figure 5 The device includes an acquisition module 510, a conversion module 520, a first alignment module 550, and a second alignment module 540, wherein:

[0093] The acquisition module 510 is used to acquire audio and transcribed text, and to acquire the phoneme sequence corresponding to the transcribed text.

[0094] The conversion module 520 is used to convert the phoneme sequence into a syllable type sequence and a speech attribute sequence;

[0095] The first alignment module 530 is used to perform syllable-level alignment based on the audio and the audio syllable type sequence;

[0096] The second alignment module 540 is used to determine the phoneme-level alignment result based on the syllable-level alignment result, the speech attribute sequence, the speech attribute sequence prediction model, and the optimal alignment path retrieval algorithm, and to complete the alignment operation between the audio and the transcribed text.

[0097] Optionally, the syllable types in the syllable type sequence include four types: V, VC, CV, and CVC, and each syllable type includes one or more phonemes;

[0098] The speech attribute sequence includes nine attributes: low vowel, mid vowel, high vowel, glissando, lisping, nasal, fricative, affricate, and plosive. Each attribute corresponds to one phoneme.

[0099] Optionally, the first alignment module 530 is used for:

[0100] S31. The audio is processed using a speech activity detection algorithm to distinguish between silent segments and non-silent segments. The non-silent segments are converted into loudness envelopes that change over time. Based on the loudness, syllable segments to be adjusted are obtained, and each syllable segment to be adjusted is divided into 3 sub-segments.

[0101] S32. Train the syllable model using the syllable segment to be adjusted and the corresponding sub-segment, and train the silence model using the silence segment to obtain a new alignment result; the syllable model is a hidden Markov model with 12 states, and the silence model is a hidden Markov model with 5 states.

[0102] S33. Compare the new alignment result with the previous alignment result. If the new alignment result is different from the previous alignment result, set the syllable segment in the new alignment result as the syllable segment to be adjusted and proceed to S32. If the new alignment result is the same as the previous alignment result, determine the new alignment result as the syllable-level alignment result.

[0103] Optionally, the hidden Markov model with 12 states includes five parts: start, syllable beginning, syllable core, syllable end, and end. The start and end parts are each set with one state, the syllable beginning is set with four states, and the syllable core and syllable end are each set with three states.

[0104] The hidden Markov model with five states includes five parts: start, syllable beginning, syllable core, syllable end, and syllable ending, with one state set for each part.

[0105] Optionally, the first alignment module 530 is used for:

[0106] The syllable model is fitted using the syllable segment to be adjusted, the silence model is fitted using the silence segment to be adjusted, the first version of the syllable model and silence model is trained using the HTK toolkit, the model parameters are learned using the Welch algorithm, and the new alignment result is solved using the Viterbi algorithm.

[0107] Optionally, the second alignment module 540 is used for:

[0108] S41. Build an initial speech attribute sequence prediction model, wherein the speech attribute sequence prediction model is based on a recurrent neural network (RNN).

[0109] S42. Using connectionist temporal classification (CTC) cost and reconstruction cost as training objectives, train the initialized speech attribute sequence prediction model using syllable-level alignment results and the speech attribute sequence.

[0110] S43. After training is completed, the audio is input into the trained speech attribute sequence prediction model to obtain the probability matrix of each frame belonging to each speech attribute.

[0111] S44. Using the best alignment path retrieval algorithm, select the best alignment path in the probability matrix, determine the phoneme-level alignment result using the best alignment path, and then complete the alignment operation between the audio and the transcribed text.

[0112] Optionally, the main body of the network architecture of the speech attribute sequence prediction model consists of an encoder and a decoder. Both the encoder and the decoder are composed of two bidirectional long short time memory (Bi-LSTM), and each Bi-LSTM layer has 512 units.

[0113] For CTC cost, a dense layer with softmax activation is used after the decoder. The dense layer contains 10 network units, corresponding to 9 speech attribute classes and a blank class, respectively.

[0114] For reconstruction costs, a dense layer with tanh activation is used after the decoder.

[0115] Optionally, the second alignment module 540 is used for:

[0116] S441. Using Viterbi's dynamic programming algorithm, calculate the cumulative score matrix of the probability matrix;

[0117] S442. Remove blank labels from the cumulative score matrix, use beam search decoding to retrieve the best path in the cumulative score matrix, and determine the speech attribute sequence with time points based on the best path.

[0118] S443. Determine the phoneme sequence with time points based on the speech attribute sequence with time points, and complete the alignment of the transcribed text and audio based on the phoneme sequence with time points.

[0119] In this embodiment of the invention, audio and transcribed text are collected to obtain the phoneme sequence corresponding to the transcribed text; the phoneme sequence is converted into a syllable type sequence and a speech attribute sequence; syllable-level alignment is performed based on the audio and the audio syllable type sequence; and phoneme-level alignment is determined based on the syllable-level alignment result, the speech attribute sequence, the speech attribute sequence prediction model, and the optimal alignment path retrieval algorithm, thus completing the audio-to-transcribed text alignment operation. The alignment method of this invention does not require manual alignment to obtain training samples or training the model using training samples; instead, it trains the model while using it, saving training costs and time costs and greatly improving the efficiency of audio-to-text alignment. Furthermore, by utilizing common syllable types and speech attributes as prior knowledge, this method can be extended to alignment tasks of any language, making it an unsupervised automatic alignment method, thus expanding the scope and range of its application.

[0120] Figure 6 This is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present invention. The electronic device 600 may vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 601 and one or more memories 602. The memory 602 stores at least one instruction, which is loaded and executed by the processor 601 to implement the steps of the above-described Chinese transcribing text spelling check method.

[0121] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the aforementioned Chinese transcribing text spelling check method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0122] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0123] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An unsupervised method for automatic audio-text alignment, characterized in that, The method includes: S1. Acquire audio and transcribed text, and obtain the phoneme sequence corresponding to the transcribed text; S2. Convert the phoneme sequence into a syllable type sequence and a speech attribute sequence; S3. Align the audio at the syllable level according to the audio and the syllable type sequence of the audio; S4. Based on the syllable-level alignment results, the speech attribute sequence, the speech attribute sequence prediction model, and the optimal alignment path retrieval algorithm, determine the phoneme-level alignment results, and then complete the alignment operation between the audio and the transcribed text. Specifically, step S3, which aligns syllables at the syllable level based on the audio and the syllable type sequence of the audio, includes: S31. The audio is processed using a speech activity detection algorithm to distinguish between silent segments and non-silent segments. The non-silent segments are converted into loudness envelopes that change over time. The syllable segments to be adjusted are obtained based on the loudness envelopes. Each syllable segment to be adjusted is divided into 3 sub-segments. S32. Train the syllable model using the syllable segment to be adjusted and the corresponding sub-segment, and train the silence model using the silence segment to obtain a new alignment result; the syllable model is a Hidden Markov Model with 12 states, and the silence model is a Hidden Markov Model with 5 states. S33. Compare the new alignment result with the previous alignment result. If the new alignment result is different from the previous alignment result, set the syllable segment in the new alignment result as the syllable segment to be adjusted and proceed to S32. If the new alignment result is the same as the previous alignment result, determine the new alignment result as the syllable-level alignment result. Specifically, step S4 determines the phoneme-level alignment result based on the syllable-level alignment result, the speech attribute sequence, the speech attribute sequence prediction model, and the optimal alignment path retrieval algorithm, thereby completing the alignment operation between the audio and the transcribed text, including: S41. Build an initial speech attribute sequence prediction model, wherein the speech attribute sequence prediction model is based on a recurrent neural network (RNN). S42. Using connectionist temporal classification (CTC) cost and reconstruction cost as training objectives, train the initialized speech attribute sequence prediction model using syllable-level alignment results and the speech attribute sequence. S43. After training is completed, the audio is input into the trained speech attribute sequence prediction model to obtain the probability matrix of each frame belonging to each speech attribute. S44. Using the best alignment path retrieval algorithm, select the best alignment path in the probability matrix, determine the phoneme-level alignment result using the best alignment path, and then complete the alignment operation between the audio and the transcribed text.

2. The method according to claim 1, characterized in that, The syllable types in the syllable type sequence include four types: V, VC, CV, and CVC. Each syllable type includes one or more phonemes. The speech attribute sequence includes nine attributes: low vowel, mid vowel, high vowel, glissando, lisping, nasal, fricative, affricate, and plosive. Each attribute corresponds to one phoneme.

3. The method according to claim 1, characterized in that, The hidden Markov model with 12 states includes five parts: start, syllable beginning, syllable core, syllable end, and end. The start and end parts are each set with 1 state, the syllable beginning is set with 4 states, and the syllable core and syllable end are each set with 3 states. The hidden Markov model with five states includes five parts: start, syllable beginning, syllable core, syllable end, and syllable ending, with one state set for each part.

4. The method according to claim 3, characterized in that, S32 uses the syllable segment to be adjusted and the corresponding sub-segment to train the syllable model, and uses the silence segment to train the silence model, to obtain a new alignment result, including: The syllable model is fitted using the syllable segment to be adjusted, the silence model is fitted using the silence segment to be adjusted, the first version of the syllable model and silence model is trained using the HTK toolkit, the model parameters are learned using the Welch algorithm, and the new alignment result is solved using the Viterbi algorithm.

5. The method according to claim 1, characterized in that, The network architecture of the speech attribute sequence prediction model consists of an encoder and a decoder. Both the encoder and decoder are composed of two bidirectional long short time memory (Bi-LSTM) layers, and each Bi-LSTM layer has 512 units. For CTC cost, a dense layer with softmax activation is used after the decoder. The dense layer contains 10 network units, corresponding to 9 speech attribute classes and a blank class, respectively. For reconstruction costs, a dense layer with tanh activation is used after the decoder.

6. The method according to claim 1, characterized in that, S44 uses an optimal alignment path retrieval algorithm to select the optimal alignment path from the probability matrix, determines the phoneme-level alignment result using the optimal alignment path, and thus completes the alignment operation between the audio and the transcribed text, including: S441. Using Viterbi's dynamic programming algorithm, calculate the cumulative score matrix of the probability matrix; S442. Remove blank labels from the cumulative score matrix, use beam search decoding to retrieve the best path in the cumulative score matrix, and determine the speech attribute sequence with time points based on the best path. S443. Determine the phoneme sequence with time points based on the speech attribute sequence with time points, and complete the alignment of the transcribed text and audio based on the phoneme sequence with time points.

7. An unsupervised automatic audio-text alignment device, characterized in that, The device includes: The acquisition module is used to acquire audio and transcribe text, and to acquire the phoneme sequence corresponding to the transcribed text. The conversion module is used to convert the phoneme sequence into a syllable type sequence and a speech attribute sequence; The first alignment module is used to perform syllable-level alignment based on the audio and the audio syllable type sequence; The second alignment module is used to determine the phoneme-level alignment result based on the syllable-level alignment result, the speech attribute sequence, the speech attribute sequence prediction model, and the optimal alignment path retrieval algorithm, and to complete the alignment operation between the audio and the transcribed text. The first alignment module is used for: S31. The audio is processed using a speech activity detection algorithm to distinguish between silent segments and non-silent segments. The non-silent segments are converted into loudness envelopes that change over time. The syllable segments to be adjusted are obtained based on the loudness envelopes. Each syllable segment to be adjusted is divided into 3 sub-segments. S32. Train the syllable model using the syllable segment to be adjusted and the corresponding sub-segment, and train the silence model using the silence segment to obtain a new alignment result; the syllable model is a Hidden Markov Model with 12 states, and the silence model is a Hidden Markov Model with 5 states. S33. Compare the new alignment result with the previous alignment result. If the new alignment result is different from the previous alignment result, set the syllable segment in the new alignment result as the syllable segment to be adjusted and proceed to S32. If the new alignment result is the same as the previous alignment result, determine the new alignment result as the syllable-level alignment result. The second alignment module is used for: S41. Build an initial speech attribute sequence prediction model, wherein the speech attribute sequence prediction model is based on a recurrent neural network (RNN). S42. Using connectionist temporal classification (CTC) cost and reconstruction cost as training objectives, train the initialized speech attribute sequence prediction model using syllable-level alignment results and the speech attribute sequence. S43. After training is completed, the audio is input into the trained speech attribute sequence prediction model to obtain the probability matrix of each frame belonging to each speech attribute. S44. Using the best alignment path retrieval algorithm, select the best alignment path in the probability matrix, determine the phoneme-level alignment result using the best alignment path, and then complete the alignment operation between the audio and the transcribed text.

8. The apparatus according to claim 7, characterized in that, The syllable types in the syllable type sequence include four types: V, VC, CV, and CVC. Each syllable type includes one or more phonemes. The speech attribute sequence includes nine attributes: low vowel, mid vowel, high vowel, glissando, lisping, nasal, fricative, affricate, and plosive. Each attribute corresponds to one phoneme.

Citation Information

Patent Citations

  • Method and device for aligning phonemes

    CN109377981A

  • System and method for outlier identification to remove poor alignments in speech synthesis

    US20160365085A1