Unsupervised laotian phoneme segmentation method based on multi-feature interaction fusion

Abstract

The present application relates to a Laos language unsupervised phoneme segmentation method based on multi-feature interaction fusion, and belongs to the technical field of natural language processing. The present application comprises the following steps: collecting and processing Laos language audio data sets; extracting self-supervised features, spectral features and pitch features; fusing the self-supervised features, spectral features and pitch features to train a phoneme segmentation model; based on the principle of probability contrast loss, distinguishing adjacent frames and random interference frames to realize optimization of the phoneme segmentation model; in the inference stage, inputting the phoneme segmentation model output vector into a peak detection algorithm to generate the final phoneme boundary. The present application can better interact and fuse the advantages of different features, overcome the limitations of the direct addition method, and thus achieve better results in overall performance.

Description

Technical Field

[0001] This invention relates to an unsupervised phoneme segmentation method for Lao based on multi-feature interactive fusion, belonging to the field of natural language processing technology. Background Technology

[0002] Lao phoneme segmentation is a crucial task in speech processing, aiming to divide continuous Lao speech signals into discrete speech units, such as phonemes or subsyllables. This task plays a vital role in applications such as speech recognition, speech synthesis, and speech translation. To better understand the background technology of Lao phoneme segmentation, the following key aspects are discussed.

[0003] First, speech signal processing is one of the fundamental steps in Lao speech morphology segmentation. Preprocessing techniques, such as denoising and normalization, can improve the quality and consistency of the speech signal. Denoising helps reduce the interference of background noise on the segmentation results, while normalization can make different speech segments have similar audio features, thereby improving the accuracy of subsequent segmentation algorithms. Second, feature extraction is a crucial step in Lao speech morphology segmentation. Commonly used feature extraction methods include Mel-frequency cepstral coefficients (MFCC) and linear predictive coding (LPC). These features can capture the spectral and acoustic properties of the speech signal, providing important information about the audio content. By extracting and representing these features, more reliable input can be provided for subsequent segmentation algorithms.

[0004] Machine learning methods have been widely applied in Lao phoneme segmentation tasks. Supervised learning methods utilize labeled training data to train models, with common models including Hidden Markov Models (HMMs) and Support Vector Machines (SVMs). These models learn the relationship between speech segments and their corresponding speech units, thus accurately delineating the boundaries of speech units. Furthermore, unsupervised learning methods attempt to automatically learn segmentation models and patterns from unlabeled speech data, providing a more flexible approach to phoneme segmentation.

[0005] In Lao speech, tone is a crucial feature because tone variations can alter word meaning. Therefore, tone modeling and analysis become essential in Lao phoneme segmentation tasks. By modeling tones, the boundaries of speech units can be delineated more accurately, improving the accuracy and reliability of the segmentation results. Summary of the Invention

[0006] This invention provides an unsupervised phoneme segmentation method for Lao based on multi-feature interactive fusion, in order to improve the accuracy and reliability of the segmentation results.

[0007] The technical solution of this invention is: an unsupervised phoneme segmentation method for Lao based on multi-feature interactive fusion, the specific steps of which are as follows:

[0008] Step 1: Collect and process Lao speech datasets;

[0009] Step 2: Extract self-supervised features, spectral features, and pitch features to capture tonal variations and phoneme boundary information in Lao.

[0010] Step 3: Fuse the self-supervised features, spectral features, and pitch features to train the phoneme segmentation model;

[0011] Step 4: Based on the principle of probability contrast loss, distinguish between adjacent frames and random interference frames to optimize the phoneme segmentation model; during the inference stage, input the output vector of the phoneme segmentation model into the peak detection algorithm to generate the final phoneme boundary.

[0012] In Lao phoneme segmentation, the input data for the segmentation device consists of extracted self-supervised features, spectral features, and pitch features. The program then automatically performs progressive interactive attention fusion on these three features to obtain the final input data. By using contrastive loss, the final phoneme boundaries are obtained, thus achieving phoneme segmentation.

[0013] Furthermore, Step 1 specifically includes:

[0014] Step 1.1: We obtained public datasets from Huggingface; in addition, we obtained Lao speech data from the YouTube platform using web crawling technology; this speech data came from various sources, including Lao news and conversations.

[0015] Step 1.2: The audio was segmented using speaker logs and Voice Activity Detection (VAD) technology.

[0016] Step 1.3: Lao text data was obtained from Lao news websites using web crawling technology, with the number of text characters limited to between 50 and 150; then, Microsoft's speech synthesis technology was used to convert the text into corresponding speech samples.

[0017] Furthermore, Step 2 specifically includes:

[0018] Step 2.1: For self-supervised feature extraction, we used the open-source xlsr-300m pre-trained model to extract 1024-dimensional speech features for experiments.

[0019] Step 2.2: For spectral features, the torchaudio Kaldi toolkit is used to extract Fbank features, with the frame shift dimension set to 10ms and the frame window size dimension set to 25ms.

[0020] Step 2.3: For pitch features, use the SWIPE algorithm in the pySPTK tool to extract them, with a search frequency range of 70Hz to 350Hz.

[0021] Step 2.4: The extracted self-supervised features, spectral features, and pitch features are downsampled using a CNN. By setting different numbers of layers, the sequence length of the input model is controlled, so that the sequence lengths of different features tend to be consistent.

[0022] For dimensionality, three features are uniformly mapped to 256 dimensions. For frame count, since the frame shift of the self-supervised features is twice as long as that used for the spectral features and about four times as long as that used for the pitch features, a CNN is used to downsample the spectral and pitch features to obtain the same number of frames. For the spectral features, the convolutional layers are configured with a kernel size of 4 and a stride of 2. For the pitch features, the convolutional layers are configured with a kernel size of 5 and a stride of 4, and both features are normalized.

[0023] Furthermore, Step 3 specifically includes:

[0024] Step 3.1: Using the Wav2vec2 feature sequence w, a self-supervised feature encoding layer is obtained. The encoding process is H w2v2 =Wav2vec2Encoder(w);

[0025] Step 3.2: The spectral feature, i.e., the Fbank feature sequence f, is downsampled by D(·) to obtain the feature sequence D(f). This, along with the downsampled pitch feature sequence D(p), serves as the input to the acoustic fundamental frequency feature fusion coding layer. Encoding is performed using multi-head attention fusion, and the encoded output is the acoustic fundamental frequency feature. The encoding process is H spect =MultiHead(D(f(,D(p0,D(p)));

[0026] Step 3.3: The cross-attention mechanism fusion layer integrates the self-supervised features H w2v2 Acoustic fundamental frequency characteristics H spect The representation is obtained after interactive fusion, using the input as input. The cross-attention mechanism fusion layer can automatically learn the alignment relationship between two representations without increasing the representation length and feature dimension, achieving mutual complementarity and enhancement effects; the feature interaction fusion process is as follows:

[0027] The output H of the Wav2vec2 feature coding layer w2v2 The output H of the acoustic fundamental frequency feature coding layerspect Feature interaction fusion is achieved through a cross-attention mechanism; Q, K, and V are modeled using a cross-attention mechanism; the attention fusion is: H = Attention(Q w2v2 ,K spect V spect Using the classic attention representation, we obtain vector Q respectively. w2v2 K spect V spect The formula is Q w2v2 =H w2v2 W i Q ,K spect =H spect W i K V spect =H spect W i V ;

[0028] Among them, W i Q W i K W i V The parameter matrix is ​​initialized randomly.

[0029] Furthermore, Step 4 specifically includes:

[0030] The encoded audio representation sequence of the original Lao audio X is Z = (z1, z2, ..., z...). L ),in, 1≤i≤L, where L is the length of the audio sequence Z; the phoneme segmentation model learns the encoding function f from the audio sequence to the audio representation. FUSE :X→Z, through the function f FUSE Optimization is performed to distinguish between adjacent frames in sequence Z and randomly sampled interference frames in Z, using D(z) i D(z) represents a set of non-adjacent frames. i )={z j :|ij|>1,z j ∈Z};

[0031] From D(z) i K frames are randomly selected from ) and Given a training set consisting of m samples Minimize the following objective function: in, The expression in the form sim(u,v) represents the cosine similarity between two vectors u and v;

[0032] During the reasoning process, the Lao language audio segment X is input into the encoder to obtain Z = f FUSE (X), where the boundary score for time i is set as the dissimilarity value between the i-th frame and the (i+1)-th frame, where i = 1, 2, ..., L-1, and the boundary score is score(z). i )=-sim(z i ,z i+1 score(z) i ) is the model for the next frame z i+1 Does it belong to the current frame z? i The confidence scores of different phoneme segments are used; therefore, time points with high dissimilarity values ​​are considered as candidate boundaries for phoneme segmentation changes; peak detection algorithms are used to evaluate the dissimilarity scores (z). i The process is performed to obtain the final segmentation result. When the score exceeds the set peak prominence value δ, these frames are predicted as phoneme boundaries. The optimal δ value is determined by tuning through a cross-validation process.

[0033] The beneficial effects of this invention are:

[0034] This invention employs a multi-feature interactive fusion strategy, including independent encoding of self-supervised (Wav2vec2) features, spectral (Fbank) features, and pitch features to avoid the limitations of single features. An attention mechanism is introduced to gradually fuse these independent features, thereby more comprehensively capturing information about tone variations and phoneme boundaries in Lao. Finally, a learnable framework is used to optimize the phoneme segmentation model. Experimental results show that this phoneme segmentation method is the optimal solution obtained during the experiment, achieving an R-value of 88.50%. Attached Figure Description

[0035] Figure 1 This is a flowchart from the present invention. Detailed Implementation

[0036] Example 1: As Figure 1 As shown, the Lao language unsupervised phoneme segmentation method based on multi-feature interaction fusion has the following specific steps:

[0037] Step 1: Collect and process Lao speech datasets; the laboratory's data sources are diverse, including public datasets, web crawling, and speech synthesis.

[0038] Step 1 specifically includes:

[0039] Step 1.1: We obtained public datasets from Huggingface (an open library of natural language processing models and datasets); in addition, we obtained Lao speech data from the YouTube platform using web crawling techniques; this speech data came from various sources, including Lao news, conversations and other speech content.

[0040] Step 1.2: In order to accurately segment the audio, speaker log and voice activity detection (VAD) technology were used to segment the audio; it was divided into smaller audio segments for further analysis and processing.

[0041] Step 1.3: To obtain more Lao speech data, web crawling technology was used to retrieve Lao text data from Lao news websites. To control the text length, the number of characters was limited to between 50 and 150. Then, Microsoft's speech synthesis technology was used to convert the text into corresponding speech samples. To ensure the accuracy and quality of the speech synthesis results, the generated audio samples were evaluated and verified by Lao natives.

[0042] This data acquisition and processing workflow provides diverse and rich Lao speech datasets for the experiments. These datasets will serve as an important foundation for Lao speechme segmentation research and model optimization. By comprehensively utilizing this multi-source data, the research can better reflect the real-world characteristics and features of Lao speech, improving the performance and adaptability of the models.

[0043] Ultimately, 15,631 Lao news audio clips were selected as the training corpus, and 2,050 audio clips manually annotated by native Lao speakers were used as the test corpus.

[0044] Step 2: Extract self-supervised features, spectral features, and pitch features;

[0045] Step 2.1: For self-supervised feature extraction, we used the open-source xlsr-300m pre-trained model to extract 1024-dimensional speech features for experiments.

[0046] Step 2.2: For spectral features, the torchaudio Kaldi toolkit is used to extract Fbank features, with the frame shift dimension set to 10ms and the frame window size dimension set to 25ms.

[0047] Step 2.3: For pitch features, the SWIPE algorithm in the pySPTK tool was used for extraction, with a search frequency range of 70Hz to 350Hz. To achieve efficient computation of the input data, a CNN was used to downsample the input sequence in the experiment. By appropriately setting different layers, the sequence length of the input model was controlled, thereby making the sequence lengths of different features tend to be consistent.

[0048] Step 2.4: The extracted self-supervised features, spectral features, and pitch features are downsampled using a CNN. By setting different numbers of layers, the sequence length of the input model is controlled, so that the sequence lengths of different features tend to be consistent.

[0049] For dimensionality, three features are uniformly mapped to 256 dimensions. For frame count, since the frame shift of the self-supervised features is twice as long as that used for the spectral features and about four times as long as that used for the pitch features, a CNN is used to downsample the spectral and pitch features to obtain the same number of frames. For the spectral features, the convolutional layers are configured with a kernel size of 4 and a stride of 2. For the pitch features, the convolutional layers are configured with a kernel size of 5 and a stride of 4, and both features are normalized.

[0050] Step 3: Fuse the self-supervised features, spectral features, and pitch features to train the phoneme segmentation model;

[0051] The model employs a progressive interactive fusion of multiple language features at the encoding layer. The encoding layer consists of a Wav2vec2 encoding layer, an acoustic fundamental frequency feature fusion encoding layer, and a cross-attention mechanism fusion layer. The encoder input includes three parts: Wav2vec2 features, Fbank features, and Pitch features.

[0052] Step 3 specifically includes:

[0053] Step 3.1: Using the Wav2vec2 feature sequence w, a self-supervised feature encoding layer is obtained. The encoding process is H w2v2 =Wav2vec2Encoder(w);

[0054] Step 3.2: The spectral feature, i.e., the Fbank feature sequence f, is downsampled by D(·) to obtain the feature sequence D(f). This, along with the downsampled pitch feature sequence D(p), serves as the input to the acoustic fundamental frequency feature fusion coding layer. Encoding is performed using multi-head attention fusion, and the encoded output is the acoustic fundamental frequency feature. The encoding process is H spect =MultiHead(D(f(,D(p0,D(p)));

[0055] Step 3.3: The cross-attention mechanism fusion layer integrates the self-supervised features H w2v2 Acoustic fundamental frequency characteristics H spect The representation is obtained after interactive fusion, using the input as input. The cross-attention mechanism fusion layer can automatically learn the alignment relationship between two representations without increasing the representation length and feature dimension, achieving mutual complementarity and enhancement effects; the feature interaction fusion process is as follows:

[0056] The output H of the Wav2vec2 feature coding layer w2v2 The output H of the acoustic fundamental frequency feature coding layer spect Feature interaction fusion is achieved through a cross-attention mechanism; Q, K, and V are modeled using a cross-attention mechanism; the attention fusion is: H = Attention(Q w2v2 ,K spect V spect Using the classic attention representation, we obtain vector Q respectively. w2v2 K spect V spect The formula is Q w2v2 =H w2v2 W i Q ,K spect =H spect W i K V spect =H spect W i V ;

[0057] Among them, W i Q W i K W i V The parameter matrix is ​​initialized randomly.

[0058] Step 4: Based on the principle of probability contrast loss, distinguish between adjacent frames and random interference frames to optimize the phoneme segmentation model; during the inference stage, input the output vector of the phoneme segmentation model into the peak detection algorithm to generate the final phoneme boundary.

[0059] Furthermore, Step 4 specifically includes:

[0060] The encoded audio representation sequence of the original Lao audio X is Z = (z1, z2, ..., z...). L ),in, 1≤i≤L, where L is the length of the audio sequence Z; the phoneme segmentation model learns the encoding function f from the audio sequence to the audio representation. FUSE :X→Z, through the function f FUSE Optimization is performed to distinguish between adjacent frames in sequence Z and randomly sampled interference frames in Z, using D(z) i D(z) represents a set of non-adjacent frames. i )={z j :|ij|>1,z j ∈Z};

[0061] From D(z) i K frames are randomly selected from ) and Given a training set consisting of m samples Minimize the following objective function: in, The expression in the form sim(u,v) represents the cosine similarity between two vectors u and v;

[0062] During the reasoning process, the Lao language audio segment X is input into the encoder to obtain Z = f FUSE (X), where the boundary score for time i is set as the dissimilarity value between the i-th frame and the (i+1)-th frame, where i = 1, 2, ..., L-1, and the boundary score is score(z). i )=-sim(z i ,z i+1 score(z) i ) is the model for the next frame z i+1 Does it belong to the current frame z? i The confidence scores of different phoneme segments are used; therefore, time points with high dissimilarity values ​​are considered as candidate boundaries for phoneme segmentation changes; peak detection algorithms are used to evaluate the dissimilarity scores (z). i The process is performed to obtain the final segmentation result. When the score exceeds the set peak prominence value δ, these frames are predicted as phoneme boundaries. The optimal δ value is determined by tuning through a cross-validation process.

[0063] To illustrate the effectiveness of this invention, a comparative experiment was conducted. Four fusion methods were used to compare the phoneme segmentation results for three different features, verifying the most effective fusion method. Table 1 shows the expected segmentation results for the Lao language test.

[0064] Table 1: Expected Segmentation Results of Lao Language Test

[0065] Fusion method accuracy Recall rate F1 value R-value Direct addition 84.15 87.05 83.57 86.50 splicing-L 50.19 56.36 53.09 57.56 splicing-D 85.95 78.94 82.29 84.14 ours 86.53 86.52 86.53 88.50

[0066] The data above shows that among these methods, the direct addition method exhibits the highest recall rate, reaching 87.05%. This is because the direct addition method simply adds different features directly during feature fusion, preserving the original information of each feature, thus enabling a more comprehensive capture of useful information in the speech signal. However, while the direct addition method performs well in terms of recall, its performance on other metrics is relatively low. This is because simple feature addition leads to redundancy or mutual interference between features. The concatenation-L method performs the worst because concatenation in length causes information extraction from different features to be disordered, making it impossible to correctly obtain different feature information from the same audio segment. The concatenation-D method shows a significant improvement, but its performance is not optimal because simply superimposing feature dimensions cannot capture the most effective information of the speech signal, and redundant information between features causes interference. In contrast, the cross-attention fusion method proposed in this invention combines the correlation between different features and adaptively adjusts feature weights by introducing an attention mechanism to improve the representation ability of key information. This method can better integrate the advantages of different features and overcome the limitations of the direct addition method, thus achieving better overall performance results, namely an accuracy of 86.53%, a recall of 86.52%, an F1 score of 86.53%, and an R-value of 88.50%.

[0067] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A Lao language unsupervised phoneme segmentation method based on multi-feature interaction fusion, characterized by: The specific steps of the method are as follows: Step 1: Collect and process Lao speech datasets; Step 2: Extract self-supervised features, spectral features, and pitch features; Step 3: Fuse the self-supervised features, spectral features, and pitch features to train the phoneme segmentation model; Step 4: Based on the principle of probability contrast loss, distinguish between adjacent frames and random interference frames to optimize the phoneme segmentation model; During the inference phase, the output vector of the phoneme segmentation model is input into the peak detection algorithm to generate the final phoneme boundaries; Step 3 specifically includes: Step 3.1, Wav2vec2 feature sequence Using the Wav2vec2 feature encoding layer, self-supervised features are obtained. The encoding process is as follows ; Step 3.2, Spectral characteristics, i.e., Fbank feature sequences After sampling The feature sequence was then obtained. , and the downsampled Pitch feature sequence Pitch features, or acoustic frequencies, are used as input to the acoustic fundamental frequency feature fusion coding layer. This layer is encoded using a multi-head attention fusion method, and the resulting output is the acoustic fundamental frequency feature. The encoding process is as follows ; Step 3.3: The cross-attention mechanism fusion layer integrates self-supervised features. Acoustic fundamental frequency characteristics The representation is obtained after interactive fusion, using the input as input. The cross-attention mechanism fusion layer can automatically learn the alignment relationship between two representations without increasing the representation length and feature dimension, achieving mutual complementarity and enhancement effects; the feature interaction fusion process is as follows: ; The output of the Wav2vec2 feature encoding layer Output of the acoustic fundamental frequency feature coding layer Feature interaction fusion is performed through a cross-attention mechanism; Q, K, and V are modeled using a cross-attention mechanism; attention fusion is as follows: Using the classic attention representation, we obtain vector Q respectively. w2v2 K spect V spect The formula is ; in, , , The parameter matrix is ​​initialized randomly. Step 4 specifically includes: The encoded audio representation sequence of the original Lao audio X is as follows: ,in, , L is the length of the audio sequence Z; the phoneme segmentation model learns the encoding function from the audio sequence to the audio representation. By analyzing the function Optimization is performed to distinguish between adjacent frames in sequence Z and randomly sampled interference frames in Z, using... Indicates a set of non-adjacent frames: ; from Randomly select K frames from the middle, and Given a training set consisting of m samples Minimize the following objective function: ,in, ,for The formal expression represents the cosine similarity between two vectors u and v; During the reasoning process, the Lao language speech segment X is input into the encoder to obtain... The boundary score for time i is set as the dissimilarity value between frame i and frame i+1, where, The boundary fraction is , For the model, the next frame z i+1 Does it belong to the current frame z? i The confidence scores of different phoneme segments are used; therefore, time points with high dissimilarity values ​​are considered candidate boundaries for phoneme segmentation changes; peak detection algorithms are used to evaluate dissimilarity values. Processing is performed to obtain the final segmentation result; when the score exceeds the set peak value... At that time, these frames are predicted as phoneme boundaries, which is optimal. The value was determined through a cross-validation process.

2. The Lao language unsupervised phoneme segmentation method based on multi-feature interaction fusion according to claim 1, characterized in that: Step 1 specifically includes: Step 1.1: We obtained public datasets from Huggingface; in addition, we obtained Lao speech data from the YouTube platform using web crawling technology; this speech data came from various sources, including Lao news and conversations. Step 1.2: The audio was segmented using speaker logs and Voice Activity Detection (VAD) technology. Step 1.3: Lao text data was obtained from Lao news websites using web crawling technology, with the number of text characters limited to between 50 and 150; then, Microsoft's speech synthesis technology was used to convert the text into corresponding speech samples.

3. The Lao language unsupervised phoneme segmentation method based on multi-feature interaction fusion according to claim 1, characterized in that: Step 2 specifically includes: Step 2.1: For self-supervised feature extraction, we used the open-source xlsr-300m pre-trained model to extract 1024-dimensional speech features for experiments. Step 2.2: For spectral features, the torchaudio Kaldi toolkit is used to extract Fbank features, with the frame shift dimension set to 10ms and the frame window size dimension set to 25ms. Step 2.3: For pitch features, use the SWIPE algorithm in the pySPTK tool to extract them, with a search frequency range of 70Hz to 350Hz. Step 2.4: The extracted self-supervised features, spectral features, and pitch features are downsampled using a CNN. By setting different numbers of layers, the sequence length of the input model is controlled, so that the sequence lengths of different features tend to be consistent. For dimensionality, three features are uniformly mapped to 256 dimensions. For frame count, since the frame shift of the self-supervised features is twice as long as that used for the spectral features and about four times as long as that used for the pitch features, a CNN is used to downsample the spectral and pitch features to obtain the same number of frames. For the spectral features, the convolutional layers are configured with a kernel size of 4 and a stride of 2. For the pitch features, the convolutional layers are configured with a kernel size of 5 and a stride of 4, and both features are normalized.

Images