Lao acoustic representation method based on Lao pronunciation similarity

Abstract

The invention relates to a Lao acoustic representation method based on Lao pronunciation similarity, and belongs to the technical field of natural language processing. Comprising the following steps: uniformly preprocessing Thai and Lao audio and text data, and constructing a Thai and Lao word list; a multi-language self-supervised pre-training model is adopted, firstly, a Thai training set and a Thai word list are used for fine tuning of high-level parameters, freezing of a low-level feature extractor and optimization of CTC loss, and a Thai fine-tuned model is obtained; loading a model after Thai fine tuning, removing an original Thai classification layer, creating a new classification layer based on a Lao vocabulary and performing random initialization, further performing fine tuning on a Transform encoder and the new classification layer by using a Lao training set, and optimizing CTC loss to obtain an acoustic representation model suitable for Lao; loading an acoustic representation model, reasoning on a Lao test set, calculating CER, and evaluating acoustic representation and recognition performance. According to the method, the speech recognition performance of low-resource language Lao is remarkably improved.

Description

Technical Field

[0001] The invention relates to a Laotian acoustic representation method based on Thai and Laotian pronunciation similarity, belonging to the technical field of natural language processing. Background Art

[0002] In recent years, end-to-end speech recognition systems have received widespread attention and application. Among them, multilingual acoustic models based on large-scale self-supervised learning (such as Wav2vec2.0, Wav2vec-XLSR, etc.) have become an important means to improve multilingual ASR performance.

[0003] However, for low-resource languages ​​such as Lao, due to the limited available annotated speech data, existing multilingual models find it difficult to fully activate the cross-language shared knowledge contained in the model during the fine-tuning stage, thereby limiting the model's fine modeling of the acoustic features of low-resource languages, resulting in low recognition accuracy.

[0004] Some studies have attempted to improve ASR performance for low-resource languages ​​through direct fine-tuning, joint multilingual fine-tuning, or cross-lingual continuous pre-training. However, these approaches fail to fully consider the dynamic balance between pronunciation similarity between languages ​​and knowledge transfer. In practical applications, direct fine-tuning is prone to overfitting due to data scarcity, multilingual fine-tuning is not targeted to the target language, and continuous pre-training is limited by data size and computing resources.

[0005] Therefore, there is an urgent need for an acoustic representation method that can effectively utilize the acoustic similarities between high-resource languages ​​and target low-resource languages ​​and transfer knowledge in a phased and progressive manner to improve the performance of speech recognition in low-resource languages. Summary of the Invention

[0006] The technical problem solved by the present invention is: the present invention provides a Lao acoustic representation method based on the pronunciation similarity between Thai and Lao. The present invention can fully explore and utilize the intrinsic connection between Thai and Lao in acoustic features, optimize the multilingual self-supervised pre-training model through a two-stage fine-tuning strategy, and significantly improve the speech recognition performance of Lao, a low-resource language.

[0007] The technical solution of the present invention is: a Laotian acoustic characterization method based on Thai-Lao pronunciation similarity, the method comprising:

[0008] Step 1: Preprocess Thai and Lao audio and text data uniformly to construct Thai and Lao vocabulary tables;

[0009] Step 2: Use a multilingual self-supervised pre-trained model. First, fine-tune the high-level parameters using the Thai training set and Thai vocabulary, freeze the low-level feature extractor, and optimize the CTC loss to obtain the Thai fine-tuned model.

[0010] Step 3: Load the fine-tuned Thai model, remove the original Thai classification layer, create a new classification layer based on the Lao vocabulary and initialize it randomly, further fine-tune the Transformer encoder and the new classification layer using the Lao training set, optimize the CTC loss, and obtain an acoustic representation model suitable for Lao.

[0011] Step 4: Load the acoustic representation model, perform inference on the Lao test set, calculate the CER, and evaluate the acoustic representation and recognition performance.

[0012] Furthermore, the Step 1 includes:

[0013] Thai and Lao audio and text data were collected and cleaned. All audio data was resampled to 16kHz and the duration was controlled within 20 seconds. All punctuation marks were removed from all text tags. Thai and Lao vocabulary lists were established separately, and special symbols were added to ensure that the vocabulary lists covered all target characters. The Thai and Lao vocabulary lists included all unique characters and special symbols in the target language; special symbols included [UNK], [PAD], and “|”.

[0014] Furthermore, the Step 1 specifically includes:

[0015] Step 1.1, audio data sampling standardization:

[0016] All original Thai and Lao audio data were uniformly resampled to 16kHz to eliminate feature deviations caused by inconsistent sampling rates;

[0017] Remove samples with audio duration exceeding 20 seconds;

[0018] Step 1.2: Text label standardization and Thai and Lao vocabulary construction:

[0019] Perform uniform punctuation removal on the text transcriptions of all audio to reduce the interference of irrelevant characters on acoustic modeling;

[0020] Extract all unique characters that appear in the training corpus and, combined with special symbols, construct separate Thai and Lao vocabulary tables for Thai and Lao respectively to define the model output space;

[0021] Step 1.3, Data consistency check:

[0022] Check that the audio data corresponds to the text labels one by one and exclude missing or abnormal samples.

[0023] Furthermore, in Step 2, the multilingual self-supervised pre-training model adopts Wav2vec-XLSR-300m.

[0024] Furthermore, in the Step 2, during the fine-tuning stage, the low-level feature extractor is frozen, and only the Transformer encoder and output classification layer are trained.

[0025] Furthermore, in Step 2 and Step 3, the AdamW optimizer, mixed precision training, gradient accumulation and dynamic model checkpoint saving technology are used during the training process.

[0026] Furthermore, the Step 2 includes:

[0027] Load the pre-trained weights of Wav2vec-XLSR-300m, freeze the CNN feature extraction layer, initialize the linear classification layer according to the Thai vocabulary, fine-tune the model on the Thai training set, and optimize the CTC loss training. During this process, batch normalization, mixed precision, and dynamic model saving are performed, and the best model is retained to obtain the fine-tuned model for Thai.

[0028] Furthermore, Step 2-Step 3 specifically include:

[0029] (1) Model initialization;

[0030] We selected the publicly available multilingual self-supervised pre-trained model Wav2vec-XLSR-300m and loaded the pre-trained weights. We also froze the low-level CNN feature extraction layers and only trained the high-level Transformer encoder and output classification layers.

[0031] (2) The first stage: fine-tuning of Thai;

[0032] Using the Thai training set and Thai vocabulary, we randomly initialized the linear classification layer and used the CTC loss function for model training. During the training process, we used batch normalization, mixed precision, and learning rate hot start techniques to improve training efficiency and stability.

[0033] Fine-tune for several rounds, save model checkpoints every few steps, and finally save the best performance model;

[0034] (3) The second stage: Lao language fine-tuning;

[0035] Load the model weights after the first stage of Thai fine-tuning, remove the original Thai classification layer, and create a new and randomly initialized Lao classification layer based on the Lao vocabulary;

[0036] Using the Laotian training set, freeze the feature extraction layer and only fine-tune the Transformer encoder and the new classification layer, continuing to optimize the CTC loss.

[0037] The training strategy remains consistent with the first stage, using step evaluation and dynamic checkpoint saving mechanisms.

[0038] The present invention also provides a Laotian acoustic characterization system based on Thai-Lao pronunciation similarity, the system comprising: a module for executing the Laotian acoustic characterization method based on Thai-Lao pronunciation similarity.

[0039] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the Lao acoustic characterization method based on Thai-Lao pronunciation similarity is implemented.

[0040] The beneficial effects of the present invention are:

[0041] 1. This paper utilizes data normalization and two-stage transfer fine-tuning to fully utilize the pronunciation similarity between Thai and Lao, implements progressive knowledge transfer for the multilingual self-supervised pre-training model, and significantly improves the speech recognition performance of Lao, a low-resource language.

[0042] 2. The present invention is widely used in the fields of multilingual speech recognition and low-resource language acoustic representation, and has excellent performance improvement and application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a model structure diagram of the Laotian acoustic representation method based on Thai-Lao pronunciation similarity proposed in the present invention. DETAILED DESCRIPTION

[0044] Example 1: Figure 1 As shown, a Laotian acoustic characterization method based on Thai-Lao pronunciation similarity includes:

[0045] Step 1: Preprocess the Thai and Lao audio and text data uniformly to construct Thai and Lao vocabulary tables. Step 1 specifically includes:

[0046] Step 1.1, audio data sampling standardization:

[0047] All original Thai and Lao audio data were uniformly resampled to 16kHz to eliminate feature deviations caused by inconsistent sampling rates;

[0048] Eliminate samples with audio duration exceeding 20 seconds to ensure a reasonable distribution of sample lengths, which helps the model focus on core speech segments.

[0049] Step 1.2: Text label standardization and Thai and Lao vocabulary construction:

[0050] Punctuation marks (commas, periods, question marks, quotation marks, etc.) are uniformly removed from the text transcriptions of all audio to reduce the interference of irrelevant characters on acoustic modeling.

[0051] Extract all unique characters that appear in the training corpus and combine them with special symbols such as [UNK] (unknown character), [PAD] (filler character), and “|” (word boundary character) to construct Thai and Lao vocabulary tables for Thai and Lao respectively, which are used to define the model output space;

[0052] Step 1.3, Data consistency check:

[0053] Check that the audio data corresponds to the text labels one by one, exclude missing or abnormal samples, and ensure high data quality.

[0054] Step 2: Use a multilingual self-supervised pre-trained model. First, fine-tune the high-level parameters using the Thai training set and Thai vocabulary, freeze the low-level feature extractor, and optimize the CTC loss to obtain the Thai fine-tuned model.

[0055] Furthermore, the Step 2 includes:

[0056] Load the pre-trained weights of Wav2vec-XLSR-300m, freeze the CNN feature extraction layer, initialize the linear classification layer according to the Thai vocabulary, fine-tune the model on the Thai training set, and optimize the CTC loss training. During this process, batch normalization, mixed precision, and dynamic model saving are performed, and the best model is retained to obtain the fine-tuned model for Thai.

[0057] Step 3: Load the fine-tuned Thai model, remove the original Thai classification layer, create a new classification layer based on the Lao vocabulary and initialize it randomly, further fine-tune the Transformer encoder and the new classification layer using the Lao training set, optimize the CTC loss, and obtain an acoustic representation model suitable for Lao.

[0058] In Step 2 and Step 3, the AdamW optimizer, mixed precision training, gradient accumulation, and dynamic model checkpoint saving techniques are used during the training process.

[0059] Furthermore, Step 2-Step 3 specifically include:

[0060] (1) Model initialization;

[0061] We selected the publicly available multilingual self-supervised pre-trained model Wav2vec-XLSR-300m and loaded the pre-trained weights. We also froze the low-level CNN feature extraction layers and only trained the high-level Transformer encoder and output classification layers to prevent low-level feature drift.

[0062] (2) The first stage: fine-tuning of Thai;

[0063] Using the Thai training set and Thai vocabulary, we randomly initialized the linear classification layer and used the CTC (Connectionist Temporal Classification) loss function for model training. During training, we used batch normalization, mixed precision (FP16), and learning rate warmup techniques to improve training efficiency and stability.

[0064] Fine-tune for 15 rounds, save model checkpoints every 400 steps, and finally save the best performing model;

[0065] (3) The second stage: Lao language fine-tuning;

[0066] Load the model weights after the first stage of Thai fine-tuning, remove the original Thai classification layer, and create a new and randomly initialized Lao classification layer based on the Lao vocabulary;

[0067] Using the Laotian training set, freeze the feature extraction layer and only fine-tune the Transformer encoder and the new classification layer, continuing to optimize the CTC loss.

[0068] The training strategy remains consistent with the first stage, using step evaluation and dynamic checkpoint saving mechanisms.

[0069] Step 4: Load the acoustic representation model, perform inference on the Lao test set, calculate the CER, and evaluate the acoustic representation and recognition performance.

[0070] During the inference evaluation, the Lao test set was used for inference, and the character error rate (CER) was used as the performance evaluation indicator to comprehensively evaluate the acoustic representation ability of the model.

[0071] Among them, the overall two-stage fine-tuning process can be shown in Table 1.

[0072] (1) Model training and optimization details:

[0073] The entire process uses the AdamW optimizer with parameter settings of β1 = 0.9, β2 = 0.999, an initial learning rate of 3e-4, a batch size of 16, and two-step gradient accumulation to reduce GPU memory pressure. Gradient_checkpointing is also enabled to further reduce memory consumption and adapt to single-GPU training. The training, validation, and test sets are all from the Fleurs multilingual public dataset to ensure that the experimental results are reproducible and generalizable.

[0074] (2) Result analysis and advantages description:

[0075] Experiments show that compared with traditional schemes such as direct fine-tuning, multilingual joint fine-tuning, cross-language continuous pre-training, and extended fine-tuning, the method of the present invention reduces the CER on the Lao test set by 11.7%, a significant improvement.

[0076] The progressive transfer learning strategy can better utilize the phonetic similarities between Thai and Lao languages, effectively activate cross-lingual representation knowledge in the model parameter space, and alleviate the bottleneck of low-resource data.

[0077] The method is flexible and applicable to other low-resource language transfer modeling scenarios with high phonetic similarity, and has broad application prospects.

[0078] Table 1 shows the overall two-stage fine-tuning process algorithm

[0079]

[0080]

[0081] The present invention also provides a Laotian acoustic characterization system based on Thai-Lao pronunciation similarity, the system comprising:

[0082] Thai and Lao vocabulary building modules, used to uniformly preprocess Thai and Lao audio and text data to build Thai and Lao vocabulary;

[0083] The Thai fine-tuning module uses a multilingual self-supervised pre-trained model to fine-tune high-level parameters using the Thai training set and Thai vocabulary, freeze the low-level feature extractor, and optimize the CTC loss to obtain a Thai fine-tuned model.

[0084] The Lao fine-tuning module is used to load the Thai fine-tuned model, remove the original Thai classification layer, create a new classification layer based on the Lao vocabulary and randomly initialize it, further fine-tune the Transformer encoder and the new classification layer using the Lao training set, optimize the CTC loss, and obtain an acoustic representation model suitable for Lao.

[0085] The inference evaluation module is used to load the acoustic representation model, perform inference on the Lao test set, calculate the CER, and evaluate the acoustic representation and recognition performance.

[0086] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the Lao acoustic characterization method based on Thai-Lao pronunciation similarity is implemented.

[0087] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A Laotian acoustic characterization method based on the similarity of Thai and Lao pronunciations, characterized by: The method comprises: Step 1: Preprocess Thai and Lao audio and text data uniformly to construct Thai and Lao vocabulary tables; Step 2: Use a multilingual self-supervised pre-trained model. First, fine-tune the high-level parameters using the Thai training set and Thai vocabulary, freeze the low-level feature extractor, and optimize the CTC loss to obtain the Thai fine-tuned model. Step 3: Load the fine-tuned Thai model, remove the original Thai classification layer, create a new classification layer based on the Lao vocabulary and initialize it randomly, further fine-tune the Transformer encoder and the new classification layer using the Lao training set, optimize the CTC loss, and obtain an acoustic representation model suitable for Lao. Step 4: Load the acoustic representation model, perform inference on the Lao test set, calculate the CER, and evaluate the acoustic representation and recognition performance.

2. The Laotian acoustic characterization method based on Thai-Lao pronunciation similarity according to claim 1, characterized in that: Step 1 includes: Thai and Lao audio and text data were collected and cleaned. All audio data was resampled to 16kHz and the duration was controlled within 20 seconds. All text tags removed all punctuation marks. Thai and Lao vocabulary lists were established respectively, and special symbols were added to ensure that the vocabulary lists covered all target characters. The Thai and Lao vocabulary lists included all unique characters and special symbols of the target language; special symbols included [UNK], [PAD], and "|".

3. The Laotian acoustic characterization method based on Thai-Lao pronunciation similarity according to claim 1, characterized in that: The Step 1 specifically includes: Step 1.1, audio data sampling standardization: All original Thai and Lao audio data were uniformly resampled to 16kHz to eliminate feature deviations caused by inconsistent sampling rates; Remove samples with audio duration exceeding 20 seconds; Step 1.2: Text label standardization and Thai and Lao vocabulary construction: Perform uniform punctuation removal on the text transcriptions of all audio to reduce the interference of irrelevant characters on acoustic modeling; Extract all unique characters that appear in the training corpus and, combined with special symbols, construct independent Thai and Lao vocabulary tables for Thai and Lao respectively to define the model output space; Step 1.3, Data consistency check: Check that the audio data corresponds to the text labels one by one and exclude missing or abnormal samples.

4. The Laotian acoustic characterization method based on Thai-Lao pronunciation similarity according to claim 1, characterized in that: In Step 2, the multilingual self-supervised pre-training model uses Wav2vec-XLSR-300m.

5. The Laotian acoustic characterization method based on Thai-Lao pronunciation similarity according to claim 1, characterized in that: In the Step 2, during the fine-tuning stage, the low-level feature extractor is frozen, and only the Transformer encoder and output classification layer are trained.

6. The Laotian acoustic characterization method based on Thai-Lao pronunciation similarity according to claim 1, characterized in that: In Step 2 and Step 3, the AdamW optimizer, mixed precision training, gradient accumulation, and dynamic model checkpoint saving techniques are used during the training process.

7. The Laotian acoustic characterization method based on Thai-Lao pronunciation similarity according to claim 1, characterized in that: Step 2 includes: Load the pre-trained weights of Wav2vec-XLSR-300m, freeze the CNN feature extraction layer, initialize the linear classification layer according to the Thai vocabulary, fine-tune the model on the Thai training set, and optimize the CTC loss training. During this process, batch normalization, mixed precision, and dynamic model saving are performed, and the best model is retained to obtain the fine-tuned model for Thai.

8. The Laotian acoustic characterization method based on Thai-Lao pronunciation similarity according to claim 1, characterized in that: The Step 2-Step 3 specifically include: (1) Model initialization; We selected the publicly available multilingual self-supervised pre-trained model Wav2vec-XLSR-300m and loaded the pre-trained weights. We also froze the low-level CNN feature extraction layers and only trained the high-level Transformer encoder and output classification layers. (2) The first stage: fine-tuning of Thai; Using the Thai training set and Thai vocabulary, we randomly initialized the linear classification layer and used the CTC loss function for model training. During the training process, we used batch normalization, mixed precision, and learning rate hot start techniques to improve training efficiency and stability. Fine-tune for several rounds, save model checkpoints every few steps, and finally save the best performance model; (3) The second stage: Lao language fine-tuning; Load the model weights after the first stage of Thai fine-tuning, remove the original Thai classification layer, and create a new and randomly initialized Lao classification layer based on the Lao vocabulary; Using the Laotian training set, freeze the feature extraction layer and only fine-tune the Transformer encoder and the new classification layer, continuing to optimize the CTC loss. The training strategy remains consistent with the first stage, using step evaluation and dynamic checkpoint saving mechanisms.

9. A Laotian acoustic representation system based on the similarity of Thai and Lao pronunciations, characterized in that: The system includes: a module for executing the Lao acoustic characterization method based on Thai-Lao pronunciation similarity according to any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the Lao acoustic characterization method based on Thai-Lao pronunciation similarity as described in any one of claims 1 to 8 is implemented.

Images