Data construction and dynamic resampling fine-tuning method and system for multi-dialect speech recognition

Abstract

The application provides a data construction and dynamic resampling fine-tuning method and system for multi-dialect speech recognition. Firstly, the existing speech recognition model is used to recognize and transcribe the dialect audio, clean it, and perform environment-related data enhancement on the audio based on acoustic environment simulation. Secondly, the dialect type, gender and speech speed features of the audio are extracted, and the data set is divided into multiple category buckets according to the feature combination. In the batch generation stage of model fine-tuning, exponential decay probability is used for sampling in the bucket to meet the allocated basic extraction quota, and dynamic balance scores are used to guide cross-bucket compensation. Finally, combined with the ladder data enhancement suitable for the number of historical sample extractions, the model parameter update is completed. The application solves the problem that the existing dialect recognition method cannot simultaneously consider multi-dimensional attribute balance, sample coverage and over-sampling risk control, and improves the recognition robustness and generalization ability of the model in harsh recording environments and multi-dialect cross-scenarios.

Description

Technical Field

[0001] This invention belongs to the field of automatic speech recognition, specifically relating to a data construction and dynamic resampling fine-tuning method and system for multi-dialect speech recognition. Background Technology

[0002] In recent years, thanks to the rapid development of deep learning technology, Automatic Speech Recognition (ASR) has made breakthrough progress in resource-rich languages ​​such as Mandarin and English, and has been widely applied in intelligent interaction, speech transcription, and other scenarios. However, in practical applications, existing mainstream speech recognition models often experience severe performance degradation and a significant drop in recognition accuracy when faced with various Chinese dialects. This is due to two main reasons: firstly, China has a wide variety of dialects, and different regional dialects exhibit significant acoustic differences from standard Mandarin in terms of tone, initials, finals, and vocabulary, making it difficult for models to generalize directly; secondly, improving the model's recognition ability in dialect scenarios heavily relies on fine-tuning training with specific dialect data, but currently available open-source or commercial dialect datasets are limited in variety and cannot cover a broad range of dialect categories.

[0003] Faced with the enormous demand for multi-dialect speech recognition, acquiring and constructing training data from publicly available dialect audio and video resources has become a promising solution. However, a mature solution for obtaining high-quality annotated dialect audio from publicly available dialect audio and video resources is still lacking. Directly utilizing these audio and video resources to construct datasets and fine-tune and train models faces two main technical challenges:

[0004] First, the dataset construction process faces a contradiction between the accuracy of automatic annotation and the actual robustness of the model. On the one hand, the audio quality of publicly available audio and video resources varies greatly. If an automatic annotation strategy is adopted, the generated transcribed text will have a large number of inaccurate annotations. On the other hand, if the audio is uniformly denoised to improve the annotation quality, and then the denoised audio is used to fine-tune the ASR model, the fine-tuned model will lack robustness in real acoustic scenarios. Therefore, data augmentation is needed for the annotated dialect audio. However, traditional stationary white noise augmentation methods are prone to overfitting of the speech recognition model and cannot realistically reproduce hardware degradation phenomena such as poor microphone recording, equipment aging distortion, or popping sounds. Existing technologies struggle to balance high-quality pseudo-label generation and high-robust extreme environment simulation in a single processing pipeline.

[0005] Second, the fine-tuning training phase faces a severe problem of multi-dimensional class imbalance in the dataset. Due to the varying difficulty in acquiring dialect audio and video resources across different regions, mixed dialect datasets not only exhibit a severe long-tail distribution in the core dimension of "dialect type" (i.e., a large number of Mandarin and minority dialect samples, and a small number of majority dialect samples), but also suffer from severe cross-imbalance in dimensions such as "gender" and "speech rate." During model fine-tuning, this multi-dimensional imbalance can easily lead to performance shifts in the model—that is, the model may favor dialect types, genders, or speech rates where it has a data advantage, thereby weakening its generalization ability in multi-dialect scenarios. Currently, the mainstream solutions to the problem of class imbalance in training data fall into two categories: one is resampling methods, which change the distribution of training data by oversampling minority class samples or undersampling majority class samples; the other is adjusting the loss function, which assigns differentiated loss weights to samples of different classes. Current resampling methods can usually only cope with uneven distribution in a single dimension (such as only for dialect categories) and cannot simultaneously take into account multi-dimensional class balance, sampling coverage, and oversampling risk control. Summary of the Invention

[0006] Purpose of the Invention: The purpose of this invention is to provide a data construction and dynamic resampling fine-tuning method and system for multi-dialect speech recognition. This method utilizes publicly available dialect audio and video resources to construct a multi-dialect audio dataset that highly reproduces real and harsh acoustic environments. It also addresses the problem of data class imbalance caused by the intersection of multiple attributes such as dialect type, gender, and speech rate during model fine-tuning. This overcomes the performance bias of the model in a few categories and improves the generalization ability and recognition robustness of the speech recognition model in complex and variable environments and multi-dialect scenarios.

[0007] Technical Solution: To achieve the above-mentioned objectives, the first aspect of this invention provides a method for data construction and dynamic resampling fine-tuning for multi-dialect speech recognition, comprising the following steps:

[0008] Step 1: Obtain audio and video resources containing dialect speech, extract human voice segments within a specified duration range, and perform recognition, transcription, and cleaning using existing speech recognition models. After performing environment-related data augmentation on the audio based on acoustic environment simulation, construct a multi-dialect audio dataset.

[0009] Step 2: For the multi-dialect audio dataset, extract the dialect type, gender, and speech rate features of each audio file, and define mutually exclusive subsets with three feature dimension subclass combinations as category buckets, and divide the dataset into multiple category buckets according to feature combinations;

[0010] Step 3: In the batch generation stage of fine-tuning the target speech recognition model, the probability of candidate samples being extracted is calculated using exponential decay probability within each category bucket, and samples are sampled to meet the allocated basic extraction quota. Each sample has an upper limit on the number of times it can be used. Samples exceeding the upper limit are not included in the extractable sample pool. If the number of candidate samples in a bucket is insufficient and a quota gap occurs, the category bucket with the smallest weighted Hamming distance to the feature combination of the bucket to be compensated is selected for cross-bucket compensation sampling, and a dynamic balance score is calculated.

[0011] Step 4: Perform dynamic balanced score gating verification on the current batch that has completed sampling; for batches that have been verified and approved, perform stepwise data augmentation based on the historical number of times the samples have been sampled, and send the processed batch data into the target speech recognition model for forward and backward propagation to update the network parameters until all samples in the training dataset have been selected at least once.

[0012] Furthermore, in step 1, the environmental data enhancement of the audio based on acoustic environment simulation includes:

[0013] By using the modulo operation of the hash value of the audio path, deterministic local sampling of a set ratio is achieved, and the sampled audio is used as clean speech for environmental data enhancement.

[0014] The waveform-level fusion of real non-stationary environmental noise with clean speech includes: aligning the noise length with the speech length, scaling the energy based on different target signal-to-noise ratios to obtain a scaling factor, and adding the clean speech with the product of the scaling factor and the noise to obtain the fused audio.

[0015] Simulating hardware degradation and poor recording environments includes: using a bandpass filter to cut the frequency of the fused audio and performing amplitude clipping on the filtered audio to simulate hardware degradation; and superimposing spatial reverberation on the audio after hardware degradation simulation to simulate poor recording environments.

[0016] Furthermore, step 2 specifically includes:

[0017] The gender of each audio sample is extracted using a pre-trained gender recognition model;

[0018] Based on the ratio of the number of words in the transcribed text to the audio duration of each audio sample, the speech rate of each audio sample is calculated, and several speech rate intervals are divided according to the set speech rate dividing line.

[0019] The extracted gender, speech rate, and known dialect category are used as three feature dimensions. The dataset is then divided according to feature combinations. There are three mutually exclusive subsets, where all samples in the subset belong to their corresponding feature combination subclasses. These mutually exclusive subsets are defined as buckets; the total number of category buckets is... It equals the product of the number of dialect subcategories, the number of speech rate range subcategories, and the number of gender-fixed subcategories.

[0020] Furthermore, the basic extraction quota is determined by the ratio of batch size to the total number of category buckets; in step 3, within each category bucket, the exponential decay formula is used to calculate the extraction quota for each candidate sample. The probability of being selected :

[0021]

[0022]

[0023] in, , Candidate samples , The number of times the history was extracted. Candidate samples The total number of candidate samples in the bucket. This is the probability decay coefficient. This is a fixed upper limit on the number of times a single sample can be drawn. Then Set to 0 and sample Candidate samples are excluded from the pool within the bucket.

[0024] Furthermore, in step 3, the batch is calculated according to the following formula. Dynamic balance score :

[0025]

[0026] in, Represents the dimension of dialect types, Representing the gender dimension, Represents the speech rate dimension. , , These represent the actual sample proportions of subclass c for each dimension. , , These represent the target ideal proportions for each subclass c in each dimension. , , These are the weight parameters for each dimension; when a category bucket generates a quota gap, the weighted Hamming distance between the feature combination of the bucket to be compensated and the candidate compensation buckets is calculated, and samples are extracted from the category bucket with the smallest weighted Hamming distance for compensation; if multiple category buckets simultaneously meet the condition of having the smallest distance to the bucket to be compensated, the bucket with the largest remaining amount in the candidate pool is selected for compensation; the weight parameters for each dimension in the weighted Hamming distance are consistent with the weight parameters in the dynamic balance score.

[0027] Furthermore, the weights of the dialect category dimension, the gender dimension, and the speech rate dimension are dynamically adjusted at different training stages. The weight of the dialect category dimension is higher than that of the gender dimension, and the weight of the gender dimension is higher than that of the speech rate dimension, with the sum of the three being 1. In the early and middle stages of training, the weight of the dialect category dimension is set to be greater than 0.5. In the later stages of training, after the dialect performance of the model tends to be balanced, the weight of the dialect category dimension is smoothly transitioned to 0.5.

[0028] Furthermore, in step 4, the specific methods for gating verification and tiered data augmentation are as follows:

[0029] Set a gating parameter. If the dynamic balance score of the current batch is less than or equal to the gating parameter, the batch is directly allowed; otherwise, all audio files in the category bucket where the quota shortfall first appeared are put back into the candidate pool, and the maximum number of times audio files in that category bucket are extracted is increased. Reset to 1, and record the historical number of samples drawn in this round. Decrease by 1 and return to step 3 to perform sample extraction again; for batches that have been approved, extract samples based on the historical number of extractions. Perform differentiated speech enhancement: when Use the original audio at the time, as Additions include data enhancements that provide a step-by-step upgrade to the audio, such as adding noise, pitch shifting, frequency domain masking, and frequency domain stretching.

[0030] The second aspect of this invention provides a data construction and dynamic resampling fine-tuning system for multi-dialect speech recognition, comprising:

[0031] The dialect dataset construction module is used to acquire audio and video resources containing dialect speech, extract human voice segments within a specified duration range, and perform recognition, transcription and cleaning through existing speech recognition models. After performing environment-related data augmentation on the audio based on acoustic environment simulation, a multi-dialect audio dataset is constructed.

[0032] The multi-dimensional feature extraction and classification module is used to extract the dialect type, gender and speech rate features of each audio file for the multi-dialect audio dataset, and define mutually exclusive subsets with three feature dimension subclass combinations as category buckets, and divide the dataset into multiple category buckets according to feature combinations.

[0033] The dynamic balancing sampling and compensation module is used in the batch generation stage of fine-tuning the target speech recognition model. It calculates the probability of candidate samples being extracted in each category bucket using exponential decay probability and samples them to meet the allocated basic extraction quota. Each sample has an upper limit on the number of times it can be used. Samples exceeding the upper limit are not included in the extractable sample pool. If the number of candidate samples in a bucket is insufficient and a quota gap occurs, the category bucket with the smallest weighted Hamming distance to the feature combination of the bucket to be compensated is selected for cross-bucket compensation sampling, and the dynamic balancing score is calculated.

[0034] The gating verification and model update module is used to perform dynamic balanced score gating verification on the current batch that has completed sampling. For batches that have been verified and approved, stepwise data augmentation is performed based on the historical number of times the samples have been sampled. The processed batch data is then fed into the target speech recognition model for forward and backward propagation to update the network parameters until all samples in the training dataset have been selected at least once.

[0035] The third aspect of the present invention provides a computer system including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the data construction and dynamic resampling fine-tuning method for multi-dialect speech recognition described in the first aspect.

[0036] The fourth aspect of the present invention provides a computer program product, comprising a computer program that, when executed by a processor, implements the steps of the data construction and dynamic resampling fine-tuning method for multi-dialect speech recognition described in the first aspect.

[0037] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0038] 1. In terms of data construction, this invention uses existing speech recognition models for recognition, transcription and cleaning, and enhances the audio with environment-related data based on acoustic environment simulation. This effectively solves the defect that traditional stationary white noise can easily lead to overfitting of speech recognition models, ensures the reproducibility of noise-adding processing of massive data, and improves the recognition stability of the model in extreme acoustic environments.

[0039] 2. Regarding model fine-tuning, to address the issue of imbalanced category distribution in the training set, this invention constructs fine-grained category buckets across three dimensions: dialect type, gender, and speech rate. During the batch generation stage of fine-tuning the target speech recognition model, the probability of candidate samples being drawn within each category bucket is calculated using exponential decay probability, and sampling is performed to meet the allocated basic drawing quota. If the number of candidate samples within a bucket is insufficient, resulting in a quota gap, the category bucket with the smallest weighted Hamming distance to the bucket to be compensated is selected for cross-bucket compensation sampling, and a dynamic balance score is calculated. Through the exponential decay probability sampling mechanism and the cross-bucket compensation strategy guided by the dynamic balance score, as well as the stepwise data augmentation adapted to the historical number of draws, this invention solves the problem that existing methods cannot simultaneously balance multi-dimensional attributes, sample coverage, and control over oversampling risk.

[0040] 3. Experiments show that, compared with existing technologies, this invention achieves real-time dynamic balance of training data in multiple dimensions, effectively alleviates the performance deviation of the model caused by long-tailed data distribution and cross-imbalance, and significantly improves the generalization ability and recognition robustness of the speech recognition model in multi-dialect scenarios and real complex recording environments. Attached Figure Description

[0041] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0042] Figure 2 This is a flowchart illustrating the construction of a multi-dialect audio dataset according to an embodiment of the present invention.

[0043] Figure 3 This is a flowchart of the dynamic resampling fine-tuning process according to an embodiment of the present invention. Detailed Implementation

[0044] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0045] like Figure 1 As shown in the figure, the present invention discloses a data construction and dynamic resampling fine-tuning method for multi-dialect speech recognition, which mainly includes the following steps:

[0046] Step 1: Obtain audio and video resources containing dialect speech, extract human voice segments within a specified duration range, and perform recognition, transcription, and cleaning using existing speech recognition models. After performing environment-related data augmentation on the audio based on acoustic environment simulation, construct a multi-dialect audio dataset.

[0047] Step 2: For the multi-dialect audio dataset, extract the dialect type, gender, and speech rate features of each audio file, and define mutually exclusive subsets with three feature dimension subclass combinations as category buckets, and divide the dataset into multiple category buckets according to feature combinations;

[0048] Step 3: In the batch generation stage of fine-tuning the target speech recognition model, the probability of candidate samples being extracted is calculated using exponential decay probability within each category bucket, and samples are sampled to meet the allocated basic extraction quota. Each sample has an upper limit on the number of times it can be used. Samples exceeding the upper limit are not included in the extractable sample pool. If the number of candidate samples in a bucket is insufficient and a quota gap occurs, the category bucket with the smallest weighted Hamming distance to the feature combination of the bucket to be compensated is selected for cross-bucket compensation sampling, and a dynamic balance score is calculated.

[0049] Step 4: Perform dynamic balanced score gating verification on the current batch that has completed sampling; for batches that have been verified and approved, perform stepwise data augmentation based on the historical number of times the samples have been sampled, and send the processed batch data into the target speech recognition model for forward and backward propagation to update the network parameters until all samples in the training dataset have been selected at least once.

[0050] Preferably, the environmental data enhancement of audio based on acoustic environment simulation in step 1 includes:

[0051] By using the modulo operation of the hash value of the audio path, deterministic local sampling of a set ratio is achieved, and the sampled audio is used as clean speech for environmental data enhancement.

[0052] The waveform-level fusion of real non-stationary environmental noise with clean speech can specifically include: aligning the noise length with the speech length, scaling the energy based on different target signal-to-noise ratios to obtain a scaling factor, and adding the clean speech with the product of the scaling factor and the noise to obtain the fused audio.

[0053] Simulating hardware degradation and poor recording environments can specifically include: using a bandpass filter to cut the frequency of the fused audio and performing amplitude clipping on the filtered audio to simulate hardware degradation; and superimposing spatial reverberation on the audio after hardware degradation simulation to simulate a poor recording environment.

[0054] The following is combined with Figure 2 This section provides a detailed explanation of the construction process of the multi-dialect audio dataset in step 1. Step 1 may specifically include:

[0055] Step 11: Obtain dialect audio and video resources and convert them to a unified audio format. For dialect video resources, FFmpeg can be used to convert the video file into a mono, 16kHz sampling rate, 16-bit PCM format .wav audio file. For dialect audio resources, they can be used directly.

[0056] Step 12: Use VAD to segment long audio files into audio segments. Specifically, you can use a pre-trained speech activity detection (VAD) model, such as Silero VAD, to perform speech activity detection on audio files and segment each audio file into audio segments of 5-10 seconds.

[0057] Step 13: To improve the accuracy of annotation, noise reduction and dereverberation are performed on all audio segments. Specifically, a noise reduction model, such as MPSENet, can be used to perform noise reduction and dereverberation on all audio segments.

[0058] Step 14: Use a high-performance dialect model (such as FireRedASR, Spark Speech Recognition Large Model, etc.) to annotate the audio and obtain the transcribed text. Due to limitations in audio quality, noise reduction models, and ASR model performance, the generated transcribed text may contain a small number of empty results, results misidentified as English, and results that are only partially recognized. Therefore, the transcribed text needs to be cleaned to filter out empty results, recognition results containing English, and results with fewer than a set threshold of recognized characters. Arabic numerals in the recognition results should be converted to Chinese numerals. Finally, the audio segments are uniformly numbered and archived according to their dialect category to prepare for subsequent determination of the audio dialect category based on the category index.

[0059] Step 15: Perform data augmentation based on specific acoustic environment simulation, which may include:

[0060] Step 151: In the data sampling control phase, conventional pseudo-random numbers are abandoned, and deterministic local sampling with a set proportion is performed using the modulo operation of the MD5 hash value of the audio file path. Let the audio file path be... The preset sampling ratio is The determination formula is:

[0061]

[0062] in This means retrieving the last 8 hexadecimal characters of the MD5 hash value of the audio path `path` and converting it to a decimal integer; if Then the audio will be included in the enhancement process, thereby ensuring the global reproducibility of noise-adding processing of massive data;

[0063] Step 152: The audio entering the enhancement stage is treated as clean speech, and waveform-level fusion is performed between the real non-stationary environmental noise and the clean speech. After aligning the noise length with the speech length, energy scaling is performed based on the target signal-to-noise ratio (SNR). Specifically, when the noise length is greater than or equal to the speech length, a noise segment of the same length as the speech is extracted; when the noise length is less than the speech length, the noise is cyclically spliced ​​and truncated to the speech length. A relatively high SNR (15dB) is set in the early stages of training to allow the model to learn to recognize clean features; as the epochs increase, the SNR threshold is gradually reduced (down to 0dB) to gradually increase the learning difficulty.

[0064] Let the pure speech signal be Its energy is The aligned noise signal is Its energy is The target signal-to-noise ratio is scaling factor The calculation formula is:

[0065]

[0066] The base audio after fusion is This achieves natural and diverse background sound coverage;

[0067] Step 153: Perform a hardware degradation simulation process, using a bandpass filter to remove the blended audio. High and low frequency bands are obtained Subsequently, to simulate distortion caused by microphone overload, equipment aging, or excessive recording gain, amplitude clipping is performed on the filtered audio. Let the amplitude threshold be... The clipped audio signal is represented as follows:

[0068]

[0069] This highly replicates the sound-receiving defects of a poor-quality microphone. Then, spatial reverberation is superimposed on the hardware-degraded simulated audio signal to further refine the clipped audio signal. Room impact response Convolution operations are performed to simulate echoes and reverberation in enclosed recording environments such as offices, interrogation rooms, and conference rooms, ultimately enhancing the audio. The calculation formula is:

[0070]

[0071] in The room impulse response can be obtained from actual measurements or generated through room acoustic simulation methods. Through convolution processing, the resulting signal highly replicates the harsh recording environment. This is used for subsequent training, thereby significantly improving the model's recognition stability in extreme acoustic environments.

[0072] In this embodiment, the construction of the multidimensional category bucket in step 2 may specifically include:

[0073] Step 21: Extract the gender of each audio sample using a pre-trained gender recognition model (such as Step-Audio 2);

[0074] Step 22: Each audio data point includes transcribed text. Based on the ratio of the number of words in the transcribed text to the audio duration, calculate the speech rate for each audio sample, such as:

[0075]

[0076] Then, based on the set speech rate dividing line, several speech rate intervals are divided;

[0077] Step 23: Combine the gender and speech rate extracted in Steps 21 and 22 with the known dialect category as three feature dimensions, and divide the dataset obtained in Step 1 into three parts according to feature combinations. There are mutually exclusive subsets, where samples within each subset belong to their corresponding feature combination subclass, such as "Lanzhou-Yinchuan Mandarin - Female Voice - Fast" and "Central Plains Mandarin - Male Voice - Slow". Here, the mutually exclusive subsets are defined as buckets. If there are dialect types... Subcategories, speech speed ranges include If there are two fixed subclasses for gender, then:

[0078]

[0079] In this embodiment, to maximize the balance of sample size across all dimensions, the basic extraction quota K is allocated evenly to each bucket, calculated as follows:

[0080]

[0081] in R represents the batch size, and R is the possible remainder. If R=0, then K samples are drawn from N buckets at the same time. If R≠0, then K samples are drawn from N buckets first, and then R buckets are randomly selected to draw 1 sample.

[0082] In step 3, within each category bucket, instead of random sampling, an exponential decay formula is used to calculate the value of each candidate sample. The probability of being selected :

[0083]

[0084]

[0085] in, , Candidate samples , The number of times the history was extracted before training began. ,sample After each draw, Add one to yourself. This is a fixed upper limit on the number of times a single sample can be drawn. Then Set to 0 and sample The threshold for removing candidate samples from the bucket is used to limit the number of times a single sample is drawn, preventing overfitting due to excessive sampling of a single sample, and improving the coverage of sampled samples in the training set. It is necessary to dynamically adjust by estimating the degree of imbalance under the current dataset partition. Candidate samples The total number of candidate samples in the bucket. This is the probability decay coefficient. The larger the sample size, the more... The faster the rate of decay of the probability of growth being drawn, the more this coefficient needs to be set. The value is dynamically adjusted. When it gets bigger, it should be reduced. This is to increase the probability of selecting samples that are drawn more frequently.

[0086] In this embodiment, the cross-bucket compensation sampling method in step 3 is as follows:

[0087] Define the current batch In dimensions (such as dialect types) ,gender Speech rate Subclass The actual sample proportion is The target ideal percentage is (Uniform distribution), calculate the deviation score between the actual proportion and the target proportion of all subcategories, i.e., the Dynamic Balance Score (DBS):

[0088]

[0089] in, Represents the dimension of dialect types, Representing the gender dimension, Represents the speech rate dimension. , , These represent the actual sample proportions of subclass c for each dimension. , , These represent the target ideal proportions for each subclass c in each dimension. , , These are the weight parameters for each dimension.

[0090] When a quota shortfall occurs in a certain category bucket, the weighted Hamming distance between the feature combinations of the bucket to be compensated and the candidate buckets to be compensated is calculated. Samples are preferentially extracted from the category bucket with the smallest weighted Hamming distance for compensation. If multiple category buckets simultaneously meet the condition of having the smallest distance to the bucket to be compensated, the bucket with the largest remaining candidate pool is selected for compensation. In this embodiment, the Hamming distance refers to the number of different features at corresponding positions between two category buckets. For example, "Central Plains-Male-Fast" and "Jianghuai-Male-Slow" have two different positions, so the Hamming distance is 2. The weighted Hamming distance has different weights for each position, and the weight parameters for each dimension are the same. , , By calculating the weighted Hamming distance between different combinations, we can see the change in the balance score after compensating with one of the combinations; the larger the weighted Hamming distance, the greater the change in the balance score.

[0091] For example, if the "Hebei-Shandong Mandarin - Female Voice - Fast" bucket generates two quota gaps, this embodiment will prioritize selecting the other bucket with the smallest weight dimension (speech speed dimension) for compensation. That is, samples are extracted from the "Hebei-Shandong Mandarin - Female Voice - Slow" bucket. If the number of available samples in that bucket is less than the quota gap, the "Hebei-Shandong Mandarin - Male Voice - Fast" bucket will continue to be extracted for compensation, and so on.

[0092] Weight , , Dynamically adjust at different stages of training to meet the needs. , In the early and middle stages of training, set up... A value greater than 0.5 indicates that the dialect performance of the model will tend to balance in the later stages of training. Smoothly transition to 0.5. For example, set weights according to dimension priority during the early and middle stages of training. , , In the later stages of training, after the model's dialect performance tends to balance, the weights are smoothly transitioned to... , , .

[0093] In this embodiment, the specific methods for gating verification and tiered data augmentation in step 4 are as follows:

[0094] Step 41: Set a gating parameter If the current batch's DBS(B) ≤ If the current batch's DBS(B) is greater than the specified value, the batch will be released directly; otherwise, the batch will be released directly. If the quota shortfall occurs earliest, all audio files in the category bucket that first appeared will be put back into the candidate pool, and the maximum number of times audio files in that category bucket can be drawn will be increased. Reset to 1, and record the historical number of samples drawn in this round. Decrease by 1 and return to step 3 to perform sample extraction;

[0095] Step 42: For the released batches, determine the number of samples drawn in history. Perform differentiated speech enhancement: when Use the original audio when 0 < At that time, with This includes data enhancement that provides a step-by-step upgrade to the audio (such as adding noise, pitch shifting, frequency masking, frequency stretching, etc.). For example, Use the original audio without any processing; Use a light frequency domain mask at times; Add a slight temporal mask at the time; Add mild white noise at times; Add heavy white noise at times; Add heavy frequency domain mask + heavy time domain mask + heavy white noise;

[0096] Step 43: Feed the sampled batch data into the target ASR model for forward and backward propagation, update the network parameters, and then return to step 3 to start the next training step until all samples in the training dataset have been selected at least once.

[0097] Figure 3 The complete flow of the dynamic resampling fine-tuning method in this embodiment is illustrated, as follows: Figure 3As shown, firstly, multidimensional metadata features are extracted to construct mutually exclusive category buckets. Then, the basic extraction quota for each category bucket is calculated. Next, the training step begins, determining if the number of candidate samples in a bucket is less than the basic extraction quota. If the result is yes, the category bucket with the smallest weighted Hamming distance is selected for compensation, and the dynamic balance score is calculated. Then, candidate samples are extracted from the bucket, and the extraction count counter is incremented. If the number of sample extractions exceeds the upper threshold, the sample is removed from the candidate sample pool. If the result is no, candidate samples are directly extracted from the bucket, and the extraction count counter is incremented. If the number of sample extractions exceeds the upper threshold, the sample is removed from the candidate sample pool. After extraction, the next step, gating verification, is performed: determining if the current dynamic balance score is less than or equal to the gating parameter. If the result is yes, the current batch sampling is directly allowed, and stepwise data augmentation is performed based on the number of sample extractions. If the result is no, all audio samples from the category bucket with the earliest quota shortfall are re-added to the candidate pool, and the upper limit for the number of extractions of audio samples in that category bucket is increased. Reset to 1, and record the historical number of samples drawn in this round. Decrease by 1 and return to the training step to start sampling again; after completing batch release and data augmentation, send the current batch of samples into the model for training. Finally, perform a judgment within the training epoch: determine whether all samples in the training set have been sampled. If the judgment result is no, continue to the next training batch; if the judgment result is yes, the training epoch ends.

[0098] The effectiveness of the present invention will be evaluated below with reference to specific experiments.

[0099] In this embodiment, the target speech recognition model uses the Whisper-small model, and the dataset used is the KeSpeech dataset (KeSpeech exhibits significant class imbalance in three dimensions: dialect, gender, and speech rate). The process is as follows:

[0100] Based on steps 1 and 2, the original audio sample dataset is obtained, and the audio is preprocessed. The specific process is as follows:

[0101] Download the KeSpeech dataset from HuggingFace or ModelScope, and organize the train_phase1 and dev_phase1 parts for the ASR task, adding known dialect category metadata labels to all audio;

[0102] The gender of each audio sample is extracted using the pre-trained gender recognition model wav2vec2-large-xlsr-53-gender-recognition-librispeech, and the metadata tag "gender" is added to all audio samples, indicating whether they are male or female.

[0103] Based on the transcribed text of each audio data, the speech rate of all audio is calculated, and then 180 (words per minute) is set as the speech rate dividing line. Based on this, a metadata tag "speed_tag" is added to all audio, indicating whether it is "fast" or "slow".

[0104] Based on the metadata tags of dialect category, gender, and speech rate, add a feature combination metadata tag "Bucket_id" to all audio, such as "Jiaoliao_female_slow_speech" and "Southwest_female_fast_speech".

[0105] The training set is divided into multiple category buckets based on the "Bucket_id" of the training set. Then, based on the batch size and the total number of category buckets during the training process, the basic extraction quota K for each bucket is allocated on an average basis.

[0106] Based on step 3, a dynamic sampler based on multidimensional attribute balancing is constructed. Specifically, within the PyTorch framework, the `torch.utils.data.Sampler` class is inherited, and its `__iter__` method's batch iteration logic is overridden to implement exponentially decaying probability sample extraction based on historical sampling counts and a cross-bucket compensation mechanism based on dynamic balancing scores. The exponential decay formula is used to calculate the probability of each candidate sample. The probability of being selected And according to the overall K samples are drawn from each bucket, with a fixed upper limit on the number of samples drawn per sample. probability decay coefficient When the number of candidate samples M in a certain bucket is less than the basic extraction quota K, a quota gap is generated. The weighted Hamming distance between the feature combinations of the bucket to be compensated and the candidate compensation buckets is calculated, and samples are extracted from the category bucket with the smallest weighted Hamming distance for compensation.

[0107] Based on step 4, an adaptive data collator is constructed to work in real-time with the dynamic sampler and performs stepped data augmentation. Specifically, an `AdaptiveWhisperCollator` class is defined and its `__call__` method is overridden. In the batch assembly stage of the `DataLoader`, a custom class takes over the `collate_fn` method, reads the global historical sampling frequency of samples in the current batch, and implements stepped acoustic perturbations of increasing difficulty at the feature tensor level. The gating parameters... The initial value is set to 0.15.

[0108] In this embodiment, Whisper-small is used as the fine-tuning experimental model. During the fine-tuning process, a dynamic sampler is used to take over the batch_sampler method of the Dataloader, and a Collator is used to take over the collate_fn method of the Dataloader. The input is a structured CSV file with metadata tags, audio paths, and audio transcribed text, and the model is fine-tuned based on dynamic resampling.

[0109] To demonstrate the effectiveness of the proposed method, it was applied to the KeSpeech dataset (using a portion of the training set and the entire test set) to fine-tune the Whisper-small model. The results were compared with those obtained using conventional resampling methods (downsampling), as shown in Tables 1 and 2. Here, CER represents the average word error rate, used to measure the model's recognition accuracy; the original model represents the model without fine-tuning; and conventional downsampling represents the model obtained by fine-tuning using conventional resampling methods. The results show that the dynamic resampling method proposed in this invention achieves optimal class balance across multiple dimensions, including dialect category, gender, and speech rate, significantly reducing performance differences between different categories.

[0110] Table 1 Comparison of experimental results for dialect types

[0111]

[0112] Table 2 Comparison of Experimental Results by Gender / Speech Speed

[0113]

[0114] Based on the same inventive concept, this invention provides a data construction and dynamic resampling fine-tuning system for multi-dialect speech recognition, comprising: a dialect dataset construction module, used to acquire audio and video resources containing dialect speech, extract human voice segments within a specified duration range, and perform recognition, transcription, and cleaning through an existing speech recognition model, and construct a multi-dialect audio dataset after performing environment-related data enhancement on the audio based on acoustic environment simulation; a multi-dimensional feature extraction and classification module, used to extract dialect type, gender, and speech rate features for each audio segment in the multi-dialect audio dataset, and define mutually exclusive subsets with three feature dimension subclass combinations as category buckets, and divide the dataset into multiple category buckets according to feature combinations; and a dynamic balancing sampling and compensation module, used for batch generation during fine-tuning of the target speech recognition model. In each stage, the probability of a candidate sample being selected is calculated using exponential decay probability within each bucket, and sampling is performed to meet the allocated basic sampling quota. Each sample has an upper limit on the number of times it can be used; samples exceeding the upper limit are not included in the sample pool. If the number of candidate samples in a bucket is insufficient, resulting in a quota gap, the bucket with the smallest weighted Hamming distance to the bucket to be compensated is selected for cross-bucket compensation sampling, and a dynamic balance score is calculated. The gating verification and model update module is used to perform dynamic balance score gating verification on the current batch after sampling is completed. For batches that are verified and allowed, stepwise data augmentation is performed based on the historical sampling count of the samples, and the processed batch data is fed into the target speech recognition model for forward and backward propagation to update the network parameters until all samples in the training dataset have been selected at least once.

[0115] The specific working processes of each module described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. The division of modules is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules may be combined or integrated into another system.

[0116] Based on the same inventive concept, the present invention provides a computer system including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the steps of the data construction and dynamic resampling fine-tuning method for multi-dialect speech recognition.

[0117] Based on the same inventive concept, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data construction and dynamic resampling fine-tuning method for multi-dialect speech recognition.

[0118] The program code used to implement the method of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the steps of the method of the present invention to be performed. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a standalone software package, or entirely on a remote machine or server. All aspects not detailed in this invention are well-known to those skilled in the art.

Claims

1. A method for data construction and dynamic resampling fine-tuning for multi-dialect speech recognition, characterized in that, Includes the following steps: Step 1: Obtain audio and video resources containing dialect speech, extract human voice segments within a specified duration range, and perform recognition, transcription, and cleaning using existing speech recognition models. After performing environment-related data augmentation on the audio based on acoustic environment simulation, construct a multi-dialect audio dataset. Step 2: For the multi-dialect audio dataset, extract the dialect type, gender, and speech rate features of each audio file, and define mutually exclusive subsets with three feature dimension subclass combinations as category buckets, and divide the dataset into multiple category buckets according to feature combinations; Step 3: In the batch generation stage of fine-tuning the target speech recognition model, the probability of candidate samples being extracted is calculated using exponential decay probability within each category bucket, and samples are sampled to meet the allocated basic extraction quota. Each sample has an upper limit on the number of times it can be used. Samples exceeding the upper limit are not included in the extractable sample pool. If the number of candidate samples in a bucket is insufficient and a quota gap occurs, the category bucket with the smallest weighted Hamming distance to the feature combination of the bucket to be compensated is selected for cross-bucket compensation sampling, and a dynamic balance score is calculated. Step 4: Perform dynamic balanced score gating verification on the current batch that has completed sampling; for batches that have been verified and approved, perform stepwise data augmentation based on the historical number of times the samples have been sampled, and send the processed batch data into the target speech recognition model for forward and backward propagation to update the network parameters until all samples in the training dataset have been selected at least once.

2. The data construction and dynamic resampling fine-tuning method for multi-dialect speech recognition according to claim 1, characterized in that, In step 1, the environmental data enhancement of the audio based on acoustic environment simulation includes: By using the modulo operation of the hash value of the audio path, deterministic local sampling of a set ratio is achieved, and the sampled audio is used as clean speech for environmental data enhancement. The waveform-level fusion of real non-stationary environmental noise with clean speech includes: aligning the noise length with the speech length, scaling the energy based on different target signal-to-noise ratios to obtain a scaling factor, and adding the clean speech with the product of the scaling factor and the noise to obtain the fused audio. Simulating hardware degradation and poor recording environments includes: using a bandpass filter to cut the frequency of the fused audio and performing amplitude clipping on the filtered audio to simulate hardware degradation; and superimposing spatial reverberation on the audio after hardware degradation simulation to simulate poor recording environments.

3. The data construction and dynamic resampling fine-tuning method for multi-dialect speech recognition according to claim 1, characterized in that, Step 2 specifically includes: The gender of each audio sample is extracted using a pre-trained gender recognition model; Based on the ratio of the number of words in the transcribed text to the audio duration of each audio sample, the speech rate of each audio sample is calculated, and several speech rate intervals are divided according to the set speech rate dividing line. The extracted gender, speech rate, and known dialect category are used as three feature dimensions. The dataset is then divided according to feature combinations. There are three mutually exclusive subsets, where all samples in the subset belong to their corresponding feature combination subclasses. These mutually exclusive subsets are defined as buckets; the total number of category buckets is... It equals the product of the number of dialect subcategories, the number of speech rate range subcategories, and the number of gender-fixed subcategories.

4. The data construction and dynamic resampling fine-tuning method for multi-dialect speech recognition according to claim 1, characterized in that, The basic extraction quota is determined by the ratio of batch size to the total number of category buckets; in step 3, within each category bucket, the exponential decay formula is used to calculate the extraction quota for each candidate sample. The probability of being selected : ； ； in, , Candidate samples , The number of times the history was extracted. Candidate samples The total number of candidate samples in the bucket. This is the probability decay coefficient. This is a fixed upper limit on the number of times a single sample can be drawn. Then Set to 0 and sample Candidate samples are excluded from the pool within the bucket.

5. The data construction and dynamic resampling fine-tuning method for multi-dialect speech recognition according to claim 1, characterized in that, In step 3, the batch size is calculated according to the following formula. Dynamic balance score : ； in, Represents the dimension of dialect types, Representing the gender dimension, Represents the speech rate dimension. , , These represent the actual sample proportions of subclass c for each dimension. , , These represent the target ideal proportions for each subclass c in each dimension. , , These are the weight parameters for each dimension; when a category bucket generates a quota gap, the weighted Hamming distance between the feature combination of the bucket to be compensated and the candidate compensation buckets is calculated, and samples are extracted from the category bucket with the smallest weighted Hamming distance for compensation; if multiple category buckets simultaneously meet the condition of having the smallest distance to the bucket to be compensated, the bucket with the largest remaining amount in the candidate pool is selected for compensation; the weight parameters for each dimension in the weighted Hamming distance are consistent with the weight parameters in the dynamic balance score.

6. The data construction and dynamic resampling fine-tuning method for multi-dialect speech recognition according to claim 5, characterized in that, The weights of dialect type, gender, and speech rate are dynamically adjusted at different training stages; the weight of dialect type is higher than that of gender, the weight of gender is higher than that of speech rate, and the sum of the three is 1. In the early and middle stages of training, the weight of the dialect category dimension is set to be greater than 0.

5. In the later stages of training, after the dialect performance of the model tends to be balanced, the weight of the dialect category dimension is smoothly transitioned to 0.

5.

7. The data construction and dynamic resampling fine-tuning method for multi-dialect speech recognition according to claim 1, characterized in that, In step 4, the specific methods for gated verification and tiered data augmentation are as follows: Set a gating parameter. If the dynamic balance score of the current batch is less than or equal to the gating parameter, the batch is directly allowed; otherwise, all audio files in the category bucket where the quota shortfall first appeared are put back into the candidate pool, and the maximum number of times audio files in that category bucket are extracted is increased. Reset to 1, and record the historical number of samples drawn in this round. Decrease by 1 and return to step 3 to perform sample extraction again; for batches that have been approved, extract samples based on the historical number of extractions. Perform differentiated speech enhancement: when Use the original audio at the time, as Additions include data enhancements that provide a step-by-step upgrade to the audio, such as adding noise, pitch shifting, frequency domain masking, and frequency domain stretching.

8. A data construction and dynamic resampling fine-tuning system for multi-dialect speech recognition, characterized in that, include: The dialect dataset construction module is used to acquire audio and video resources containing dialect speech, extract human voice segments within a specified duration range, and perform recognition, transcription and cleaning through existing speech recognition models. After performing environment-related data augmentation on the audio based on acoustic environment simulation, a multi-dialect audio dataset is constructed. The multi-dimensional feature extraction and classification module is used to extract the dialect type, gender and speech rate features of each audio file for the multi-dialect audio dataset, and define mutually exclusive subsets with three feature dimension subclass combinations as category buckets, and divide the dataset into multiple category buckets according to feature combinations. The dynamic balancing sampling and compensation module is used in the batch generation stage of fine-tuning the target speech recognition model. It calculates the probability of candidate samples being extracted in each category bucket using exponential decay probability and samples them to meet the allocated basic extraction quota. Each sample has an upper limit on the number of times it can be used. Samples exceeding the upper limit are not included in the extractable sample pool. If the number of candidate samples in a bucket is insufficient and a quota gap occurs, the category bucket with the smallest weighted Hamming distance to the feature combination of the bucket to be compensated is selected for cross-bucket compensation sampling, and the dynamic balancing score is calculated. The gating verification and model update module is used to perform dynamic balanced score gating verification on the current batch that has completed sampling. For batches that have been verified and approved, stepwise data augmentation is performed based on the historical number of times the samples have been sampled. The processed batch data is then fed into the target speech recognition model for forward and backward propagation to update the network parameters until all samples in the training dataset have been selected at least once.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the steps of the data construction and dynamic resampling fine-tuning method for multi-dialect speech recognition according to any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the data construction and dynamic resampling fine-tuning method for multi-dialect speech recognition according to any one of claims 1-7.

Images