A multi-granularity screening method for low-quality scarce speech data

Abstract

The application discloses a multi-granularity screening method for low-quality scarce voice data, which comprises the following steps: constructing a data quality evaluation and screening mechanism based on a unit granularity, calculating a deviation score based on model output and labeling, and using the deviation score to represent the deviation degree between the model prediction result and the data label; meanwhile, calculating a likelihood score based on the probability distribution of the model output, and using the likelihood score to represent the adaptation degree between the input sentence and the model; constructing a weight matrix by using the likelihood score and the deviation score of the voice data and the labeled text, and realizing high-quality data screening of different granularities by means of the weight matrix, and screening high-quality data which is helpful to model training. Therefore, the application can effectively utilize high-quality data for training on the basis of greatly retaining original data, thereby improving the accuracy of the model output result; in the scenes of data scarcity, low labeling quality and low voice quality, the application can effectively screen data and improve the performance of the training model.

Description

Technical Field

[0001] This invention belongs to the field of speech recognition technology, specifically relating to a multi-granularity screening method for low-quality and scarce speech data. Background Technology

[0002] With the development of deep learning, automatic speech recognition technology has been widely applied in many fields, such as intelligent voice assistants, real-time captioning, speech translation, and customer service systems. In the training process of speech recognition models, high-quality datasets are crucial for model performance. However, the collected speech data often suffers from inconsistent quality due to various reasons (such as environmental noise, unclear speech, and text annotation errors). This low-quality data can negatively impact model training, increasing training time and reducing the model's generalization performance. Therefore, when building a speech recognition system, how to select high-quality speech-text pairs from large-scale datasets becomes a critical issue.

[0003] Traditional data filtering methods primarily rely on manual intervention or use speech features (such as signal-to-noise ratio and speech rate) or text characteristics (such as length and punctuation distribution) to set rules for filtering data. However, manual filtering is often time-consuming and labor-intensive, difficult to apply to large-scale datasets, and its quality cannot be guaranteed. Rule-based automatic filtering lacks flexibility and is prone to missing irregular noise or labeling errors. Methods that use model prediction discrepancies to filter data, judging data quality by calculating the difference between the prediction results and the labels, have higher performance and accuracy, but rely on the performance of existing models and often require a large amount of training data.

[0004] For resource-constrained environments, taking dialect speech recognition as an example, due to the high difficulty of collecting and labeling dialect data, dialect speech recognition tasks generally suffer from problems such as insufficient data, inconsistent labeling, noise, and mixed accents. Chinese patent application CN115424616A provides a data filtering technique suitable for training acoustic models for speech recognition. Its core idea is to measure the reliability of audio-text pairs by the consistency between "alignment results" and "automatic recognition results," thereby constructing a high-quality training set. However, this patented technology mainly addresses the "noise problem in training data caused by incorrectly labeled text," and is a general filtering scheme suitable for "large amounts of data with varying quality." It lacks a mechanism for multi-granular quality evaluation and refined utilization of "low-quality and scarce speech data." Chinese patent application CN116863917A discloses a method for filtering Vietnamese speech recognition data based on acoustic spatial distribution constraints. It focuses on Vietnamese speech recognition in low-resource conditions and addresses the data imbalance problem of tonal languages ​​under low-resource conditions by improving model performance through "pseudo-labels + acoustic spatial distribution constraints + self-training". However, this data filtering technology mainly focuses on the two dimensions of "pseudo-label quality" and "tone distribution balance", and still has shortcomings in terms of granularity, quality dimension and scarce data utilization strategy.

[0005] Therefore, existing methods are insufficient in terms of efficiency, accuracy, and robustness, and cannot fully cope with complex noisy environments or labeling errors. To improve the training efficiency and performance of speech recognition models, a method that can automatically and accurately select high-quality data is urgently needed. Summary of the Invention

[0006] To address the issues of high annotation error rates and noise / accent interference in datasets, this invention provides a multi-granularity screening method for low-quality, scarce speech data. This method allows for the specification of granularity and screening criteria, removing most annotation errors and noise while preserving high-quality speech signals. This improves the efficiency and accuracy of model training and testing, and is applicable to speech recognition tasks in low-resource environments such as dialect speech recognition, domain-specific speech recognition, multi-dialect speech recognition, and children's speech recognition.

[0007] A multi-granularity screening method for low-quality, scarce speech data includes the following steps: (1) Use the original dataset to train the model, and use the resulting model as the original model; (2) Specify the granularity of data filtering according to the requirements of the task scenario, and use the original model to calculate the likelihood score and deviation score of the original dataset in units (characters, words, phonemes or sentences); (3) The likelihood score and deviation score are weighted and summed according to the requirements of the task scenario, and the result is used as the quality score of each unit; (4) Perform feature transformation on the quality scores, use the transformed results as unit weights to construct a weight matrix, and pair the weight matrix with the original dataset to construct a new dataset; (5) Train the original model using the new dataset, and fine-tune the model using the weight matrix to obtain the refined model.

[0008] Furthermore, in step (1), the original model is obtained by training any of the following models using the original dataset; Statistical modeling-based models (such as Gaussian mixture models) can model and classify speech signals by probabilistically modeling the distribution of speech features. Models based on recurrent neural networks (such as long short-term memory networks) are used to learn the temporal features of speech signals by leveraging their ability to model time-series data. Models based on encoder-decoder structures (such as the Encoder-Decoder model) extract high-level semantic representations of input speech features through an encoder and generate corresponding output sequences through a decoder, thereby achieving end-to-end speech modeling or recognition.

[0009] Furthermore, step (2) specifies the granularity of data filtering, specifically: If sentences are used as units, then each voice sample data is filtered. If words or characters in a sentence are used as units, the pronunciation and annotation quality of each character in each speech data point are evaluated and filtered. If the phonemes or initials and finals in a sentence are used as units, the pronunciation and annotation quality of each phoneme in each speech data is evaluated and filtered.

[0010] Furthermore, the likelihood score in step (2) is calculated as follows: When the original model adopts a statistical modeling-based model, the likelihood score of the corresponding sample is calculated based on the likelihood probability or log-likelihood value of the sample features under each Gaussian component, which is applicable to the unit at the sentence level. When the original model adopts a model based on a recurrent neural network, the probability distribution is weighted or aggregated according to the predicted probability distribution output by the model at each time step, and the likelihood score of the corresponding sample is calculated, which is applicable to the unit at the sentence level. When the original model adopts an encoder-decoder structure, the likelihood score of the sample is calculated based on the conditional probability, average log probability, or attention weight information of the sequence generated in the decoding stage, supporting various levels of units.

[0011] The likelihood score refers to the probability that the model will generate speech data samples. Its value can be obtained from the likelihood probability, log-likelihood, mean prediction probability, conditional probability, or their normalized transformations. The calculation method is as follows: , For the first t Step model for the first i The likelihood score represents the probability of a token. The closer the likelihood score is to 1, the more sharp the probability distribution, and the easier it is to predict the current token. The closer the likelihood score is to 0, the more even the probability distribution, and the less contextual information is available, with the entire vocabulary competing for the number of valid candidates.

[0012] Furthermore, the deviation score in step (2) is calculated as follows: When the original model is not based on an encoder-decoder structure, the original model is used to predict each sample in the original dataset to obtain the model's predicted output sequence. The model's predicted output sequence is then compared with the corresponding reference labeled sequence for that sample. The edit distance between the two is calculated, and normalization is performed to eliminate the influence of sentence length to characterize the deviation of the sample. The normalized result is recorded as the deviation score, which is applicable to the unit at the sentence level. The edit distance is used to measure the minimum number of editing operations required to convert the model's predicted output sequence into the reference labeled sequence. The editing operations include at least one or more of the following: insertion, deletion, and replacement operations. When the original model adopts an encoder-decoder structure, the probability distribution of the output of the original model at the corresponding position of each sample in the original dataset is obtained, and the probability distribution is compared with the reference probability distribution corresponding to the sample. The KL divergence between the two is calculated as a loss function to characterize the degree of distribution difference between the model prediction result and the reference label. Then, the gradient during backpropagation is used to characterize the deviation of the sample. The normalized result is recorded as the deviation score, which supports various levels of units. The KL divergence is used to measure the information loss introduced by using the model to predict the probability distribution when given a reference probability distribution. The gradient of this loss reflects the degree of contribution of the sample to the model training.

[0013] The deviation score refers to the degree of deviation of the model's prediction result from the label. Its value can be obtained from the edit distance, information entropy, the mean squared error between the label and the output, or their normalization transformation. The closer the normalized deviation score is to 1, the greater the difference between the model's prediction and the reference label, the more "abnormal" the current sample is, the more difficult it is for the model to learn the sample, and the further the sample deviates from the training distribution. The closer the deviation score is to 0, the closer the model's prediction is to the reference label, the easier it is for the current sample to learn, and the less the sample deviates from the training distribution.

[0014] Furthermore, the calculation expression for the quality fraction of each unit in step (3) is as follows: Quality score = λ × Likelihood score + (1 - λ) × (1 - Deviation score) Specifically, the weight parameter λ is set or dynamically adjusted based on the task scenario, data quality, and model training stage, and is determined through human experience, statistical analysis, or model learning methods. In application scenarios where the credibility of the labeled information is low or the data is scarce, the value of the weight parameter λ is increased to enhance the influence of the likelihood score in quality assessment. In application scenarios where the reliability of the labeled information is high or the requirements for output accuracy are high, the value of the weight parameter λ is decreased to enhance the influence of the deviation score in quality assessment.

[0015] Furthermore, in step (4), the quality fraction is transformed using any of the following transformation methods; Feature transformations based on temporal smoothing (including methods such as moving average, weighted average, exponential smoothing, or time window-based filtering): Define convolution kernels to perform temporal smoothing on the quality scores of adjacent units to reduce the impact of local outliers on the overall quality assessment results and improve the continuity and stability of quality scores in the time dimension. Feature transformation based on nonlinear mapping and interval filtering: Sort all units in ascending order of quality scores, define a filtering interval with an upper bound of α and a lower bound of β, select data ranked in the interval [α, β] as high-quality data and mark the unit weight as 1, and the rest as low-quality data and mark the unit weight as 0. Feature transformations based on normalization and normalization (including standardization based on mean and standard deviation, probability mapping based on normal distribution parameter estimation, min-max normalization, scaling or piecewise normalization) are used to normalize or normalize quality scores so that the quality scores are distributed between 0 and 1, and the statistical distribution conforms to the preset distribution characteristics, so as to eliminate the distribution bias caused by different data batches, different granularity units or different model output scales.

[0016] Further, the specific implementation of step (5) is as follows: First, the original model is used for training. During training, the dataset after pairing speech, text, and weight matrix is ​​used. In the weight matrix, low-quality data is set to low weights and high-quality data is set to high weights. When calculating the loss function and performing gradient descent, the weight matrix is ​​used to weight the loss matrix element by element. The weight matrix is ​​applied first and then gradient descent is performed. The training continues until convergence, and a new model is finally obtained. When the unit weight in the weight matrix is ​​set to 0, this part of the data does not actually participate in gradient descent and has no impact on the model, thereby achieving data filtering.

[0017] A computer device includes a memory and a processor, wherein the memory stores a computer program and the processor executes the computer program to implement the above-described multi-granularity screening method for low-quality, scarce speech data.

[0018] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described multi-granularity screening method for low-quality, scarce speech data.

[0019] This invention's method allows for data filtering at different granularities based on specific task scenarios. A quality score for each data point is derived using likelihood and deviation scores, reflecting the data's quality. While retaining as much data as possible, it filters out high-quality speech-text pairs, reducing the impact of mislabeling and noise on model training and thus improving training efficiency. This invention does not directly calculate single metrics such as edit distance or word error rate. Instead, it performs statistical analysis at different levels based on the scenario, combining deviation and likelihood scores as data filtering criteria. This multi-granularity data filtering effectively reduces the impact of labeling errors and noise on model training while retaining more labeled information, thereby improving model performance. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the multi-granularity screening method for low-quality and scarce voice data according to the present invention. Detailed Implementation

[0021] To describe the present invention in more detail, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] like Figure 1 As shown, the multi-granularity screening method for low-quality, scarce speech data of the present invention includes the following steps: Step S1: Train the model using the original dataset, and use the resulting model as the original model; Step S2: Specify the granularity of data filtering, and use the original model to calculate the likelihood score and deviation score of the original dataset in units (character, word, phoneme or sentence); Step S3: Based on the scenario requirements of the task, perform a weighted calculation on the likelihood score and the deviation score, and use the result as the quality score for each unit; Step S4: Perform feature transformation on the quality scores, use the calibrated results as weights to construct a weight matrix, and pair the weight matrix with the original dataset to construct a new dataset; Step S5: Train the model using the new dataset, and fine-tune the original model using the weight matrix to obtain the refined model; Step S6: Test the test set using the final model and calculate the accuracy of the test set.

[0023] Example 1: Experiment with the speech_commands dataset The `speech_commands` dataset is a voice command dataset created by the TensorFlow and AIY teams. This dataset consists of 1-second-long commands, totaling 105,829 samples, containing 35 different commands, with a sampling rate of 16kHz. Following the official algorithm, the dataset is divided in a 7:2:1 ratio: the training set contains 75,466 samples, the validation set contains 20,986 samples, and the test set contains 9,377 samples. For commands shorter than one second, padding is used to bring the total length to one second during data processing.

[0024] In this embodiment, to simulate a scenario with low annotation quality, annotation errors in the original training set are manually injected, with 1%, 3%, and 5% errors injected respectively. Together with the original training set, these form four training sets. The method of injecting annotation errors is to replace them with incorrect instructions. The four training sets are as follows: Group 0: Control group, with no labeling errors.

[0025] Group 1: 1% of annotation errors were injected.

[0026] Group 2: 3% of annotation errors were injected.

[0027] Group 3: 5% of annotation errors were injected.

[0028] In this embodiment, step S1 is specifically implemented as follows: Data processing section: Converts speech signals into 80-dimensional fbank features with a frame length of 25ms and a frame shift of 10ms, and maps instructions to numerical indices.

[0029] Model selection: ResNet18 was used as the model architecture, cross entropy was used as the loss function, and the batch size was 32.

[0030] Adam was used as the optimizer with a learning rate of 0.0001; cross entropy was used as the loss function, and the cross entropy between the probability distribution generated by the model and the true probability distribution was calculated using the following formula: ,in It is the probability distribution of the actual text labels. It is the probability distribution generated by the model.

[0031] For the training sets of four different groups, four models were trained respectively, and the model parameters were saved. The model obtained from group 0 was used as the baseline model, and the models from the other three groups were used as experimental groups for subsequent experiments.

[0032] In this embodiment, step S2 is implemented as follows: For each instruction in the speech_commands dataset, calculate its corresponding loss gradient; since the loss function uses cross-entropy, the gradient calculation formula used is mean squared error. Calculate the loss gradient values ​​for the three experimental groups respectively and record them.

[0033] This invention aims to filter out mislabeled information while preserving the original high-quality data to the maximum extent. Considering each character, speech recognition can be viewed as a classification task. To ensure that the score reflects the importance of the sample to model training, the high-quality data selection criterion is to evaluate the impact of training with that sample on the model's accuracy across the entire dataset. In the actual training process, this impact is the change in the model's accuracy before and after training, i.e., the output... f ( x ) and annotation y The difference with respect to time is the derivative, defined as: By utilizing the chain rule, weights are introduced into the model. Then the expression can be written as: The changes in weights are known during backpropagation, and are known as gradients: Since the derivative of the loss function with respect to the weights does not change within the same training batch, it can be considered a constant. c By combining the two equations, we can determine whether there is a sample or not. j Impact on model accuracy: Based on this, and according to the relationship between the gradient and the loss function, the formula can be further derived as follows: In this formula, the loss function uses cross entropy, so the derivative of the function value can be obtained by calculation: and The logits are roughly orthogonal across different logits and have similar magnitudes across different logits and training instances, so this part can be ignored. Therefore, the norm of the error vector can be directly used to approximate the evaluation result; according to the softmax function: Substituting into the formula for calculating cross-entropy, we obtain the derivative of the loss with respect to logits as follows: Therefore, the gradient of the corresponding loss function is the mean square error of the label probability distribution, which can be calculated using its L2 norm as a metric; the greater the loss, the greater the impact on the corresponding sample.

[0034] In this embodiment, step S3 is specifically implemented as follows: α = 0, at which point the quality score = 1 - deviation score.

[0035] In this embodiment, step S4 is implemented as follows: The quality scores of the top 10% and 90% of units in each of the three experimental groups are statistically analyzed and recorded for subsequent screening. The reason for this selection is that, from the perspective of how data affects model training performance, the loss values ​​of the bottom 10% of data have little impact on the model and are considered low-quality data that cannot enhance the model's generalization performance; removing them can shorten training time. Conversely, the top 10% of data are far from the model's hyperplane, making them difficult to fit during training; these are unrepresentative outliers and are likely noise or labeling errors. Therefore, using these two values ​​as screening criteria can identify high-quality data.

[0036] Using the above selection criteria, units in the original training set are selected. Since this dataset is an instruction set, a weight matrix is ​​not constructed. Instead, the units belonging to the selection results are directly constructed into a new dataset. Three experimental groups are formed, and three corresponding new datasets are constructed.

[0037] In this embodiment, step S5 is implemented as follows: The quality score of each instruction and the number of labeled errors during screening are used as samples; the quality score of the erroneously injected instructions is used as a negative sample; and the quality score of the correct instructions is used as a positive sample. An ROC curve is then plotted. The ROC curve, or Receiver Operating Characteristic curve, is a curve plotted using a series of different binary classification methods (cutoff values ​​or decision thresholds), with the true positive rate (sensitivity) on the ordinate and the false positive rate (1-specificity) on the abscissa. The area under the ROC curve is defined as the area under the ROC curve. In this experiment, the AUC value is used to evaluate the screening ability of this method.

[0038] To verify whether the screening strategy of removing the last 10% of the data can effectively eliminate mislabeled data, this experiment counted the number of mislabeled samples in the screened data to measure the screening performance.

[0039] set up The total amount of original data. The amount of data that was filtered out (approximately 10% of the dataset), This represents the number of incorrectly labeled samples in the data that were removed. This represents the total number of mislabeled samples in the original dataset.

[0040] The percentage of labeling errors in different groups is recorded as follows: Defined as: To further demonstrate the overall coverage of erroneous sample removal, an indicator is introduced. Its definition is: In terms of model performance evaluation metrics, this experiment uses accuracy and word error rate (WER) to measure the recognition performance of the ASR system. A lower WER value indicates a better model performance, while a higher WER value indicates a worse model performance. To measure the number of all words in the test set, The accuracy is defined as the number of words correctly identified by the model: The results are shown in Table 1: Table 1 The AUC values ​​of the three datasets show that the screening ability of this invention is significantly improved in all three groups.

[0041] In this embodiment, step S6 is implemented as follows: the three trained models are tested on the test set respectively, and the accuracy on the test set is calculated.

[0042] set up The word error rate before fine-tuning (or the baseline model), The relative improvement rate (RERR) is defined as the word error rate after fine-tuning (or model improvement). The results are shown in Table 2: Table 2 The accuracy on the test sets of the three datasets shows that the recognition performance of the present invention is significantly improved in all three groups. This method can not only filter out the correct instructions and eliminate most of the labeling errors, thereby improving the recognition performance of the model, but also filter out other unrepresentative outliers in the dataset, which has a better guiding role in the training of the model, making the results after the filtering training even better than the results of the original dataset.

[0043] Example 2: Experiment with the TIMIT dataset The TIMIT dataset is an acoustic-phoneme continuous speech corpus built collaboratively by Texas Instruments, MIT, and Stanford Research Institute. The dataset uses a 16kHz sampling frequency and contains 6300 sentences, each spoken by 630 individuals from eight major dialect regions in the United States. All sentences were manually segmented and labeled at the phoneme level. The recording environment was relatively quiet, with no significant channel or ambient noise. The dataset is approximately 650MB in size and uses the standard Kaldi partitioning method, dividing all speakers into three parts: training set (462), development set (50), and evaluation set (24 speakers).

[0044] In this embodiment, to simulate a real dialect speech recognition scenario, annotation errors in the original training set were manually injected at 1%, 3%, and 5% respectively. These, along with the original training set, constituted four training sets. When injecting annotation errors, mimicking real-world scenarios, insertion, deletion, and replacement errors were injected in a 1:1:3 ratio, as shown in the four groups below: Group 0: Control group, with no labeling errors.

[0045] Group 1: 1% of annotation errors were injected.

[0046] Group 2: 3% of annotation errors were injected.

[0047] Group 3: 5% of annotation errors were injected.

[0048] The specific implementation method of step S1 in this embodiment is as follows: Data processing section: Select an 80-dimensional fbank as a feature and map phonemes to numerical indices.

[0049] Model Selection: A transformer-based encoder-decoder model is used. The model consists of two main parts: an encoder, primarily used to extract features from the speech signal; and a decoder, primarily used to decode the features and map them to the text sequence. The encoder and decoder interact through an attention mechanism, ultimately outputting the text. Each attention module contains four attention heads, each with 256 dimensions. The feedforward network hidden layers have 2048 dimensions. The encoder contains 12 stacked attention blocks, and the decoder contains six stacked attention blocks.

[0050] Adam was used as the optimizer with a learning rate of 0.002 and a batch size of 16; cross entropy was used as the loss function.

[0051] The training sets for the four groups are as follows: Training set 1: The original training set, without any injected labeling errors.

[0052] Training set 2: The original training set, with 1% of labeling errors injected.

[0053] Training set 3: The original training set, with 3% of labeling errors injected.

[0054] Training set 4: The original training set, with 5% of the labeling errors injected.

[0055] The model was trained until convergence, resulting in the original models for four groups. The model obtained from training set 1 was used as the control group, while the models from the other three groups were used as the experimental groups for subsequent experiments.

[0056] In this embodiment, step S2 is implemented as follows: For each speech signal in the TIMIT dataset, the mean square error of its corresponding probability is calculated in units of phonemes. Three experimental datasets were prepared, and the loss gradient and the predicted probability distribution after probabilistic nonlinear mapping were calculated and recorded for each of the four groups. The specific calculation method for the probability distribution is as follows: in: Indicates the first i The quality score of each unit This represents the corresponding weight.

[0057] In this embodiment, step S3 is implemented as follows: define fusion weights α=0, α=0.5, and α=1 respectively, and calculate the quality scores of all phonemes in all sentences as subsequent screening indicators.

[0058] In this embodiment, step S4 is implemented as follows: a weight matrix is ​​constructed for each sentence, with the matrix length equal to the number of phonemes in the sentence. For each phoneme in the sentence, if the quality score of the phoneme ranks between the two indicators of 10% and 90%, the mask value of the phoneme is set to 1; otherwise, it is set to 0. In the experimental group with α=0, temporal smoothing and no temporal smoothing are used as experimental controls.

[0059] In this embodiment, step S5 is implemented as follows: The quality score and labeling error of each phoneme during screening are used as samples; phonemes with incorrect labeling are used as negative samples, and phonemes with correct labeling are used as positive samples. The AUC value is calculated using these samples to evaluate the screening capability of the method. Furthermore, to verify whether removing the last 10% of data can effectively remove labeling errors, statistics are also compiled here. To reflect the screening performance, different experimental groups were selected based on different α values. When α=0, the quality score = 1 - the deviation score, which can be considered as the screening result based on the deviation score; when α=1, the quality score = the likelihood score, which can be considered as the screening result based on the likelihood score. The results are shown in the table below: Table 3.1: Filtering results of deviation scores for the TIMIT dataset (α=0) Table 3.2: Likelihood score selection results for the TIMIT dataset (α=1) Table 3.3: Quality score screening results of the TIMIT dataset (α=0.5) Using the three pre-trained original models, we trained them on the corresponding new dataset, setting the learning rate to 1 / 10 of the original. During this training, the calculated loss was applied using a weight matrix. Matrix multiplication was used to multiply the loss of each unit by the corresponding weight. After this processing, gradient backpropagation was performed. Once the training converged, a new model was obtained.

[0060] In this embodiment, step S6 is implemented as follows: The three trained models are tested on the test set, and the accuracy on the test set is calculated. The results are shown in the table below. Table 4.1: Recognition results on the TIMIT dataset (α=0) Table 4.2: Recognition results on the TIMIT dataset (α=0.5) Table 4.3: Recognition results on the TIMIT dataset (α=1) Table 4.4: Recognition results on the TIMIT dataset (α=0+smoothing) The accuracy on the test sets of the three datasets shows that the recognition performance of the present invention is significantly improved in all three groups. This method can not only filter out the correct instructions and eliminate most of the labeling errors, thereby improving the recognition performance of the model, but also filter out other unrepresentative outliers in the dataset, which has a better guiding role in the training of the model, making the results after the filtering training even better than the results of the original dataset.

[0061] A horizontal comparison of experimental results under different values ​​of α reveals that in application scenarios with high labeling error rates, increasing the value of the weighting parameter λ to enhance the influence of the likelihood score on quality assessment yields better results. However, if there are too many labeling errors, the likelihood score given by the original model will also have low reliability. In this case, the weighting parameter should be appropriately reduced to enhance the influence of the deviation score on the quality score.

[0062] Example 3: Experiment with Hangzhou Dialect Dataset The Hangzhou dialect dataset is composed of Hangzhou dialect data collected and labeled by DataTang Company. It has a sampling rate of 16kHz, 16bit, uncompressed wav, mono, and a valid data duration of 248 hours. It includes data from 370 Hangzhou residents, of whom 307 are female and 63 are male. The accuracy of the labeled sentences is over 95%, and the audio quality is good, with no noise, plosives, or background sounds. The training set, test set, and validation set are divided in an 8:1:1 ratio.

[0063] The annotation errors in the Hangzhou dialect dataset were not manually injected, but rather 1000 speech sentences were randomly selected and manually verified to identify the annotation errors. Therefore, in this experiment, we only conducted experiments on the original training set.

[0064] The specific implementation method of step S1 in this embodiment is as follows: Data processing section: Select an 80-dimensional fbank as the feature and map each word to a numerical index.

[0065] Model Selection: A conformer-based encoder-decoder model, mainly divided into two parts: an encoder, primarily used to extract features from the speech signal; and a decoder, primarily used to decode the features and map them to the text sequence. The encoder and decoder interact through an attention mechanism, ultimately outputting text. The encoder is based on conformer and contains 12 stacked conformer modules, each with a 2048-dimensional feedforward hidden layer. The attention module contains 8 attention heads, each 256-dimensional, and the convolutional module has a kernel size of 15. The decoder is based on transformer and contains 3 stacked attention modules.

[0066] Adam was used as the optimizer with a learning rate of 0.002 and a batch size of 16; KL divergence was used as the loss function.

[0067] In this embodiment, the specific implementation method of steps S2 to S6 is the same as that of the TIMIT dataset experiment. The experiment is conducted with α=0, so it will not be repeated here. The experimental results are shown in Tables 5 and 6. The evaluation of the screening results in the experiment is achieved by manually proofreading 1000 randomly selected speech sentences in the data. The annotation error rate is 0.01. The recognition results are tested on the Hangzhou dialect test set, so the accurate annotation error rate cannot be determined.

[0068] Table 5 Table 6 It can be seen that the effect of this method is relatively limited when the amount of data is large, but it can still improve the performance of the model.

[0069] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. Those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. A multi-granularity screening method for low-quality, scarce speech data, characterized in that, The steps include the following: (1) Use the original dataset to train the model, and use the resulting model as the original model; (2) Specify the granularity of data filtering according to the requirements of the task scenario, and use the original model to calculate the likelihood score and deviation score of the original dataset in units; (3) The likelihood score and deviation score are weighted and summed according to the requirements of the task scenario, and the result is used as the quality score of each unit; (4) Perform feature transformation on the quality scores, use the transformed results as unit weights to construct a weight matrix, and pair the weight matrix with the original dataset to construct a new dataset; (5) Train the original model using the new dataset, and fine-tune the model using the weight matrix to obtain the refined model.

2. The multi-granularity screening method for low-quality, scarce speech data according to claim 1, characterized in that: In step (1), the original model is obtained by training any of the following models using the original dataset; The statistical modeling-based model achieves the modeling and classification of speech signals by probabilistically modeling the distribution of speech features. Based on the recurrent neural network model, its ability to model time series data is used to learn the temporal features of speech signals. The encoder-decoder model extracts high-level semantic representations of input speech features through the encoder and generates corresponding output sequences through the decoder, thus achieving end-to-end speech modeling or recognition.

3. The multi-granularity screening method for low-quality, scarce speech data according to claim 1, characterized in that, In step (2), the granularity of data filtering is specified, specifically: If sentences are used as units, then each voice sample data is filtered. If words or characters in a sentence are used as units, the pronunciation and annotation quality of each character in each speech data point are evaluated and filtered. If the phonemes or initials and finals in a sentence are used as units, the pronunciation and annotation quality of each phoneme in each speech data is evaluated and filtered.

4. The multi-granularity screening method for low-quality, scarce speech data according to claim 1, characterized in that, The likelihood score in step (2) is calculated as follows: When the original model adopts a statistical modeling-based model, the likelihood score of the corresponding sample is calculated based on the likelihood probability or log-likelihood value of the sample features under each Gaussian component, which is applicable to the unit at the sentence level. When the original model adopts a model based on a recurrent neural network, the probability distribution is weighted or aggregated according to the predicted probability distribution output by the model at each time step, and the likelihood score of the corresponding sample is calculated, which is applicable to the unit at the sentence level. When the original model adopts an encoder-decoder structure, the likelihood score of the sample is calculated based on the conditional probability, average log probability, or attention weight information of the sequence generated in the decoding stage, supporting various levels of units.

5. The multi-granularity screening method for low-quality, scarce speech data according to claim 1, characterized in that, The deviation score in step (2) is calculated as follows: When the original model is not based on an encoder-decoder structure, the original model is used to predict each sample in the original dataset to obtain the model's predicted output sequence. The model's predicted output sequence is then compared with the corresponding reference labeled sequence for that sample. The edit distance between the two is calculated, and normalization is performed to eliminate the influence of sentence length to characterize the deviation of the sample. The normalized result is recorded as the deviation score, which is applicable to the unit at the sentence level. The edit distance is used to measure the minimum number of editing operations required to convert the model's predicted output sequence into the reference labeled sequence. The editing operations include at least one or more of the following: insertion, deletion, and replacement operations. When the original model adopts an encoder-decoder structure, the probability distribution of the output of the original model at the corresponding position of each sample in the original dataset is obtained, and the probability distribution is compared with the reference probability distribution corresponding to the sample. The KL divergence between the two is calculated as a loss function to characterize the degree of distribution difference between the model prediction result and the reference label. Then, the gradient during backpropagation is used to characterize the deviation of the sample. The normalized result is recorded as the deviation score, which supports various levels of units. The KL divergence is used to measure the information loss introduced by using the model to predict the probability distribution when given a reference probability distribution. The gradient of this loss reflects the degree of contribution of the sample to the model training.

6. The multi-granularity screening method for low-quality, scarce speech data according to claim 1, characterized in that, The expression for calculating the quality score of each unit in step (3) is as follows: Quality score = λ × Likelihood score + (1 - λ) × (1 - Deviation score) Specifically, the weight parameter λ is set or dynamically adjusted based on the task scenario, data quality, and model training stage, and is determined through human experience, statistical analysis, or model learning methods. In application scenarios where the credibility of the labeled information is low or the data is scarce, the value of the weight parameter λ is increased to enhance the influence of the likelihood score in quality assessment. In application scenarios where the reliability of the labeled information is high or the requirements for output accuracy are high, the value of the weight parameter λ is decreased to enhance the influence of the deviation score in quality assessment.

7. The multi-granularity screening method for low-quality, scarce speech data according to claim 1, characterized in that: In step (4), the quality fraction is transformed using any of the following transformation methods; Feature transformation based on temporal smoothing: Define a convolution kernel to perform temporal smoothing on the quality scores of adjacent units, so as to reduce the impact of local outliers on the overall quality evaluation results and improve the continuity and stability of quality scores in the time dimension. Feature transformation based on nonlinear mapping and interval filtering: Sort all units in ascending order of quality scores, define a filtering interval with an upper bound of α and a lower bound of β, select data ranked in the interval [α, β] as high-quality data and mark the unit weight as 1, and the rest as low-quality data and mark the unit weight as 0. Feature transformation based on normalization and normalization: The quality score is normalized or normalized so that the quality score is distributed between 0 and 1, and the statistical distribution conforms to the preset distribution characteristics, so as to eliminate the distribution bias caused by different data batches, different granularity units or different model output scales.

8. The multi-granularity screening method for low-quality, scarce speech data according to claim 1, characterized in that, The specific implementation of step (5) is as follows: First, the original model is used for training. During training, the dataset after pairing speech, text, and weight matrix is ​​used. In the weight matrix, low-quality data is set to low weight and high-quality data is set to high weight. When calculating the loss function and performing gradient descent, the weight matrix is ​​used to weight the loss matrix element by element. The weight matrix is ​​applied first and then gradient descent is performed. The training continues until convergence, and a new model is finally obtained. When the unit weight in the weight matrix is ​​set to 0, this part of the data does not actually participate in gradient descent and has no impact on the model, thereby achieving data filtering.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: The processor is used to execute the computer program to implement the multi-granularity screening method for low-quality, scarce speech data as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the multi-granularity screening method for low-quality, scarce speech data as described in any one of claims 1 to 8.

Images