A speech recognition method, apparatus, device, storage medium, and product

Abstract

Embodiments of the present application provide a speech recognition method, device, equipment, storage medium and product. The technical solution provided by the embodiments of the present application obtains the recognition result of the to-be-processed speech information by analyzing and processing the to-be-processed speech information by using a speech recognition model, the speech recognition model is obtained by semi-supervised training based on unannotated sample audio and annotated sample audio after filtering processing and distribution balancing processing, the distribution of the unannotated sample audio after filtering processing and distribution balancing processing is more easily fitted to the distribution of the correctly annotated audio, the semi-supervised training of the speech recognition model is fully utilized by using the unannotated audio sample, the human and time costs of manual annotation are reduced, the influence of a large number of unannotated sample audios on the training efficiency and the recognition accuracy of the speech recognition model is reduced, and the training efficiency of the speech recognition model and the speech recognition effect are effectively improved.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a speech recognition method, apparatus, device, storage medium and product. Background Technology

[0002] With the development of network and artificial intelligence technologies, the application of automatic speech recognition is becoming increasingly widespread. Speech recognition is generally performed using a trained speech recognition model. This model requires a certain amount of sample data for training, and obtaining accurate output labels (speech text) from this sample data requires significant human and time investment. Sample data is typically obtained by labeling unlabeled online audio. Due to the large volume of unlabeled online audio, especially for less common languages, large-scale labeling operations result in substantial time consumption and labeling costs. If unsupervised training methods are used directly with unlabeled data, it is difficult to guarantee the accuracy of the model.

[0003] Semi-supervised training of speech recognition models is based on the consistency principle, introducing a separate task module or loss function to help the speech recognition model learn a good feature representation corresponding to the input. However, existing semi-supervised training methods generally add all unannotated sample data to the training or randomly select sample data to add to the training. However, a large amount of training data can easily consume a lot of time, resulting in low training efficiency of the speech recognition model. On the other hand, randomly selected sample data is difficult to guarantee data quality, resulting in poor recognition performance of the speech recognition model. Summary of the Invention

[0004] This application provides a speech recognition method, apparatus, device, storage medium, and product to solve the technical problems of low training efficiency and poor recognition effect of speech recognition models in related technologies, and effectively improve the training efficiency and speech recognition effect of speech recognition models.

[0005] In a first aspect, embodiments of this application provide a speech recognition method, including:

[0006] Acquire the voice information to be processed;

[0007] The speech information to be processed is input into the trained speech recognition model. The speech recognition model analyzes and processes the speech information to obtain the recognition result corresponding to the speech information to be processed. The speech recognition model is obtained through semi-supervised training based on unlabeled sample audio and labeled sample audio after filtering and distribution balancing. The initial teacher model of the semi-supervised training is obtained through supervised training.

[0008] In a second aspect, embodiments of this application provide a speech recognition device, including a speech acquisition module and a speech recognition module, wherein:

[0009] The voice acquisition module is configured to acquire voice information to be processed;

[0010] The speech recognition module is configured to input the speech information to be processed into a trained speech recognition model, and to analyze and process the speech information to be processed through the speech recognition model to obtain the recognition result corresponding to the speech information to be processed. The speech recognition model is obtained through semi-supervised training based on unlabeled sample audio and labeled sample audio after filtering and distribution balancing. The initial teacher model of the semi-supervised training is obtained through supervised training.

[0011] In a third aspect, embodiments of this application provide a voice recognition device, including: a memory and one or more processors;

[0012] The memory is used to store one or more programs;

[0013] When the one or more programs are executed by the one or more processors, the one or more processors implement the speech recognition method as described in the first aspect.

[0014] In a fourth aspect, embodiments of this application provide a non-volatile storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the speech recognition method as described in the first aspect.

[0015] In a fifth aspect, embodiments of this application provide a computer program product comprising a computer program stored in a computer-readable storage medium, wherein at least one processor of the device reads from the computer-readable storage medium and executes the computer program, causing the device to perform the speech recognition method as described in the first aspect.

[0016] This application embodiment analyzes and processes the speech information to be processed using a speech recognition model to obtain the recognition result of the speech information to be processed. The speech recognition model is obtained by semi-supervised training based on unlabeled sample audio after filtering and distribution balancing, and labeled sample audio. The distribution of unlabeled sample audio after filtering and distribution balancing is more likely to fit the distribution of correctly labeled audio. By making full use of unlabeled audio samples for semi-supervised training of the speech recognition model, the manpower and time cost of manual annotation is reduced, the impact of a large number of unlabeled sample audio on training efficiency and speech recognition model recognition accuracy is reduced, and the training efficiency and speech recognition effect of the speech recognition model are effectively improved. Attached Figure Description

[0017] Figure 1This is a flowchart of a speech recognition method provided in an embodiment of this application;

[0018] Figure 2 This is a schematic diagram of a semi-supervised training process for a speech recognition model provided in an embodiment of this application;

[0019] Figure 3 This is a schematic diagram of a first filtering process provided in an embodiment of this application;

[0020] Figure 4 This is a schematic diagram of a distributed balancing process provided in an embodiment of this application;

[0021] Figure 5 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of this application;

[0022] Figure 6 This is a schematic diagram of the structure of a voice recognition device provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but additional steps not included in the drawings may also be present. The above processes can correspond to methods, functions, procedures, subroutines, subroutines, etc.

[0024] The speech recognition method provided in this application can be applied to speech recognition scenarios, such as speech-to-text scenarios. It can achieve more efficient training of speech recognition models and more accurate speech recognition results in less commonly spoken languages ​​(such as Bengali, Arabic, and Egyptian dialects). It aims to utilize unlabeled audio samples, after filtering and distribution balancing, to more easily fit the distribution of correctly labeled audio, as well as labeled audio samples, for semi-supervised training of the speech recognition model, effectively improving the training efficiency and speech recognition performance. Traditional speech recognition schemes typically train the model using all unlabeled sample data or randomly selected sample data. However, large amounts of training data are time-consuming, resulting in low training efficiency. Randomly selected sample data, on the other hand, struggles to guarantee data quality, leading to poor recognition performance. Therefore, this application provides a speech recognition method to address the technical problems of low training efficiency and poor speech recognition performance in existing speech recognition schemes.

[0025] Figure 1 A flowchart of a speech recognition method provided in an embodiment of this application is given. The speech recognition method provided in this embodiment of the application can be executed by a speech recognition device, which can be implemented by hardware and / or software and integrated into a speech recognition device.

[0026] The following description uses a speech recognition device performing a speech recognition method as an example. (Reference) Figure 1 The speech recognition method includes:

[0027] S110: Obtain the voice information to be processed.

[0028] For example, the solution acquires the voice information to be processed for speech recognition. The voice information provided can be audio recorded by a speech recognition device through an audio acquisition module, or it can be obtained from local storage or a user-specified download address. Optionally, the voice information can be a pre-recorded complete audio file, or it can be real-time acquired or uploaded audio stream information.

[0029] S120: Input the speech information to be processed into the trained speech recognition model. The speech recognition model analyzes and processes the speech information to be processed to obtain the recognition result corresponding to the speech information to be processed. The speech recognition model is obtained through semi-supervised training based on unlabeled sample audio and labeled sample audio after filtering and distribution balancing.

[0030] The speech recognition device provided in this solution is pre-configured with a trained speech recognition model. This model is obtained through semi-supervised training based on unlabeled and labeled audio samples after filtering and distribution balancing. The initial unlabeled audio samples can be obtained online, for example, by pulling multiple unlabeled audio samples from an online speech service platform, for use in the semi-supervised training of the speech recognition model. The initial teacher model for the semi-supervised training is obtained through supervised training; that is, the initial teacher model used in the first round of semi-supervised training is obtained through supervised training using labeled audio samples. This supervised-trained initial teacher model serves as the initial teacher model for the first round of semi-supervised training, while subsequent rounds of semi-supervised training will use the speech recognition model obtained in the previous round of semi-supervised training as the teacher model.

[0031] For example, after obtaining the speech information to be recognized, the speech information is input into a speech recognition model for recognition processing. Upon receiving the speech information, the speech recognition model analyzes and processes it, and outputs the recognition result corresponding to the speech information.

[0032] The annotations (labels) for the sample audio can be the corresponding text text. These annotations can be determined manually, for example, by staff filling in the corresponding text text based on the content expressed in the sample audio. After collecting the original unlabeled sample audio, filtering and distribution balancing can be performed. Optionally, the filtering and distribution balancing of the unlabeled sample audio can be based on the decoding results of the unlabeled sample audio using a teacher model that has undergone semi-supervised training on the speech information to be processed (the speech recognition model obtained after semi-supervised training of the teacher model can be used as the teacher model for the next round of semi-supervised training; the initial teacher model can be determined based on the pre-trained base model). The labeled sample audio can be the sample data used to train the initial teacher model.

[0033] For example, the teacher model decodes the decoded text corresponding to unlabeled audio samples. Based on the length of the decoded text, the decoding score (which reflects the probability that the decoded text corresponds to an unlabeled audio sample), and the audio distribution of the decoded text and labeled audio samples, the unlabeled audio samples are filtered and balanced. Optionally, the audio distribution can be represented by the distribution of modeling tokens corresponding to words, characters, and / or text. The unlabeled audio sample distribution after filtering and distribution balancing better fits the correctly labeled audio distribution (e.g., it is closer to the audio distribution of the validation and / or test sets used to train the initial teacher model) compared to the original unlabeled audio sample distribution. The speech recognition model obtained through semi-supervised training based on the filtered and balanced unlabeled audio samples and labeled audio samples is more accurate in audio recognition and does not require labeling a large amount of sample data, effectively improving the training efficiency of the speech recognition model. The validation and test sets include multiple labeled audio samples.

[0034] In one possible embodiment, such as Figure 2 The provided diagram illustrates a semi-supervised training process for a speech recognition model. The semi-supervised training steps for the speech recognition model in this scheme include steps S101-S103:

[0035] S101: Use the teacher model to decode unlabeled sample audio to obtain decoded text and text path scores.

[0036] For example, an initial teacher model is determined. The labeled audio samples provided in this scheme serve as training samples for the teacher model. This teacher model can be trained based on the labeled audio samples. The teacher model analyzes and processes the input speech information and outputs the corresponding decoded text and / or the corresponding text path score. During the analysis and processing of the speech information, multiple text paths (different text paths include different combinations of characters and words) and a text path score for each text path are obtained. The model outputs the text corresponding to the text path with the highest score as the decoded text. This teacher model will participate in the first round of semi-supervised training as the initial base model. The first round of semi-supervised training will iteratively train the teacher model, resulting in a speech recognition model with better speech recognition capabilities (i.e., the teacher model after iterative training). This speech recognition model can then be used as the teacher model for the next round of semi-supervised training, achieving iterative upgrades to the speech recognition model.

[0037] Obtain unlabeled audio samples for training the speech recognition model. Input the unlabeled audio samples into the teacher model in sequence. Use the teacher model to decode each unlabeled audio sample to obtain the decoded text and the text path score corresponding to each unlabeled audio sample.

[0038] In one possible embodiment, the teacher model provided by this solution can be a basic acoustic model with the lowest loss value on the validation set without parameter averaging, or a basic acoustic model with the lowest word error rate or phrase error rate on the test set after parameter averaging. For example, the training data for the basic acoustic model includes a training set, a validation set, and a test set, each containing multiple labeled audio samples. The basic acoustic model is trained using the training set, and validated and tested using the validation and test sets. In this case, the basic acoustic model with the lowest loss value on the validation set without parameter averaging, or the basic acoustic model with the lowest word error rate or phrase error rate on the test set after parameter averaging, can be selected as the teacher model for different scenarios, languages, or dialects. This determines the basic acoustic model that best fits the real audio text, improving the training effect of the speech recognition model and ensuring speech recognition quality.

[0039] S102: Perform first filtering on unlabeled audio samples based on decoded text and text path scores, and perform distribution balancing on unlabeled audio samples based on decoded text.

[0040] For example, the unlabeled audio samples are first filtered based on the decoded text and text path score of each sample. This can be achieved by sorting the unlabeled audio samples according to the text length or the token length corresponding to the decoded text, filtering out some samples in descending order of text length or token length, and / or filtering out some samples in ascending order of text path score. This first filtering process removes some unlabeled audio samples that are prone to inaccurate decoding, reducing the negative impact of the distribution balancing process and improving the training effect of the speech recognition model.

[0041] In one possible embodiment, such as Figure 3 As shown in the schematic diagram of the first filtering process, this scheme performs a first filtering process on unlabeled audio samples based on decoded text and text path scores, including:

[0042] S1021: Convert each decoded text from a word sequence into a modeling token sequence.

[0043] S1022: Calculate the linear regression slope, linear regression intercept, and standard deviation of the linear regression model using the validation set and / or test set corresponding to the teacher model.

[0044] S1023: Calculate the normalized score for each unlabeled audio sample based on the linear regression slope, linear regression intercept, standard deviation, token length of the decoded text, and text path score.

[0045] S1024: Based on the set score threshold and normalized score, perform the first filtering process on the unlabeled sample audio.

[0046] In this model, the decoded text output by the teacher model after decoding unlabeled audio samples is represented as a word sequence. Alternatively, the decoded text can also be represented as a modeling token sequence, where a predetermined correspondence exists between words and tokens (determined based on a predefined speech recognition dictionary). For example, after obtaining the decoded text and text path score output by the teacher model for each unlabeled audio sample, each decoded text is converted from a word sequence into a modeling token sequence.

[0047] Furthermore, a validation set and / or test set are obtained to train the teacher model. The teacher model is used to decode the validation set and / or test set to obtain the decoding score and token length for each data point in the validation set and / or test set. A linear regression model is calculated based on the decoding score and token length of each data point in the validation set and / or test set (e.g., using both the validation set and test set as data for calculating the linear regression model), and the linear regression slope, linear regression intercept, and standard deviation of the linear regression model are determined. Here, the horizontal axis corresponding to the linear regression model represents the token length of the modeled token sequence corresponding to the decoded text, and the vertical axis represents the decoding score corresponding to the decoded text.

[0048] Optionally, when calculating the linear regression model, the sample set corresponding to the training teacher model, or the labeled data participating in the training of the acoustic model, can also be added to the calculation of the linear regression model to ensure the data volume for calculating the linear regression model.

[0049] In one embodiment, the calculation of the linear regression slope, linear regression intercept, and standard deviation of the linear regression model can be performed based on the word sequence of the decoded text. Specifically, the teacher model is used to decode the validation set and / or test set to obtain the decoding score and text length of each data point in the validation set and / or test set. The linear regression model is then calculated based on the decoding score and text length of each data point in the validation set and / or test set, and the linear regression slope, linear regression intercept, and standard deviation of the linear regression model are determined. Here, the horizontal axis corresponding to the linear regression model represents the text length of the decoded text, and the vertical axis represents the decoding score of the decoded text.

[0050] Furthermore, based on the linear regression slope, linear regression intercept, standard deviation, token length of the decoded text corresponding to each unlabeled audio sample, and text path score determined above, a normalized score is calculated for each unlabeled audio sample. Then, based on a set score threshold and the determined normalized score, a first filtering process is performed on each unlabeled audio sample. For example, unlabeled audio samples with a normalized score lower than the set score threshold are filtered out, while those with a normalized score reaching the set score threshold are retained.

[0051] In one possible embodiment, the normalized score provided by this solution can be determined based on the following normalized score calculation formula:

[0052]

[0053] Where s(S, l) is the normalized score, S is the text path score, l is the token length, μ is the linear regression slope, β is the linear regression intercept, and σ is the standard deviation. After determining the linear regression slope, linear regression intercept, and standard deviation, the token length and text path score corresponding to each unlabeled audio sample are substituted into the normalized score calculation formula to determine the normalized score corresponding to each unlabeled audio sample. This scheme accurately calculates the normalized score corresponding to each unlabeled audio sample through the normalized score calculation formula, improving the accuracy and computational efficiency of the normalized score, improving the filtering efficiency and filtering quality of unlabeled audio samples, and ensuring the training effect of the speech recognition model.

[0054] Here, μl+β can be understood as the decoding score based on the token length prediction using a linear regression model. When the text path score of the unlabeled audio sample is greater than the predicted decoding score, the quality of the corresponding unlabeled audio sample is considered to be good, and the corresponding normalized score is positive. Conversely, when the text path score of the unlabeled audio sample is less than the predicted decoding score, the quality of the corresponding unlabeled audio sample is considered to be poor, and the corresponding normalized score is negative.

[0055] After obtaining the normalized scores corresponding to each unlabeled audio sample, a distribution balancing process is performed on the unlabeled audio samples after the first filtering process and the sample data used to train the teacher model (e.g., validation set and / or test set) based on the decoded text. This process ensures that the unlabeled audio samples selected by the first filtering process can better fit the correctly labeled audio samples, and the word / character distribution in their labels is highly consistent with the word / character distribution in the correctly labeled audio samples (e.g., labeled audio samples in the validation set and / or test set), effectively improving the training effect of the speech recognition model.

[0056] This scheme calculates the linear regression slope, intercept, and standard deviation of the linear regression model on the validation and / or test sets. Based on these parameters, it calculates the normalized score for unlabeled audio samples. Then, based on the score threshold and the normalized score, it performs a first filtering process on the unlabeled audio samples. This filtering process removes unlabeled audio samples with a significant negative impact on training performance (low text path score and short text or token length) while retaining unlabeled audio samples with a significant positive impact on training performance (high text path score and short text or token length). This filtering process on unlabeled audio samples using the NST (Noisy Student Training) linear regression strategy improves the training effect of the speech recognition model and enhances speech recognition quality.

[0057] In one possible embodiment, such as Figure 4 As shown in the schematic diagram of the distribution balancing process, this scheme includes the following steps when performing distribution balancing on unlabeled audio samples based on decoded text:

[0058] S1025: Calculate the first relative entropy loss of each unlabeled sample audio with respect to the validation set and / or test set on the token distribution, and add the first relative entropy loss to the first loss set.

[0059] S1026: Based on the first sorting order of the first relative entropy loss in the first loss set, select a first set number of unlabeled sample audios and add them to the unlabeled sample audio set.

[0060] S1027: Copy the unlabeled sample audio set to the first sample audio set, and calculate the second relative entropy loss of the first sample audio set and the validation set and / or test set on the token distribution.

[0061] S1028: Add the unlabeled sample audio that was not selected to the unlabeled sample audio set in sequence to obtain multiple second sample audio sets, calculate the third relative entropy loss of each second sample audio with the validation set and / or test set on the token distribution, and add the difference between the second relative entropy loss and the loss of each third relative entropy loss to the second loss set.

[0062] S1029: Based on the second sorting order of the loss difference in the second loss set, select a second set number of unlabeled sample audios and add them to the unlabeled sample audio set until the number of unlabeled sample audios in the unlabeled sample audio set reaches a set threshold.

[0063] For example, the first relative entropy loss (KL loss value) of each unlabeled sample audio with respect to the validation set and / or test set on the token distribution is calculated, and the first relative entropy loss values ​​corresponding to these unlabeled sample audios are added to the first loss set. In this scheme, the calculation of the relative entropy loss value can be performed after integrating the validation set and the test set.

[0064] Furthermore, the first relative entropy loss values ​​in the first loss set are sorted according to a first sorting order. Based on the first sorting order of the first relative entropy loss values ​​in the first loss set, a first predetermined number of unlabeled audio samples are selected and added to the unlabeled audio sample set. In one embodiment, the first sorting order is an ascending order, that is, retaining the first predetermined number of unlabeled audio samples with the smallest first relative entropy loss value. The first predetermined number can be 1 / 10 to 1 / 5 of the total amount of unlabeled audio sample data after the first filtering process. The unlabeled audio samples added to the unlabeled audio sample set are not selected again in the subsequent distribution balancing process.

[0065] Optionally, after adding a first set number of unlabeled sample audios to the unlabeled sample audio set, it can be determined whether the amount of unlabeled sample audio data in the unlabeled sample audio set has reached a set threshold. If so, the distribution balancing process ends; otherwise, it jumps to S1027 (or executes step S1027 if it is determined that the unlabeled sample audio set is not empty). Specifically, before calculating the first relative entropy loss of each unlabeled sample audio with the validation set and / or test set on the token distribution, it can be determined whether the unlabeled sample audio set is empty. If the unlabeled sample audio set is empty, the first relative entropy loss of each unlabeled sample audio with the validation set and / or test set on the token distribution is calculated (i.e., step S1025 is executed); if the unlabeled sample audio set is not empty, step S1027 is executed.

[0066] Furthermore, when the unlabeled sample audio set is not empty and the amount of unlabeled sample audio data in the unlabeled sample audio set does not reach the set quantity threshold, the current unlabeled sample audio set is copied as the first sample audio set, and the second relative entropy loss degree of the first sample audio set and the validation set and / or test set on the token distribution is calculated.

[0067] Unlabeled audio samples that were not selected are sequentially added to the unlabeled audio sample set, resulting in multiple second audio sample sets. The third relative entropy loss of each second audio sample with respect to the validation set and / or test set on the token distribution is calculated. The difference between the second relative entropy loss and each third relative entropy loss is calculated, and these difference values ​​are added to the second loss set (the first and second loss sets can be the same or different). Optionally, after each calculation of the second relative entropy loss corresponding to a second audio sample and the addition of the corresponding difference value to the second loss set, the unlabeled audio sample added to the current unlabeled audio sample set is removed, and the next unselected unlabeled audio sample is added to the unlabeled audio sample set, resulting in the next second audio sample set. The difference value is then calculated for the next second audio sample set until the second loss set records the difference value corresponding to each unselected unlabeled audio sample added to the unlabeled audio sample set. A larger difference value indicates a stronger ability of the unlabeled audio sample to reduce the loss between the unlabeled audio sample set and the validation set and / or test set, and a greater improvement in the training effect of the speech recognition model.

[0068] Further, the loss differences in the second loss set are sorted according to the second sorting order, and based on the second sorting order of the loss differences in the second loss set, a second set number of unlabeled audio samples are selected and added to the unlabeled audio sample set until the number of unlabeled audio samples in the unlabeled audio sample set reaches a set threshold. That is, after adding the second set number of unlabeled audio samples to the unlabeled audio sample set, it is determined whether the amount of unlabeled audio samples in the unlabeled audio sample set reaches the set threshold. If so, the distribution balancing process ends; otherwise, it jumps to S1027. In one embodiment, the second sorting order is from largest to smallest, that is, the second set number of unlabeled audio samples with the largest loss difference are retained. The second set number can be the same as or different from the first set number. Optionally, the second set number can be 1 / 10 to 1 / 5 of the total amount of unlabeled audio sample data after the first filtering process. The unlabeled audio samples added to the unlabeled audio sample set are not selected again in the subsequent distribution balancing process.

[0069] This scheme balances the distribution of unlabeled audio samples based on their relative entropy loss on the token distribution of the validation and / or test sets. This makes the unlabeled audio samples fit the validation and / or test sets better, ensuring that the final selected unlabeled audio samples can better fit the validation and / or test sets. The word / character distribution in the decoded text corresponding to the unlabeled audio samples is highly consistent with the word / character distribution in the validation / test sets, effectively improving the training effect of the speech recognition model.

[0070] S103: Semi-supervised training of the teacher model is performed using unlabeled and labeled audio samples to obtain a speech recognition model.

[0071] For example, using the unlabeled audio samples obtained after the above filtering and distribution balancing processes, and the labeled audio samples used when training the initial teacher model, the teacher model is semi-supervised to obtain a speech recognition model.

[0072] Optionally, after completing one round of semi-supervised training of the speech recognition model, this model can be used as a new teacher model for the next round of semi-supervised training, utilizing both unlabeled and labeled audio samples. This can involve continuing to use previously collected unlabeled audio samples, collecting new unlabeled audio samples, or adding new unlabeled audio samples to the existing ones. Before each round of semi-supervised training, the unlabeled audio samples need to be updated. For example, the speech recognition model from the previous round (i.e., the teacher model for this round) can be used to decode the unlabeled audio samples to obtain better decoded text and text path scores. For newly collected or added unlabeled audio samples, after filtering and distribution balancing, the teacher model is used in conjunction with labeled audio samples for the next round of semi-supervised training, until the set number of rounds of semi-supervised training is completed, resulting in the final speech recognition model. This scheme utilizes a teacher model to decode unlabeled audio samples, and then filters and balances the unlabeled audio samples based on the decoding results. This filters out unlabeled audio samples that have a significant negative impact on semi-supervised training or that differ greatly from the distribution of the sample data. The teacher model is then semi-supervised trained using both unlabeled and labeled audio samples to obtain a speech recognition model, effectively improving the training effect of the speech recognition model.

[0073] In one possible embodiment, this scheme sets a predetermined ratio for the unlabeled and labeled audio samples used in the semi-supervised training of the speech recognition model (where the number of unlabeled audio samples is greater than the number of labeled audio samples, for example, a ratio of 2:1 or 3:1). The number of rounds of semi-supervised training of the speech recognition model based on the unlabeled and labeled audio samples after filtering and distribution balancing is set to a predetermined number (e.g., 3-5 rounds). By setting the ratio of unlabeled and labeled audio samples in the semi-supervised training and the number of rounds of semi-supervised training, the semi-supervised training efficiency of the speech recognition model is effectively improved while ensuring the effectiveness of the semi-supervised training.

[0074] In one possible embodiment, after using the teacher model to decode the unlabeled sample audio and obtain the decoded text and text path score, this scheme can also perform a second filtering process on the unlabeled sample audio based on the text path score, the text length of the decoded text, a first score threshold, and a first length threshold.

[0075] For example, after decoding unlabeled audio samples using a teacher model, some long audio decoding results may be inaccurate. A second filtering process can be performed on the unlabeled audio samples based on a pre-set first score threshold and a first length threshold. Then, the second-filtered unlabeled audio samples are subjected to the first filtering process and distribution balancing. For instance, based on the first score threshold and the text path score of each unlabeled audio sample, unlabeled audio samples with text path scores lower than the first score threshold are identified. Based on the first length threshold and the text length of the decoded text of each unlabeled audio sample, unlabeled audio samples with text lengths greater than the first length threshold are identified. The intersection of the two identified unlabeled audio samples is then filtered out, achieving the second filtering process for the unlabeled audio samples. This scheme reduces the impact of poor-quality unlabeled audio samples on the subsequent first filtering and distribution balancing processes in the NST strategy by filtering out unlabeled audio samples with long decoded texts and low text path scores based on the first score threshold and the first length threshold, thus ensuring the training effect and accuracy of the speech recognition model.

[0076] In one possible embodiment, after performing a first filtering process on the unlabeled sample audio based on the decoded text and the text path score, and performing a distribution balancing process on the unlabeled sample audio based on the decoded text, a third filtering process can be performed on the unlabeled sample audio after the filtering and distribution balancing processes based on the text path score, the text length of the decoded text, a second score threshold, and a second length threshold.

[0077] For example, after performing the first filtering and distribution balancing on the unlabeled audio samples, a third filtering can be performed based on a second score threshold and a second length threshold. For instance, based on the second score threshold and the text path score of each unlabeled audio sample, unlabeled audio samples with text path scores lower than the second score threshold are identified. Then, based on the second length threshold and the text length of the decoded text of each unlabeled audio sample, unlabeled audio samples with text lengths greater than the second length threshold are identified. The intersection of the two identified unlabeled audio samples is then filtered out, achieving a three-stage filtering process for the unlabeled audio samples. Optionally, the second score threshold and the second length threshold can be adjusted based on the first score threshold and the first length threshold. For example, the second score threshold can be greater than the first score threshold, and the second length threshold can be less than the first length threshold. This solution reduces long audio samples with poor decoding performance in the unlabeled audio samples by performing a third filtering on the filtered and distribution balancing processed unlabeled audio samples, thereby improving the training effect of the speech recognition model and the accuracy of speech recognition.

[0078] In one possible embodiment, audio enhancement processing can be applied to both the unlabeled and labeled audio samples after filtering and distribution balancing to improve the robustness of the speech recognition model. For example, spectral enhancement processing can be applied to the unlabeled audio samples, and noise augmentation and / or spectral enhancement processing can be applied to the labeled audio samples.

[0079] When applying the speech recognition method provided in this solution to speech recognition of less commonly spoken languages ​​(such as Bengali and Egyptian dialects of Arabic), the annotation process for these languages ​​is more difficult, the manual annotation process is complex, and the annotation quality is hard to control. Traditional semi-supervised training often uses unlabeled samples with long decoded text and low decoding scores (path scores). These unlabeled samples can easily lead to a decrease in the recognition performance of the speech recognition model after iteration. This solution improves the fit of the unlabeled sample audio to the correctly labeled samples (such as the validation set and / or test set) by performing multiple filtering and distribution balancing processes, effectively reducing the word error rate (WER) for speech recognition of less commonly spoken languages ​​and improving the speech recognition effect.

[0080] The above describes a process where a speech recognition model is used to analyze and process the speech information to obtain the recognition result. This speech recognition model is obtained through semi-supervised training on unlabeled audio samples after filtering and distribution balancing, as well as labeled audio samples. The distribution of unlabeled audio samples after filtering and distribution balancing more easily fits the distribution of correctly labeled audio samples. This fully utilizes unlabeled audio samples for semi-supervised training, reducing the manpower and time costs of manual annotation and minimizing the impact of a large number of unlabeled audio samples on training efficiency and the accuracy of the speech recognition model, effectively improving the training efficiency and speech recognition performance. Simultaneously, a teacher model is used to decode the unlabeled audio samples, and based on the decoding results, filtering and distribution balancing are applied to remove unlabeled audio samples that have a significant negative impact on semi-supervised training or that differ greatly from the sample data distribution, effectively improving the training effect of the speech recognition model.

[0081] Figure 5 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of this application. (Reference) Figure 3 The voice recognition device includes a voice acquisition module 51 and a voice recognition module 52.

[0082] The speech acquisition module 51 is configured to acquire speech information to be processed; the speech recognition module 52 is configured to input the speech information to be processed into the trained speech recognition model, and analyze and process the speech information to be processed through the speech recognition model to obtain the recognition result corresponding to the speech information to be processed. The speech recognition model is obtained through semi-supervised training based on unlabeled sample audio and labeled sample audio after filtering and distribution balancing. The initial teacher model for semi-supervised training is obtained through supervised training.

[0083] The above describes how a speech recognition model is used to analyze and process the speech information to obtain the recognition result. This speech recognition model is obtained through semi-supervised training based on unlabeled audio samples after filtering and distribution balancing, as well as labeled audio samples. The distribution of unlabeled audio samples after filtering and distribution balancing is more likely to fit the distribution of correctly labeled audio samples. By making full use of unlabeled audio samples for semi-supervised training of the speech recognition model, the manpower and time cost of manual annotation is reduced, and the impact of a large number of unlabeled audio samples on training efficiency and speech recognition accuracy is reduced, effectively improving the training efficiency and speech recognition effect of the speech recognition model.

[0084] In one possible embodiment, the speech recognition device further includes a model training module, which is configured to perform semi-supervised training on the speech recognition model to obtain the speech recognition model. The semi-supervised training steps of the speech recognition model in the model training module are configured as follows:

[0085] The teacher model is used to decode unlabeled audio samples to obtain decoded text and text path scores;

[0086] The unlabeled audio samples are first filtered based on the decoded text and text path scores, and then the distribution of the unlabeled audio samples is balanced based on the decoded text.

[0087] The teacher model is semi-supervised and trained using unlabeled and labeled audio samples to obtain a speech recognition model. The labeled audio samples are used as training samples for the teacher model.

[0088] In one possible embodiment, the model training module is configured to perform a first filtering process on unlabeled sample audio based on the decoded text and text path scores as follows:

[0089] Each decoded text is converted from a word sequence into a modeling token sequence;

[0090] Calculate the linear regression slope, linear regression intercept, and standard deviation of the linear regression model using the validation set and / or test set corresponding to the teacher model;

[0091] Based on the linear regression slope, linear regression intercept, standard deviation, token length of the decoded text, and text path score, the normalized score corresponding to each unlabeled audio sample is calculated.

[0092] Based on the set score threshold and normalized score, the unlabeled audio samples are subjected to the first filtering process.

[0093] In one possible implementation, the normalized score is determined based on the following formula:

[0094]

[0095] Where s(S, l) is the normalized score, S is the text path score, l is the token length, μ is the linear regression slope, β is the linear regression intercept, and σ is the standard deviation.

[0096] In one possible embodiment, the model training module is configured to perform distribution balancing on unlabeled sample audio based on the decoded text as follows:

[0097] Calculate the first relative entropy loss of each unlabeled sample audio with respect to the validation set and / or test set on the token distribution, and add the first relative entropy loss to the first loss set;

[0098] Based on the first sorting order of the first relative entropy loss in the first loss set, a first set number of unlabeled sample audios are selected and added to the unlabeled sample audio set;

[0099] Copy the unlabeled sample audio set to the first sample audio set, and calculate the second relative entropy loss of the first sample audio set and the validation set and / or test set on the token distribution;

[0100] Unlabeled sample audio that was not selected is added to the unlabeled sample audio set in turn to obtain multiple second sample audio sets. The third relative entropy loss of each second sample audio with the validation set and / or test set on the token distribution is calculated, and the difference between the second relative entropy loss and the loss of each third relative entropy loss is added to the second loss set.

[0101] Based on the second sorting order of the loss differences in the second loss set, a second set number of unlabeled audio samples are selected and added to the unlabeled audio sample set until the number of unlabeled audio samples in the unlabeled audio sample set reaches a set threshold.

[0102] In one possible embodiment, after the model training module uses the teacher model to decode the unlabeled sample audio and obtain the decoded text and text path score, it is further configured as follows:

[0103] Based on the text path score, the text length of the decoded text, the first score threshold, and the first length threshold, a second filtering process is performed on the unlabeled audio samples.

[0104] In one possible embodiment, after performing a first filtering process on the unlabeled audio samples based on the decoded text and text path scores, and performing a distribution balancing process on the unlabeled audio samples based on the decoded text, the model training module is further configured as follows:

[0105] A third filtering process is performed on the unlabeled audio samples after filtering and distribution balancing, based on the text path score, the text length of the decoded text, the second score threshold, and the second length threshold.

[0106] In one possible embodiment, the teacher model is a basic acoustic model that has the lowest loss value on the validation set and has not undergone parameter averaging, or a basic acoustic model that has the lowest word error rate or phrase error rate on the test set and has undergone parameter averaging.

[0107] In one possible embodiment, the ratio of unlabeled sample audio to labeled sample audio for semi-supervised training of the speech recognition model is a set ratio, and the number of rounds of semi-supervised training of the speech recognition model based on the unlabeled sample audio and labeled sample audio after filtering and distribution balancing is a set number of rounds.

[0108] It is worth noting that in the above-described embodiments of the voice recognition device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of the present invention.

[0109] This application also provides a speech recognition device that can integrate the speech recognition apparatus provided in this application. Figure 6 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of this application. (Reference) Figure 6 The speech recognition device includes: an input device 63, an output device 64, a memory 62, and one or more processors 61; the memory 62 is used to store one or more programs; when one or more programs are executed by one or more processors 61, the one or more processors 61 implement the speech recognition method provided in the above embodiments. The speech recognition device, apparatus, and computer provided above can be used to execute the speech recognition method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0110] This application also provides a non-volatile storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the speech recognition method provided in the above embodiments. Of course, the computer-executable instructions provided in this application are not limited to the speech recognition method provided above, but can also perform related operations in the speech recognition method provided in any embodiment of this application. The speech recognition device, apparatus, and storage medium provided in the above embodiments can execute the speech recognition method provided in any embodiment of this application. Technical details not described in detail in the above embodiments can be found in the speech recognition method provided in any embodiment of this application.

[0111] Based on the above embodiments, this application also provides a computer program product. The technical solution of this application, in essence or in other words, the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer program product is stored in a storage medium and includes several instructions to cause a computer device, mobile terminal, or processor therein to execute all or part of the steps of the speech recognition method provided in the various embodiments of this application.

Claims

1. A voice recognition method, characterized by, The method comprises: obtaining to-be-processed voice information; inputting the to-be-processed voice information into a trained voice recognition model, analyzing and processing the to-be-processed voice information through the voice recognition model to obtain a recognition result corresponding to the to-be-processed voice information, wherein the voice recognition model is obtained through semi-supervised training based on filtered and distribution-balanced unannotated sample audio and annotated sample audio, and an initial teacher model of the semi-supervised training is obtained through supervised training; the semi-supervised training steps of the voice recognition model comprise: decoding the unannotated sample audio using a teacher model to obtain decoding text and text path scores; performing first filtering on the unannotated sample audio based on the decoding text and the text path scores, and performing distribution balancing on the unannotated sample audio based on the decoding text; and performing semi-supervised training on the teacher model using the unannotated sample audio and the annotated sample audio to obtain a voice recognition model, wherein the annotated sample audio is a training sample of the teacher model; wherein the distribution balancing of the unannotated sample audio based on the decoding text comprises: calculating a first relative entropy loss degree of each unannotated sample audio and a validation set and / or a test set in token distribution, and adding the first relative entropy loss degree to a first loss set; selecting a first set number of unannotated sample audios in a first sorting order of the first relative entropy loss in the first loss set and adding them to an unannotated sample audio set; copying the unannotated sample audio set as a first sample audio set, calculating a second relative entropy loss degree of the first sample audio set and the validation set and / or the test set in token distribution; adding the unselected unannotated sample audios to the unannotated sample audio set in turn to obtain a plurality of second sample audio sets, calculating a third relative entropy loss degree of each second sample audio and the validation set and / or the test set in token distribution, and adding a loss degree difference between the second relative entropy loss degree and each third relative entropy loss degree to a second loss set; selecting a second set number of unannotated sample audios in a second sorting order of the loss degree difference in the second loss set and adding them to the unannotated sample audio set until the number of unannotated sample audios in the unannotated sample audio set reaches a set number threshold.

2. The voice recognition method of claim 1, wherein, the first filtering of the unannotated sample audio based on the decoding text and the text path scores comprises: converting each decoding text from a word sequence into a modeling token sequence; calculating a linear regression slope, a linear regression intercept, and a standard deviation of a linear regression model using a validation set and / or a test set corresponding to the teacher model; calculating a normalized score corresponding to each unannotated sample audio based on the linear regression slope, the linear regression intercept, the standard deviation, the token length of the decoding text, and the text path score. The unlabelled sample audio is first filtered based on a set score threshold and the normalized score.

3. The voice recognition method of claim 2, wherein, The normalized score is determined based on the following formula: wherein, is the normalized score, is the text path score, is the token length, is the linear regression slope, is the linear regression intercept, is the standard deviation.

4. The voice recognition method of claim 1, wherein, After the decoding processing of the unlabelled sample audio by the teacher model to obtain the decoded text and the text path score, the method further includes: The unlabelled sample audio is second filtered based on the text path score, the text length of the decoded text, a first score threshold and a first length threshold.

5. The voice recognition method of claim 1, wherein, After the first filtering of the unlabelled sample audio based on the decoded text and the text path score, and the distribution balancing of the unlabelled sample audio based on the decoded text, the method further includes: The unlabelled sample audio after the filtering and the distribution balancing is third filtered based on the text path score, the text length of the decoded text, a second score threshold and a second length threshold.

6. The voice recognition method of claim 1, wherein, The teacher model is a base acoustic model with the lowest loss value on a validation set and without parameter averaging processing, or a base acoustic model with the lowest word error rate or character error rate on a test set and with parameter averaging processing.

7. The voice recognition method of claim 1, wherein, The proportion of the unlabelled sample audio and the labelled sample audio for the semi-supervised training of the speech recognition model is a set proportion, and the number of rounds of the semi-supervised training of the speech recognition model based on the unlabelled sample audio and the labelled sample audio after the filtering and the distribution balancing is a set number of rounds.

8. A speech recognition apparatus characterized by comprising: The speech recognition model includes a speech obtaining module, a speech recognition module and a model training module, wherein: The speech obtaining module is configured to obtain to-be-processed speech information; The speech recognition module is configured to input the to-be-processed speech information into a trained speech recognition model, analyze and process the to-be-processed speech information by the speech recognition model to obtain a recognition result corresponding to the to-be-processed speech information, and obtain the recognition result corresponding to the to-be-processed speech information by the speech recognition model based on the unlabelled sample audio and the labelled sample audio after the filtering and the distribution balancing, wherein the initial teacher model for the semi-supervised training of the speech recognition model is obtained by supervised training; The model training module is configured to semi-supervise and train the speech recognition model to obtain the speech recognition model, and the semi-supervised training steps of the speech recognition model include: decoding processing of unlabelled sample audio by a teacher model to obtain decoded text and a text path score; first filtering of the unlabelled sample audio based on the decoded text and the text path score, and distribution balancing of the unlabelled sample audio based on the decoded text; semi-supervised training of the teacher model by the unlabelled sample audio and the labelled sample audio to obtain the speech recognition model, wherein the labelled sample audio is a training sample of the teacher model; In the distribution balancing of the unlabelled sample audio based on the decoded text, the model training module is configured to: Calculate a first relative entropy loss degree of each of the unlabelled sample audio and the validation set and / or the test set on token distribution, and add the first relative entropy loss degree to a first loss set; selecting, based on a first ranking order of the first relative entropy loss in the first loss set, a first preset number of the unlabeled sample audios to add to the unlabeled sample audio set; copying the unlabeled sample audio set as a first sample audio set, and calculating a second relative entropy loss degree of the first sample audio set and the validation set and / or the test set in token distribution; adding the unlabeled sample audios not selected to the unlabeled sample audio set in turn to obtain a plurality of second sample audio sets, calculating a third relative entropy loss degree of each of the second sample audios and the validation set and / or the test set in token distribution, and adding a loss degree difference value of the second relative entropy loss degree and each of the third relative entropy loss degrees to a second loss set; selecting, based on a second ranking order of the loss degree difference value in the second loss set, a second preset number of the unlabeled sample audios to add to the unlabeled sample audio set until a preset number threshold of the unlabeled sample audios in the unlabeled sample audio set is reached.

9. A speech recognition device, characterized by comprising: a memory and one or more processors; the memory, configured to store one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the speech recognition method according to any one of claims 1-7.

10. A non-volatile storage medium for storing computer-executable instructions, characterized in that, the computer executable instructions, when executed by a computer processor, are used to perform the speech recognition method according to any one of claims 1-7.

11. A computer program product comprising a computer program, characterized in that, the computer program, when executed by a processor, implements the speech recognition method according to any one of claims 1-7.

Images