A speech recognition model training method, speech recognition method and related equipment

Abstract

The present invention provides a speech recognition model training method, a speech recognition method and related equipment, the training method comprising: using a task set to perform a first-stage training on a constructed speech recognition model to obtain a speech recognition model after the first-stage training, the task set includes tasks corresponding to a plurality of first categories respectively, in the process of performing the first-stage training, by self-learning the learning rate suitable for the tasks in the task set to control the direction of updating the model parameters, and updating the model parameters by calculating the first-order gradient; using the training speech of the second category, the speech recognition model after the first-stage training is trained in the second stage. The speech recognition model training method provided by the present invention can be used to train a speech recognition model that has a good recognition effect on the speech of the second category and has high stability, and the speech recognition model is used to recognize the speech to be recognized in the second category, and a recognition result with a high accuracy rate can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech recognition model training method, a speech recognition method and related equipment. Background Art

[0002] Speech recognition is a technology that uses machines to automatically convert human speech content into text. In recent years, with the widespread application of deep neural networks, speech recognition technology has made great progress. Most of the current speech recognition solutions are based on speech recognition models, that is, a speech recognition model that can recognize speech of a certain category (such as English training speech) is trained in advance, and then the trained speech recognition model is used to recognize the speech to be recognized in the category.

[0003] It is understandable that the effect of speech recognition depends on the performance of the speech recognition model. If you want to obtain a speech recognition model that can accurately recognize a certain category of speech (such as English speech), you need to obtain enough speech data of this category. However, not all categories have enough speech data (for example, there is a lack of speech data of small languages). If there is insufficient speech data in a certain category, the above-mentioned solution will make it impossible to train a speech recognition model with better performance, which will in turn lead to the inability to obtain accurate recognition results when recognizing the speech to be recognized in this category. Summary of the invention

[0004] In view of this, the present invention provides a speech recognition model training method, a speech recognition method and related devices, which are used to solve the problem that when there is insufficient speech data of a certain category, the existing solution cannot be used to train a speech recognition model with better performance, which leads to the problem that accurate recognition results cannot be obtained when recognizing the speech to be recognized in this category. The technical solution is as follows:

[0005] A speech recognition model training method, comprising:

[0006] The constructed speech recognition model is trained in the first stage using the task set to obtain the speech recognition model after the first stage training; wherein the task set includes tasks corresponding to a plurality of first categories respectively, the tasks are constructed according to the training speech corresponding to the first category, the first category is a category in which the total duration of speech resources is greater than or equal to a first preset duration threshold among the set multiple categories, and in the process of training the speech recognition model in the first stage, the model parameter update direction is controlled by self-learning the learning rate suitable for the tasks in the task set, and the model parameters are updated by calculating the first-order gradient;

[0007] The speech recognition model trained in the first stage is trained in the second stage using the training speech of the second category; wherein the second category is a category among the set multiple categories, the total duration of the speech resources of which is less than or equal to the second preset duration threshold.

[0008] Optionally, constructing tasks corresponding to the plurality of first categories respectively includes:

[0009] For each first category:

[0010] Obtaining the training speech of the first category;

[0011] Extracting acoustic features from the training speech of the first category;

[0012] The acoustic features extracted from the training speech of the first category are used as the tasks corresponding to the first category.

[0013] Optionally, the first phase of training the constructed speech recognition model using the task set includes:

[0014] Acquire a first number of tasks from the total task set to form a current task subset;

[0015] Based on the current task subset, the speech recognition model, and the model parameters and learning rate corresponding to the previous task subset, determine the first-order gradient for the current task subset; wherein, if the current task subset is the first task subset, the model parameters and learning rate corresponding to the previous task subset use the initial model parameters and initial learning rate;

[0016] Based on the model parameters and learning rate corresponding to the previous task subset and the first-order gradient for the current task subset, determine the model parameters and learning rate corresponding to the current task subset, and update the model parameters of the speech recognition model to the model parameters corresponding to the current task subset;

[0017] The step of obtaining a first number of tasks from the total task set to form a current task subset and subsequent steps are executed until a training end condition is met.

[0018] Optionally, determining the first-order gradient for the current task subset based on the current task subset, the speech recognition model, and the model parameters and learning rate corresponding to the previous task subset includes:

[0019] Get an unprocessed task from the current task subset as the current task;

[0020] Processing the current task using the speech recognition model, and determining the prediction loss of the speech recognition model on the current task based on the processing result of the current task and the label information corresponding to the current task;

[0021] Calculate the first-order gradient according to the prediction loss of the speech recognition model on the current task, and use the calculated first-order gradient as the first-order gradient for the current task;

[0022] Determine the parameters for the current task according to the first-order gradient for the current task, the parameters for the previous task, and the learning rate corresponding to the previous task subset, wherein the speech recognition model processes the next task based on the parameters for the current task, wherein if the current task is the first task, the parameters for the previous task adopt the model parameters corresponding to the previous task subset;

[0023] The first-order gradient for the last processed task is determined as the first-order gradient for the current task subset.

[0024] Optionally, the using the speech recognition model to process the current task, and determining the prediction loss of the speech recognition model on the current task based on the processing result of the current task and the label information corresponding to the current task, includes:

[0025] Acquire a second number of data points from the current task to form a data point set, wherein each data point in the data point set corresponds to label information;

[0026] Inputting the data point set into a speech recognition model for processing to obtain a processing result output by the speech recognition model;

[0027] The prediction loss of the speech recognition model on the current task is calculated according to the processing result output by the speech recognition model and the label information corresponding to each data point in the data point set.

[0028] Optionally, after determining the parameters for the current task, the method further includes:

[0029] Determine whether the number of iterations on the current task has reached the preset number of iterations;

[0030] If yes, then the step of obtaining an unprocessed task from the current task subset as the current task and subsequent steps are executed; if no, then the step of obtaining a second number of data points from the current task to form a data point set and subsequent steps are executed.

[0031] Optionally, the second-stage training of the speech recognition model trained in the first stage using the training speech of the second category includes:

[0032] Using one or more speech amplification methods to amplify the second category of training speech to obtain amplified speech;

[0033] The speech recognition model trained in the first stage is trained in the second stage using the training speech of the second category and the amplified speech.

[0034] A speech recognition method, comprising:

[0035] Acquire the target speech of the second category;

[0036] Recognize the target speech based on the pre-trained speech recognition model to obtain a recognition result of the target speech;

[0037] Wherein, the speech recognition model is trained using any of the speech recognition model training methods described above.

[0038] A speech recognition model training device, comprising: a first training module and a second training module;

[0039] The first training module is used to perform the first stage training on the constructed speech recognition model using the task set to obtain the speech recognition model after the first stage training; wherein the task set includes tasks corresponding to multiple first categories respectively, and the tasks are constructed according to the training speech corresponding to the first category, and the first category is a category with a total duration of speech resources greater than or equal to a first preset duration threshold among the set multiple categories. In the process of performing the first stage training on the speech recognition model, the model parameter update direction is controlled by self-learning the learning rate suitable for the tasks in the task set, and the model parameters are updated by calculating the first-order gradient;

[0040] The second training module is used to perform second-stage training on the speech recognition model trained in the first stage using training speech of the second category; wherein the second category is a category among the multiple categories whose total duration of speech resources is less than or equal to a second preset duration threshold.

[0041] A data processing device, comprising: a memory and a processor;

[0042] The memory is used to store programs;

[0043] The processor is used to execute the program to implement the various steps of the speech recognition model training method described in any one of the above items, and / or to implement the various steps of the above-mentioned speech recognition method.

[0044] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, it implements each step of the speech recognition model training method described in any one of the above items, and / or implements each step of the above-mentioned speech recognition method.

[0045] The present invention provides a speech recognition model training method, firstly, a task set is used to perform a first-stage training on a constructed speech recognition model to obtain a speech recognition model after the first-stage training, and then a second-category training speech is used to perform a second-stage training on the speech recognition model after the first-stage training. The speech recognition model training method provided by the present invention adopts a task set constructed based on a plurality of first-category training speech with rich speech resources and a training speech of a low-resource speech category, and can be trained to obtain a speech recognition model that has a good recognition effect and good stability for speech of the low-resource speech category. In addition, in the process of the first-stage training of the speech recognition model, the present invention controls the updating direction of the model parameters by self-learning the learning rate suitable for the tasks in the task set, and updates the model parameters by calculating the first-order gradient. These two strategies reduce the computing time and resources consumed by the model training, reduce the time consumption of manually adjusting the learning rate, accelerate the model convergence speed, and reduce the time consumed by the model training. Based on the speech recognition model training method provided by the present invention, the present invention also provides a speech recognition method. Since the speech recognition method adopts the speech recognition model trained by the speech recognition model training method provided by the present invention to recognize the target speech, a more accurate speech recognition result can be finally obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0047] Figure 1 A flowchart of a speech recognition model training method provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of a process for performing the first phase of training on a constructed speech recognition model using a task set provided in an embodiment of the present invention;

[0049] Figure 3 A flow chart of using a second category of training speech to perform a second stage of training on a speech recognition model trained in the first stage according to an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of the structure of a speech recognition model training device provided by an embodiment of the present invention;

[0051] Figure 5 A schematic diagram of the structure of a data processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] At present, the recognition scheme for low-resource speech (such as speech in low-resource languages) is to use the training speech of the low-resource category to train a speech recognition model, and use the trained speech recognition model to recognize the speech to be recognized in the low-resource category. Since there are few training speech in the low-resource category, the performance of the speech recognition model trained by using the training speech of the low-resource category is very poor.

[0055] In order to obtain a speech recognition model with better recognition effect for low-resource speech, the inventor of this case conducted research, and the initial idea was:

[0056] The speech recognition model is trained in the first stage using training speech of multiple resource categories (categories with rich speech resources, such as language categories with rich speech resources), and the speech recognition model trained in the first stage is trained in the second stage using training speech of low resource categories. When the speech recognition model is trained in the first stage, the training speech of multiple resource categories is used to construct a total task set, and the speech recognition model is trained in the first stage using the total task set. When the speech recognition model is trained in the first stage using the total task set, several tasks are sampled from the total task set to form a task subset T, and the task subset T is used for training. The training process includes the following inner loop and outer loop:

[0057] Inner loop: for each task T in T i Extract k data points separately to prepare support sets and queryset Then in the support set Apply the speech recognition model f affected by the parameter θ θ Perform learning, calculate loss and gradient, and use stochastic gradient descent SGD to optimize the loss Minimize and get each task T i Corresponding parameter θ' i , Among them, α is the inner loop learning rate, which is a numerical value and needs to be set or adjusted manually.

[0058] Outer loop: In the query set The accumulated meta-gradient is used above Perform meta-optimization on the Adam optimizer so that The loss in is minimized and the model parameters θ are updated as follows:

[0059] Among them, β is the outer loop learning rate, which is a setting parameter.

[0060] The inventors of this case studied the above ideas and found that due to the existence of a gradient nested relationship in the above process, second-order partial derivatives need to be calculated during model training. The calculation of second-order partial derivatives consumes a lot of time and computing resources, resulting in slow model convergence and sometimes even the possibility of unstable training.

[0061] In view of the defects of the above ideas, the inventors continued their research and finally proposed a speech recognition model training method. Through this speech recognition model training method, a speech recognition model can be trained to have good recognition effect on low-resource category speech and is relatively stable. This method perfectly overcomes the defects of the above ideas.

[0062] The speech recognition model training method provided by the present invention can be applied to electronic devices with data processing capabilities. The electronic device can be a terminal used by the user side, such as a smart phone, PC, laptop computer, tablet computer, PDA, etc. The electronic device can also be a server. The server can be a single server or a server cluster composed of multiple servers, or a cloud computing server center. The server can include a processor, a memory, a network interface, etc.

[0063] The above-mentioned terminal or server can be trained according to the speech recognition model training method provided by the present invention to obtain a speech recognition model with better performance. Those skilled in the art should understand that the above-mentioned terminal and server are only examples, and other existing or future terminals or servers that may appear should also be included in the scope of protection of the present invention if they are applicable to the present invention, and are included here by reference.

[0064] Next, the speech recognition model training method provided by the present invention is introduced through the following embodiments.

[0065] First embodiment

[0066] See also Figure 1, which shows a flow chart of a speech recognition model training method provided by an embodiment of the present invention, and may include:

[0067] Step S101: Perform the first stage training on the constructed speech recognition model using the task set to obtain the speech recognition model after the first stage training.

[0068] Optionally, the constructed speech recognition model can be a speech recognition model of a Transformer framework. Of course, this embodiment is not limited to this. The constructed speech recognition model can also be a speech recognition model of other frameworks, such as a speech recognition model of a Conformer framework. This embodiment does not limit the framework of the speech recognition model.

[0069] The task set in this embodiment includes multiple tasks corresponding to the first category respectively, and each task in the task set is constructed according to the training speech corresponding to the first category.

[0070] It should be noted that the first category is a category among the set multiple categories, in which the total duration of voice resources is greater than a first preset duration threshold (such as 1000 hours), that is, the first category is a category with rich voice resources.

[0071] Exemplarily, the set multiple categories can be set multiple language categories, then the multiple first categories are multiple language categories with rich voice resources, the task set includes tasks corresponding to the multiple language categories with rich voice resources, each task in the task set is constructed according to the voice data of the corresponding language category, assuming that the multiple language categories with rich voice resources include English, German, and French, then the task set includes tasks corresponding to English, German, and French, respectively, wherein the tasks corresponding to English are constructed according to the training voice of English, the tasks corresponding to German are constructed according to the training voice of German, and the tasks corresponding to French are constructed according to the training voice of French.

[0072] Exemplarily, the multiple categories that are set may be multiple accent categories under the same language category, then the multiple first categories are multiple accent categories with rich voice resources, the task set includes tasks corresponding to the multiple accent categories with rich voice resources, each task in the task set is constructed according to the voice data of the corresponding accent category, assuming that the multiple accent categories with rich voice resources include accent A, accent B, and accent C under a certain language category, then the task set includes tasks corresponding to accent A, accent B, and accent C, respectively, wherein the task corresponding to accent A is constructed according to the training speech of accent A, the task corresponding to accent B is constructed according to the training speech of accent B, and the task corresponding to accent C is constructed according to the training speech of accent C.

[0073] Specifically, the process of constructing a task corresponding to a first category according to a training speech of the first category may include: after obtaining the training speech of the first category, extracting acoustic features from the training speech of the first category, and using the acoustic features extracted from the training speech of the first category as the task corresponding to the first category. Exemplarily, the acoustic features extracted from the training speech of the first category may be Fbank 40-dimensional acoustic features.

[0074] In order to speed up the convergence of the model and reduce the training time of the first stage, the present invention introduces a learning rate self-learning strategy and a first-order gradient calculation strategy in the process of the first stage training of the speech recognition model, that is, the direction of model parameter update is controlled by self-learning the learning rate suitable for the tasks in the total task set, and the model parameters are updated by calculating the first-order gradient.

[0075] Step S102: using the second category of training speech, the speech recognition model trained in the first stage is trained in the second stage.

[0076] The second category is a category among the set multiple categories, in which the total duration of voice resources is less than or equal to the second preset duration threshold (for example, 50 hours), that is, the second category is a category with fewer voice resources. It should be noted that the second preset duration threshold is less than the first preset duration threshold.

[0077] Exemplarily, the set multiple categories are set multiple language categories, and the second category is a language category with fewer voice resources, that is, a low-resource language category.

[0078] Exemplarily, the multiple categories set are multiple accent categories under the same language category, and the second category is an accent category with fewer voice resources, that is, a low-resource accent category.

[0079] The embodiment of the present invention provides a speech recognition model training method, firstly, the constructed speech recognition model is trained in the first stage using the task set to obtain the speech recognition model after the first stage training, and then the speech recognition model after the first stage training is trained in the second stage using the training speech of the second category. The speech recognition model training method provided by the embodiment of the present invention adopts the task set constructed based on multiple first-category training speech with rich speech resources and the training speech of the low-resource speech category, and can be trained to obtain a speech recognition model that has good recognition effect and good stability for the speech of the low-resource speech category. In addition, in the process of the first stage training of the speech recognition model in the embodiment of the present invention, the direction of updating the model parameters is controlled by self-learning the learning rate suitable for the task in the task set, and the model parameters are updated by calculating the first-order gradient. These two strategies reduce the computing time and resources consumed by model training, reduce the time consumption of manual adjustment of the learning rate, accelerate the model convergence speed, and reduce the time consumed by model training.

[0080] Second embodiment

[0081] This embodiment introduces "Step S101: performing the first stage training on the constructed speech recognition model using the total task set to obtain the speech recognition model after the first stage training" in the above embodiment.

[0082] See also Figure 2 , showing a flow chart of the first stage of training the constructed speech recognition model using the task set, which may include:

[0083] Step S201: Acquire a first number of tasks from the total task set to form a current task subset.

[0084] The specific number of tasks obtained from the total task set, that is, the specific value of the first number, can be set according to a specific application scenario.

[0085] Step S202: Determine the first-order gradient for the current task subset based on the current task subset, the speech recognition model, and the model parameters and learning rate corresponding to the previous task subset.

[0086] It should be noted that, for the first task subset, there is actually no "previous task subset", that is, there is no "model parameters and learning rate corresponding to the previous task subset". At this time, the "model parameters and learning rate corresponding to the previous task subset" in step S202 adopt the initial model parameters and the initial learning rate.

[0087] In one possible implementation, the initial model parameters can be obtained by random initialization. In another possible implementation, the model parameters of a trained model with the same framework as the speech recognition model in this embodiment can be used as the initial model parameters of the speech recognition model in this embodiment.

[0088] It should be noted that the shape of the initial learning rate is the same as the shape of the initial model parameter, and the value of the initial learning rate can be obtained by random initialization. The present invention adopts a learning rate self-learning strategy. During the training process of the speech recognition model, based on the initial learning rate, the learning rate is automatically updated according to the training situation without manual adjustment of the learning rate.

[0089] Specifically, based on the current task subset, the speech recognition model, and the model parameters and learning rate corresponding to the previous task subset, the process of determining the first-order gradient for the current task subset may include:

[0090] Step S2021: Obtain an unprocessed task from the current task subset as the current task.

[0091] Step S2022: Use the speech recognition model to process the current task, and determine the prediction loss of the speech recognition model on the current task based on the processing result of the current task and the label information corresponding to the current task.

[0092] Among them, the processing result of the current task can represent the recognition status of the speech corresponding to the current task by the speech recognition model, and the label information corresponding to the current task can represent the text annotation information of the speech data corresponding to the current task.

[0093] Specifically, the process of processing the current task using the speech recognition model and determining the prediction loss of the speech recognition model on the current task based on the processing result of the current task and the label information corresponding to the current task may include:

[0094] Step a1: Acquire a second number of data points from the current task to form a data point set.

[0095] Among them, each data point in the data point set corresponds to label information.

[0096] Optionally, a second number of data points may be obtained from the current task according to the modeling unit of the speech recognition model. For example, if the modeling unit of the speech recognition model is a subword and the second number is 10240, feature information corresponding to each of the 10240 subwords may be obtained from the current task as 10240 data points, i.e., each data point is feature information corresponding to a subword.

[0097] This embodiment is not limited to obtaining the second number of data points from the current task according to the modeling unit of the speech recognition model, and other methods can also be used. For example, the second number of data points is obtained from the current task according to the entire speech. Assuming that the second number is 10, the feature information corresponding to the 10 speech lines is obtained from the current task, and each data point obtained from the current task according to the entire speech is the feature information corresponding to one speech line.

[0098] Step a2: input the data point set into the speech recognition model for processing to obtain the processing result output by the speech recognition model.

[0099] The speech recognition model processes each data point in the input data point set and outputs the processing result corresponding to each data point in the data point set. The processing result corresponding to each data point can represent the recognition of the speech corresponding to the data point by the speech recognition model.

[0100] Step a3: Calculate the prediction loss of the speech recognition model on the current task based on the processing result output by the speech recognition model and the label information corresponding to each data point in the data point set.

[0101] Specifically, the cross entropy loss of the speech recognition model on the currently traversed task can be calculated based on the processing result output by the speech recognition model and the label information corresponding to each data point in the data point set. If the current task is the i-th task T in the current task subset, i , then the speech recognition model is in the current task T i The prediction loss on It can be calculated by the following formula:

[0102]

[0103] Among them, x j represents the jth data point in the data point set, y j Represents data point x j The corresponding label information, f θ represents the speech recognition model, θ represents the model parameters of the speech recognition model, and f θ (x j ) represents the speech recognition model for data point x j The processing result is output after processing.

[0104] Step S2023: Calculate the first-order gradient according to the prediction loss of the speech recognition model on the current task, and use the calculated first-order gradient as the first-order gradient for the current task.

[0105] Specifically, the prediction loss of the speech recognition model on the current task can be Derivative θ is used to obtain the first-order gradient, which is used as the first-order gradient for the current task Ti The first-order gradient The first-order gradient It can be expressed as the gradient tensor G θ form.

[0106] Step S2024: Determine the parameters for the current task according to the first-order gradient for the current task, the parameters for the previous task, and the learning rate corresponding to the previous task subset.

[0107] Specifically, the learning rate corresponding to the previous task subset can be calculated according to the first-order gradient of the current task, the parameters of the previous task, and the learning rate corresponding to the previous task subset, and the calculation method shown in the following formula is used to calculate the learning rate corresponding to the current task T. i Parameters

[0108]

[0109] in, represents the parameters for the previous task, α m-1 represents the learning rate corresponding to the previous task subset (assuming that the current task subset is the mth task subset), and k represents the total number of data points in the data point set (i.e., the second number mentioned above). It should be noted that if the current task is the first task in the current task subset, then the "parameters of the previous task" adopt the model parameters θ corresponding to the previous task subset. m-1 .

[0110] In one possible implementation, steps S2022 to S2024 may be executed once. In order to obtain a better first-order gradient, in another possible implementation, steps S2022 to S2024 may be executed K times (K is an integer greater than 1, and the specific value of K may be set according to the specific application scenario).

[0111] It should be noted that in the implementation method of executing steps S2022 to S2024 K times, the data point sets processed by the speech recognition model are different. Assume that K=3, the data point set when executing steps S2022 to S2024 for the first time for the current task is D1, the data point set when executing steps S2022 to S2024 for the second time for the current task is D2, and the data point set when executing steps S2022 to S2024 for the third time for the current task is D3, then D1, D2 and D3 are different data point sets.

[0112] Step S2025, determine whether there are any unprocessed tasks in the current task subset, if not, execute step S2026, if yes, go to step S2021.

[0113] Step S2026: Determine the first-order gradient for the last processed task as the first-order gradient for the current task subset.

[0114] It should be noted that if steps S2022 to S2024 are executed K times, the first-order gradient determined for the last processed task at the Kth time is determined as the first-order gradient for the current task subset.

[0115] Step S203: Based on the model parameters and learning rate corresponding to the previous task subset and the first-order gradient for the current task subset, determine the model parameters and learning rate corresponding to the current task subset, and update the model parameters of the speech recognition model to the model parameters corresponding to the current task subset.

[0116] Specifically, the model parameters and learning rate corresponding to the previous task subset and the first-order gradient for the current task subset can be calculated according to the calculation method shown in the following formula to obtain the model parameters and learning rate corresponding to the current task subset:

[0117] θ m =θ m-1 -βG θ′ / n (3)

[0118] α m =α m-1 -βG θ′ / n (4)

[0119] Among them, θ m represents the model parameters corresponding to the current task subset, α m represents the learning rate corresponding to the current task subset, θ m-1 represents the model parameters corresponding to the previous task subset, α m-1 represents the learning rate corresponding to the previous task subset, G θ′ represents the first-order gradient for the current task subset, β is the setting parameter, and n represents the total number of tasks in the current task subset.

[0120] Step S204: Determine whether the training end condition is currently met. If so, end the first stage of training. If not, proceed to step S201.

[0121] Optionally, the training end condition may be reaching a preset number of training iterations. Of course, this embodiment is not limited thereto, and the training end condition may also be other conditions, such as model convergence.

[0122] Third embodiment

[0123] This embodiment introduces the implementation process of "Step S102: using the second category of training speech to perform second-stage training on the speech recognition model trained in the first stage" in the above embodiment.

[0124] There are many ways to implement the second stage training of the speech recognition model after the first stage training by using the second category of training speech. In one possible implementation, only the second category of training speech can be used to perform the second stage training on the speech recognition model after the first stage training. In order to obtain a speech recognition model with better performance, this embodiment provides another possible implementation. Please refer to Figure 3 , showing a schematic diagram of the process of the implementation, which may include:

[0125] Step S301: using one or more speech amplification methods to amplify the training speech of the second category to obtain amplified speech.

[0126] Specifically, for each training speech of the second category, one or more of the following speech amplification methods can be used to process the training speech: volume amplification, speed amplification (such as 0.9 times speed amplification, 1.1 times speed amplification), noise amplification (such as white noise amplification), reverberation amplification (such as small room reverberation amplification, medium room reverberation amplification, etc.), etc.

[0127] For example, after a voice is amplified in six ways, namely, volume amplification, 0.9 times speed amplification, 1.1 times speed amplification, white noise amplification, small room with reverberation amplification, and medium room with reverberation amplification, six amplified voices can be obtained. Voice amplification can increase the number of training voices.

[0128] It should be noted that each training speech of the second category is annotated with text. After a training speech is amplified to obtain an amplified speech, the amplified speech is annotated with the text annotated by the training speech. For example, the annotated text of a training speech A is "The weather is really good today", and the amplified speech obtained after amplifying the training speech A is A′, then A′ is annotated with "The weather is really good today".

[0129] Step S302: using the second category of training speech and the amplified speech to perform the second stage of training on the speech recognition model trained in the first stage.

[0130] Specifically, the amplified speech is used as the training speech and the original training speech to form the training data set of the second stage. When conducting the second stage of training, the training speech is obtained from the training data set, and the acoustic features (such as Fbank 40-dimensional acoustic features) of the obtained training speech are extracted. The extracted acoustic features are input into the speech recognition model trained in the first stage to obtain the output result of speech recognition, and the prediction loss is determined according to the output result of speech recognition and the annotated text of the training speech. The model parameters of the speech recognition model are updated according to the prediction loss. The speech recognition model trained in the first stage is iterated for multiple times according to the above process until the training end conditions are met, and the final speech recognition model is obtained after the training results.

[0131] The speech recognition model training method provided in the embodiment of the present invention uses a task set constructed based on multiple first-category training speech with rich speech resources to perform the first stage of training on the constructed speech recognition model. On the basis of the first stage of training, the present invention uses a small amount of low-resource category training speech to quickly train a speech recognition model with good recognition effect on low-resource category speech. The recognition effect of the speech recognition model greatly exceeds the recognition effect of the speech recognition model directly trained with low-resource category training speech, and also more stably exceeds the recognition effect of the speech recognition model trained in the form of "pre-training + fine-tuning" using traditional deep learning methods. The speech recognition model trained by the speech recognition model training method provided in the embodiment of the present invention has high recognition accuracy and stability.

[0132] Fourth embodiment

[0133] Based on the speech recognition model training method provided in the above embodiment, this embodiment provides a speech recognition method, which may include:

[0134] Step b1: Obtain target speech of the second category.

[0135] Among them, the second category is a category among the set multiple categories, in which the total duration of voice resources is less than or equal to the second preset duration threshold (for example, 50 hours), that is, the second category is a category with fewer voice resources, and the target voice is the voice to be recognized.

[0136] Step b2: Recognize the target speech based on the pre-trained speech recognition model to obtain a recognition result of the target speech.

[0137] The speech recognition model is trained using the speech recognition model training method provided in the above embodiment.

[0138] Since the speech recognition model trained by the speech recognition model training method provided in the above embodiment is a relatively stable speech recognition model with good recognition effect on the second category of speech, a relatively accurate speech recognition result can be obtained by performing speech recognition on the target speech of the second category.

[0139] Fifth embodiment

[0140] An embodiment of the present invention further provides a speech recognition model training device. The speech recognition model training device provided by the embodiment of the present invention is described below. The speech recognition model training device described below and the speech recognition model training method described above can refer to each other.

[0141] See also Figure 4 , shows a schematic diagram of the structure of a speech recognition model training device provided by an embodiment of the present invention, which may include: a first training module 401 and a second training module 402.

[0142] The first training module 401 is used to perform the first stage training on the constructed speech recognition model using the total task set to obtain the speech recognition model after the first stage training.

[0143] Among them, the task set includes tasks corresponding to multiple first categories respectively, and the tasks are constructed according to the training speech corresponding to the first category. The first category is a category among the set multiple categories, and the total duration of the speech resources it has is greater than or equal to the first preset duration threshold.

[0144] In the process of the first stage training of the speech recognition model, the direction of model parameter update is controlled by self-learning a learning rate suitable for the tasks in the total task set, and the model parameters are updated by calculating the first-order gradient.

[0145] The second training module 402 is used to perform a second stage of training on the speech recognition model trained in the first stage using the training speech of the second category.

[0146] The second category is a category among the multiple categories, wherein the total duration of voice resources is less than or equal to a second preset duration threshold.

[0147] Optionally, the speech recognition model training device provided by the embodiment of the present invention may further include: a task construction module. The task construction module is used to construct tasks corresponding to a plurality of first categories.

[0148] When constructing multiple tasks corresponding to the first category, the task construction module is specifically used to:

[0149] For each first category:

[0150] Obtaining the training speech of the first category;

[0151] Extracting acoustic features from the training speech of the first category;

[0152] The acoustic features extracted from the training speech of the first category are used as the task corresponding to the first category

[0153] Optionally, when the first training module 401 performs the first phase of training on the constructed speech recognition model using the task set, it is specifically used to:

[0154] Acquire a first number of tasks from the total task set to form a current task subset;

[0155] Based on the current task subset, the speech recognition model, and the model parameters and learning rate corresponding to the previous task subset, determine the first-order gradient for the current task subset; wherein, if the current task subset is the first task subset, the model parameters and learning rate corresponding to the previous task subset use the initial model parameters and initial learning rate;

[0156] Based on the model parameters and learning rate corresponding to the previous task subset and the first-order gradient for the current task subset, determine the model parameters and learning rate corresponding to the current task subset, and update the model parameters of the speech recognition model to the model parameters corresponding to the current task subset;

[0157] The step of obtaining a first number of tasks from the total task set to form a current task subset and subsequent steps are executed until a training end condition is met.

[0158] Optionally, when determining the first-order gradient for the current task subset based on the current task subset, the speech recognition model, and the model parameters and learning rate corresponding to the previous task subset, the first training module 401 is specifically used to:

[0159] Get an unprocessed task from the current task subset as the current task;

[0160] Processing the current task using the speech recognition model, and determining the prediction loss of the speech recognition model on the current task based on the processing result of the current task and the label information corresponding to the current task;

[0161] Calculate the first-order gradient according to the prediction loss of the speech recognition model on the current task, and use the calculated first-order gradient as the first-order gradient for the current task;

[0162] Determine the parameters for the current task according to the first-order gradient for the current task, the parameters for the previous task, and the learning rate corresponding to the previous task subset, wherein the speech recognition model processes the next task based on the parameters for the current task, wherein if the current task is the first task, the parameters for the previous task adopt the model parameters corresponding to the previous task subset;

[0163] The first-order gradient for the last processed task is determined as the first-order gradient for the current task subset.

[0164] Optionally, when the first training module 401 processes the current task using the speech recognition model and determines the prediction loss of the speech recognition model on the current task based on the processing result of the current task and the label information corresponding to the current task, it is specifically used to:

[0165] Acquire a second number of data points from the current task to form a data point set, wherein each data point in the data point set corresponds to label information;

[0166] Inputting the data point set into a speech recognition model for processing to obtain a processing result output by the speech recognition model;

[0167] The prediction loss of the speech recognition model on the current task is calculated according to the processing result output by the speech recognition model and the label information corresponding to each data point in the data point set.

[0168] Optionally, after determining the parameters for the current task, the first training module 401 is further configured to:

[0169] Determine whether the number of iterations on the current task has reached the preset number of iterations;

[0170] If yes, then the step of obtaining an unprocessed task from the current task subset as the current task and subsequent steps are executed; if no, then the step of obtaining a second number of data points from the current task to form a data point set and subsequent steps are executed.

[0171] Optionally, when the second training module 402 uses the second category of training speech to perform the second stage of training on the speech recognition model trained in the first stage, it is specifically used to:

[0172] Using one or more speech amplification methods to amplify the second category of training speech to obtain amplified speech;

[0173] The speech recognition model trained in the first stage is trained in the second stage using the training speech of the second category and the amplified speech.

[0174] The embodiment of the present invention provides a speech recognition model training device, which first uses a task set to perform a first-stage training on a constructed speech recognition model to obtain a speech recognition model after the first-stage training, and then uses a second-category training speech to perform a second-stage training on the speech recognition model after the first-stage training. The speech recognition model training device provided by the embodiment of the present invention uses a task set constructed based on a plurality of first-category training speech with rich speech resources and a training speech of a low-resource speech category, and can train a speech recognition model that has good recognition effect and good stability for speech of the low-resource speech category. In addition, in the process of the first-stage training of the speech recognition model by the embodiment of the present invention, the direction of updating the model parameters is controlled by self-learning the learning rate suitable for the tasks in the task set, and the model parameters are updated by calculating the first-order gradient. These two strategies reduce the computing time and resources consumed by model training, reduce the time consumption of manual adjustment of the learning rate, accelerate the model convergence speed, and reduce the time consumed by model training.

[0175] Sixth embodiment

[0176] The embodiment of the present invention further provides a speech recognition device. The speech recognition device provided by the embodiment of the present invention is described below. The speech recognition device described below and the speech recognition method described above can refer to each other.

[0177] The speech recognition device provided by the embodiment of the present invention may include: a speech acquisition module and a speech recognition module.

[0178] The speech acquisition module is used to acquire the target speech of the second category.

[0179] The speech recognition module is used to recognize the target speech based on a pre-trained speech recognition model to obtain a recognition result of the target speech.

[0180] The speech recognition model is trained using the speech recognition model training device provided in the above embodiment.

[0181] Since the speech recognition model trained by the speech recognition model training device provided by the above embodiment is a relatively stable speech recognition model with good recognition effect on the second category of speech, a relatively accurate speech recognition result can be obtained by performing speech recognition on the target speech of the second category.

[0182] Seventh embodiment

[0183] The present invention also provides a data processing device. Figure 5, showing a schematic diagram of the structure of the data processing device, the data processing device may include: at least one processor 501, at least one communication interface 502, at least one memory 503 and at least one communication bus 504;

[0184] In the embodiment of the present invention, the number of the processor 501, the communication interface 502, the memory 503, and the communication bus 504 is at least one, and the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504;

[0185] The processor 501 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention;

[0186] The memory 503 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0187] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0188] The constructed speech recognition model is trained in the first stage using the task set to obtain the speech recognition model after the first stage training; wherein the task set includes tasks corresponding to a plurality of first categories respectively, the tasks are constructed according to the training speech corresponding to the first category, the first category is a category in which the total duration of speech resources is greater than or equal to a first preset duration threshold among the set multiple categories, and in the process of training the speech recognition model in the first stage, the model parameter update direction is controlled by self-learning the learning rate suitable for the tasks in the task set, and the model parameters are updated by calculating the first-order gradient;

[0189] The speech recognition model trained in the first stage is trained in the second stage using the training speech of the second category; wherein the second category is a category among the set multiple categories, the total duration of the speech resources of which is less than or equal to the second preset duration threshold.

[0190] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0191] Eighth embodiment

[0192] An embodiment of the present invention further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0193] The constructed speech recognition model is trained in the first stage using the task set to obtain the speech recognition model after the first stage training; wherein the task set includes tasks corresponding to a plurality of first categories respectively, the tasks are constructed according to the training speech corresponding to the first category, the first category is a category in which the total duration of speech resources is greater than or equal to a first preset duration threshold among the set multiple categories, and in the process of training the speech recognition model in the first stage, the model parameter update direction is controlled by self-learning the learning rate suitable for the tasks in the task set, and the model parameters are updated by calculating the first-order gradient;

[0194] The speech recognition model trained in the first stage is trained in the second stage using the training speech of the second category; wherein the second category is a category among the set multiple categories, the total duration of the speech resources of which is less than or equal to the second preset duration threshold.

[0195] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0196] Ninth embodiment

[0197] An embodiment of the present invention further provides another data processing device, which may include: at least one processor, at least one communication interface, at least one memory and at least one communication bus;

[0198] In the embodiment of the present invention, the number of the processor, the communication interface, the memory, and the communication bus is at least one, and the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0199] The processor may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention;

[0200] The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0201] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0202] Acquire the target speech of the second category;

[0203] Recognize the target speech based on the pre-trained speech recognition model to obtain a recognition result of the target speech;

[0204] The speech recognition model is trained using the speech recognition model training method provided in the above embodiment.

[0205] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0206] Tenth embodiment

[0207] An embodiment of the present invention further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0208] Acquire the target speech of the second category;

[0209] Recognize the target speech based on the pre-trained speech recognition model to obtain a recognition result of the target speech;

[0210] The speech recognition model is trained using the speech recognition model training method provided in the above embodiment.

[0211] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0212] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0213] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0214] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech recognition model training method, characterized in that: include: The constructed speech recognition model is trained in the first stage using the task set to obtain the speech recognition model after the first stage training; wherein the task set includes tasks corresponding to a plurality of first categories respectively, the tasks are constructed according to the training speech corresponding to the first category, the first category is a category in which the total duration of speech resources is greater than or equal to a first preset duration threshold among the set multiple categories, and in the process of training the speech recognition model in the first stage, the model parameter update direction is controlled by self-learning the learning rate suitable for the tasks in the task set, and the model parameters are updated by calculating the first-order gradient; Using the training speech of the second category, the speech recognition model trained in the first stage is trained in the second stage; wherein the second category is a category in which the total duration of the speech resources is less than or equal to the second preset duration threshold among the set multiple categories, and the second preset duration threshold is less than the first preset duration threshold; The first stage of training the constructed speech recognition model using the task set includes: Acquire a first number of tasks from the total task set to form a current task subset; Based on the current task subset, the speech recognition model, and the model parameters and learning rate corresponding to the previous task subset, determine the first-order gradient for the current task subset; wherein, if the current task subset is the first task subset, the model parameters and learning rate corresponding to the previous task subset use the initial model parameters and initial learning rate; Based on the model parameters and learning rate corresponding to the previous task subset and the first-order gradient for the current task subset, determine the model parameters and learning rate corresponding to the current task subset, and update the model parameters of the speech recognition model to the model parameters corresponding to the current task subset; The step of obtaining a first number of tasks from the total task set to form a current task subset and subsequent steps are executed until a training end condition is met.

2. The speech recognition model training method according to claim 1, characterized in that: Constructing tasks corresponding to the plurality of first categories respectively, including: For each first category: Obtaining the training speech of the first category; Extracting acoustic features from the training speech of the first category; The acoustic features extracted from the training speech of the first category are used as the tasks corresponding to the first category.

3. The speech recognition model training method according to claim 1, characterized in that: The determining of the first-order gradient for the current task subset based on the current task subset, the speech recognition model, and the model parameters and learning rate corresponding to the previous task subset includes: Get an unprocessed task from the current task subset as the current task; Processing the current task using the speech recognition model, and determining the prediction loss of the speech recognition model on the current task based on the processing result of the current task and the label information corresponding to the current task; Calculate the first-order gradient according to the prediction loss of the speech recognition model on the current task, and use the calculated first-order gradient as the first-order gradient for the current task; Determine the parameters for the current task according to the first-order gradient for the current task, the parameters for the previous task, and the learning rate corresponding to the previous task subset, wherein the speech recognition model processes the next task based on the parameters for the current task, wherein if the current task is the first task, the parameters for the previous task adopt the model parameters corresponding to the previous task subset; The first-order gradient for the last processed task is determined as the first-order gradient for the current task subset.

4. The speech recognition model training method according to claim 3, characterized in that: The using the speech recognition model to process the current task, and determining the prediction loss of the speech recognition model on the current task based on the processing result of the current task and the label information corresponding to the current task, includes: Acquire a second number of data points from the current task to form a data point set, wherein each data point in the data point set corresponds to label information; Inputting the data point set into a speech recognition model for processing to obtain a processing result output by the speech recognition model; The prediction loss of the speech recognition model on the current task is calculated according to the processing result output by the speech recognition model and the label information corresponding to each data point in the data point set.

5. The speech recognition model training method according to claim 4, characterized in that: After determining the parameters for the current task, the method further includes: Determine whether the number of iterations on the current task has reached the preset number of iterations; If yes, then the step of obtaining an unprocessed task from the current task subset as the current task and subsequent steps are executed; if no, then the step of obtaining a second number of data points from the current task to form a data point set and subsequent steps are executed.

6. The speech recognition model training method according to claim 1, characterized in that: The second stage of training the speech recognition model trained in the first stage by using the training speech of the second category includes: Using one or more speech amplification methods to amplify the second category of training speech to obtain amplified speech; The speech recognition model trained in the first stage is trained in the second stage using the training speech of the second category and the amplified speech.

7. A speech recognition method, characterized in that: include: Acquire the target speech of the second category; Recognize the target speech based on the pre-trained speech recognition model to obtain a recognition result of the target speech; The speech recognition model is trained using the speech recognition model training method according to any one of claims 1 to 6.

8. A speech recognition model training device, characterized in that: include: a first training module and a second training module; The first training module is used to perform the first stage training on the constructed speech recognition model using the task set to obtain the speech recognition model after the first stage training; wherein the task set includes tasks corresponding to multiple first categories respectively, and the tasks are constructed according to the training speech corresponding to the first category, and the first category is a category in which the total duration of speech resources is greater than the first preset duration threshold among the set multiple categories. In the process of performing the first stage training on the speech recognition model, the model parameter update direction is controlled by self-learning the learning rate suitable for the tasks in the task set, and the model parameters are updated by calculating the first-order gradient; The second training module is used to perform a second stage of training on the speech recognition model trained in the first stage using a second category of training speech; wherein the second category is a category among the multiple categories whose total duration of speech resources is less than or equal to a second preset duration threshold, and the second preset duration threshold is less than the first preset duration threshold; Wherein, when the first training module uses the task set to perform the first stage training on the constructed speech recognition model, it is specifically used to: Acquire a first number of tasks from the total task set to form a current task subset; Based on the current task subset, the speech recognition model, and the model parameters and learning rate corresponding to the previous task subset, determine the first-order gradient for the current task subset; wherein, if the current task subset is the first task subset, the model parameters and learning rate corresponding to the previous task subset use the initial model parameters and initial learning rate; Based on the model parameters and learning rate corresponding to the previous task subset and the first-order gradient for the current task subset, determine the model parameters and learning rate corresponding to the current task subset, and update the model parameters of the speech recognition model to the model parameters corresponding to the current task subset; The step of obtaining a first number of tasks from the total task set to form a current task subset and subsequent steps are executed until a training end condition is met.

9. A data processing device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the speech recognition model training method as described in any one of claims 1 to 6, and / or to implement the various steps of the speech recognition method as described in claim 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the various steps of the speech recognition model training method as described in any one of claims 1 to 6, and / or implements the various steps of the speech recognition method as described in claim 7.

Images