Speech recognition method and device, computer device and storage medium

Abstract

The present disclosure relates to a method, apparatus, computer device, storage medium and computer program product for speech recognition. The method comprises: receiving a speech signal to be recognized; inputting the speech signal to be recognized into a first speech recognition model, outputting a recognition result via the first speech recognition model, the first speech recognition model being obtained by knowledge distillation training of a second speech recognition model, wherein there is a preset correlation between a temperature parameter in the knowledge distillation training process and an iteration number corresponding to a training stage of the first speech recognition model. The present method can improve the recognition accuracy of the speech recognition model while ensuring the efficiency of speech recognition.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a method, apparatus, computer device, and storage medium for speech recognition. Background Technology

[0002] With the development of vehicle intelligence technology, vehicles are supporting more and more functions. Among these, human-vehicle interaction via voice dialogue can simplify the user's operation process and greatly improve the user experience. Therefore, voice recognition technology has become a current research hotspot in human-vehicle interaction.

[0003] Traditional speech recognition methods often employ large-scale models trained on complex networks, which require significant computation and memory for both training and recognition. Therefore, smaller models trained on simpler networks are typically used on-vehicle devices for speech recognition and interaction. However, these smaller models tend to have lower accuracy, compromising speech recognition performance. Summary of the Invention

[0004] Therefore, it is necessary to provide a speech recognition method, apparatus, computer equipment, storage medium, and computer program product that can improve the accuracy of speech recognition in order to address the above-mentioned technical problems.

[0005] In a first aspect, embodiments of this disclosure provide a speech recognition method, the method comprising:

[0006] Receive the voice signal to be recognized;

[0007] The speech signal to be recognized is input into the first speech recognition model, and the first speech recognition model outputs the recognition result. The first speech recognition model is obtained by knowledge distillation training through the second speech recognition model. There is a preset correlation between the temperature parameter in the knowledge distillation training process and the number of iterations corresponding to the training stage of the first speech recognition model.

[0008] In one embodiment, the first speech recognition model is obtained by knowledge distillation training using a second speech recognition model, including:

[0009] Construct an initial first speech recognition model, wherein training parameters are set in the initial first speech recognition model;

[0010] The training speech signal is input into the initial first speech recognition model, and the initial first speech recognition model outputs the initial first recognition result.

[0011] The training speech signal is input into the second speech recognition model, and the second speech recognition model outputs the initial second recognition result.

[0012] The initial first recognition result is obtained by processing the temperature parameter that matches the number of iterations corresponding to the current training stage; the initial second recognition result is obtained by processing the temperature parameter.

[0013] The loss function value is determined based on the difference between the first recognition result and the second recognition result. The initial first speech recognition model is iteratively adjusted according to the loss function value until the preset conditions are met, and the first speech recognition model is obtained.

[0014] In one embodiment, determining the loss function value based on the difference between the first identification result and the second identification result includes:

[0015] Obtain the initial first intermediate result output by the first preset intermediate layer of the initial first speech recognition model and the initial second intermediate result output by the second preset intermediate layer of the second speech recognition model;

[0016] The initial first intermediate result is obtained by processing the intermediate temperature parameter that matches the number of iterations corresponding to the current training stage; the second intermediate result is obtained by processing the initial second intermediate result according to the intermediate temperature parameter.

[0017] The loss function value is determined based on the first difference between the first identification result and the second identification result, and the second difference between the first intermediate result and the second intermediate result.

[0018] In one embodiment, determining the loss function value based on the difference between the first identification result and the second identification result includes:

[0019] Determine the loss weights that match the number of iterations corresponding to the current training phase;

[0020] The loss function value is determined based on the difference between the first identification result and the second identification result, and the loss weight.

[0021] In one embodiment, the method for determining the temperature parameter includes:

[0022] Obtain the correlation formula between temperature parameter and iteration number, wherein the correlation formula includes temperature change parameter value and initial temperature parameter value, wherein as the iteration number increases from a first preset value to a second preset value, the temperature parameter first increases and then decreases;

[0023] Substituting the number of iterations corresponding to the current training phase into the correlation formula yields the temperature parameter corresponding to the current training phase.

[0024] In one embodiment, the method for obtaining the second speech recognition model includes:

[0025] Obtain a set of sample speech signals, the set including multiple sample speech signals labeled with speech tags;

[0026] Construct an initial speech recognition model, wherein training parameters are set in the initial speech recognition model;

[0027] The sample speech signal is input into the initial speech recognition model to generate a recognition result;

[0028] Based on the difference between the recognition result and the labeled speech tags, the initial speech recognition model is iteratively adjusted until the difference meets the preset requirements, thus obtaining the second speech recognition model.

[0029] In one embodiment, the initial speech recognition model includes an encoder, an intermediate decoder, a first decoder, and a second decoder. The step of inputting the sample speech signal into the initial speech recognition model to generate a recognition result includes:

[0030] The sample speech signal is input into the initial speech recognition model to generate a recognition result, wherein the recognition result includes a first sub-result output by the first decoder, a second sub-result output by the second decoder, and a third sub-result output by the intermediate decoder.

[0031] In one embodiment, the encoder includes a basic encoder layer and a folded encoder layer, and the intermediate decoder is disposed between the basic encoder layer and the folded encoder layer. The folded encoder layer includes multiple folded encoders, which share parameters according to a preset division method, wherein the preset division method is determined based on a preset number of repetitions and the number of folded encoder layers.

[0032] Secondly, embodiments of this disclosure also provide a speech recognition apparatus. The apparatus includes:

[0033] The receiving module is used to receive the voice signal to be recognized;

[0034] The output module is used to input the speech signal to be recognized into the first speech recognition model, and output the recognition result through the first speech recognition model. The first speech recognition model is obtained by knowledge distillation training through the second speech recognition model. There is a preset correlation between the temperature parameter in the knowledge distillation training process and the number of iterations corresponding to the training stage of the first speech recognition model.

[0035] In one embodiment, the module for acquiring the first speech recognition model includes:

[0036] The first construction module is used to construct an initial first speech recognition model, wherein the initial first speech recognition model is configured with training parameters;

[0037] The first input module is used to input the training speech signal into the initial first speech recognition model, and output the initial first recognition result through the initial first speech recognition model.

[0038] The second input module is used to input the training speech signal into the second speech recognition model, and output the initial second recognition result through the second speech recognition model;

[0039] The processing module is used to process the initial first recognition result according to the temperature parameter matching the number of iterations corresponding to the current training stage to obtain a first recognition result, and to process the initial second recognition result according to the temperature parameter to obtain a second recognition result;

[0040] The determination module is used to determine the loss function value based on the difference between the first recognition result and the second recognition result, and to iteratively adjust the initial first speech recognition model according to the loss function value until a preset condition is met to obtain the first speech recognition model.

[0041] In one embodiment, the determining module includes:

[0042] The acquisition submodule is used to acquire the initial first intermediate result output by the first preset intermediate layer of the initial first speech recognition model and the initial second intermediate result output by the second preset intermediate layer of the second speech recognition model.

[0043] The processing submodule is used to process the initial first intermediate result according to the intermediate temperature parameter that matches the number of iterations corresponding to the current training stage to obtain the first intermediate result, and to process the initial second intermediate result according to the intermediate temperature parameter to obtain the second intermediate result.

[0044] The first determining submodule is used to determine the loss function value based on the first difference between the first identification result and the second identification result, and the second difference between the first intermediate result and the second intermediate result.

[0045] In one embodiment, the determining module includes:

[0046] The second determination submodule is used to determine the loss weights that match the number of iterations corresponding to the current training phase;

[0047] The third determining submodule is used to determine the loss function value based on the difference between the first identification result and the second identification result and the loss weight.

[0048] In one embodiment, the temperature parameter determination module includes:

[0049] The first acquisition module is used to acquire the correlation formula between temperature parameters and iteration number, wherein the correlation formula includes temperature change parameter value and initial temperature parameter value, wherein as the iteration number increases from a first preset value to a second preset value, the temperature parameter first increases and then decreases.

[0050] The substitution module is used to substitute the number of iterations corresponding to the current training stage into the correlation formula to obtain the temperature parameters corresponding to the current training stage.

[0051] In one embodiment, the module for acquiring the second speech recognition model includes:

[0052] The second acquisition module is used to acquire a set of sample speech signals, the set including multiple sample speech signals labeled with speech tags;

[0053] The second construction module is used to construct an initial speech recognition model, in which training parameters are set;

[0054] The generation module is used to input the sample speech signal into the initial speech recognition model and generate a recognition result;

[0055] An adjustment module is used to iteratively adjust the initial speech recognition model based on the difference between the recognition result and the labeled speech tags until the difference meets the preset requirements, thereby obtaining a second speech recognition model.

[0056] In one embodiment, the initial speech recognition model includes an encoder, an intermediate decoder, a first decoder, and a second decoder, and the generation module includes:

[0057] A generation submodule is used to input the sample speech signal into the initial speech recognition model and generate a recognition result, wherein the recognition result includes a first sub-result output by the first decoder, a second sub-result output by the second decoder, and a third sub-result output by the intermediate decoder.

[0058] In one embodiment, the encoder includes a basic encoder layer and a folded encoder layer, and the intermediate decoder is disposed between the basic encoder layer and the folded encoder layer. The folded encoder layer includes multiple folded encoders, which share parameters according to a preset division method, wherein the preset division method is determined based on a preset number of repetitions and the number of folded encoder layers.

[0059] Thirdly, embodiments of this disclosure also provide a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described in any one of the embodiments of this disclosure.

[0060] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the method described in any one of the embodiments of this disclosure.

[0061] Fifthly, embodiments of this disclosure also provide a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method described in any one of the embodiments of this disclosure.

[0062] In this embodiment of the disclosure, during the speech recognition process, a speech signal to be recognized is received and input into a first speech recognition model. The first speech recognition model outputs the recognition result. The first speech recognition model is obtained by knowledge distillation training through a second speech recognition model. During the knowledge distillation training process, there is a preset correlation between the temperature parameter and the number of iterations corresponding to the training stage. Therefore, during the knowledge distillation training process, the temperature parameter is adaptively adjusted as the number of iterations changes, thereby ensuring the simplicity of the structure of the first speech recognition model, improving the efficiency of speech recognition, and improving the recognition accuracy of the trained first speech recognition model. Because the temperature parameter changes adaptively during the training process, the information content provided by the second speech recognition model also changes adaptively in each iteration of training, ensuring training efficiency while improving the recognition accuracy of the first speech recognition model after multiple iterations of training. Attached Figure Description

[0063] Figure 1 This is a flowchart illustrating a speech recognition method in one embodiment;

[0064] Figure 2 This is a flowchart illustrating a method for obtaining a first speech recognition model in one embodiment;

[0065] Figure 3 This is a flowchart illustrating a method for obtaining a first speech recognition model in one embodiment;

[0066] Figure 4 This is a flowchart illustrating a method for obtaining a first speech recognition model in one embodiment;

[0067] Figure 5 This is a flowchart illustrating how temperature parameters are determined in one embodiment;

[0068] Figure 6 This is a flowchart illustrating a method for obtaining a second speech recognition model in one embodiment;

[0069] Figure 7 This is a schematic diagram of the structure of the second speech recognition model in one embodiment;

[0070] Figure 8 This is a schematic diagram of the structure of a speech recognition system in one embodiment;

[0071] Figure 9 This is a schematic diagram of the structure of a speech recognition system in one embodiment;

[0072] Figure 10 This is a structural block diagram of a speech recognition device in one embodiment;

[0073] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this disclosure and are not intended to limit the embodiments of this disclosure.

[0075] In one embodiment, such as Figure 1 As shown, a speech recognition method is provided, the method comprising:

[0076] Step S110: Receive the voice signal to be recognized;

[0077] In this embodiment of the disclosure, a voice signal to be recognized is received. In one example, the application scenario of this embodiment includes a human-vehicle interaction scenario, which may include, but is not limited to, vehicle control, navigation, media, question and answer, etc. In the human-vehicle interaction scenario, the voice signal to be recognized may be obtained by processing audio data collected by a voice acquisition device, wherein the voice acquisition device may include, but is not limited to, a microphone device installed in the vehicle. The voice signal to be recognized may be obtained by processing the original audio data, wherein the processing of the original audio data may include, but is not limited to, filtering of interjections.

[0078] Step S120: The speech signal to be recognized is input into the first speech recognition model, and the recognition result is output by the first speech recognition model. The first speech recognition model is obtained by knowledge distillation training through the second speech recognition model. There is a preset correlation between the temperature parameter in the knowledge distillation training process and the number of iterations corresponding to the training stage of the first speech recognition model.

[0079] In this embodiment, after acquiring the speech signal to be recognized, the speech signal is input into a first speech recognition model, and the recognition result is output after recognition by the first speech recognition model. The first speech recognition model is obtained by knowledge distillation training through a second speech recognition model. Specifically, the core idea of ​​knowledge distillation is to train a complex model and feed the output of this complex model and labeled data into a small network. Therefore, in the knowledge distillation process, there is usually a complex large model (teacher model) and a small model (student model). In this disclosure, the first speech recognition model and the second speech recognition model are set to have the same function, both used for speech recognition. Since the first speech recognition model is obtained by knowledge distillation training through the second speech recognition model, the first speech recognition model corresponds to the student model, and the second speech recognition model corresponds to the teacher model. The complexity and computational load of the second speech recognition model are greater than those of the first speech recognition model, and the network complexity of the second speech recognition model is greater than that of the first speech recognition model. Due to the higher network complexity of the second speech recognition model, the model accuracy and recognition accuracy of the second speech recognition model are higher, but the computational load is larger. In this embodiment, the first speech recognition model is configured to have lower complexity than the second speech recognition model. In one example, when the application scenario is a human-vehicle interaction scenario, the second speech recognition model has high network complexity, high computational load, and requires a large amount of memory, making it difficult to apply to the vehicle. The first speech recognition model, however, has lower complexity and requires less memory, thus it can be applied to the vehicle for speech recognition, enabling human-vehicle interaction. In this embodiment, during the knowledge distillation training process, there is a preset correlation between the temperature parameter and the number of iterations corresponding to the training phase. During the knowledge distillation training process, the student model is trained using "soft labels" output by the teacher model. The temperature parameter affects the information content in the soft labels. Generally, the larger the temperature parameter, the "softer" the soft label, the smoother the trend, and the more information it contains, but the information reliability will decrease accordingly. Conversely, the smaller the temperature parameter, the "harder" the soft label, the less information it contains, but the information reliability will increase accordingly. In this embodiment, since there is a preset correlation between the temperature parameter and the number of iterations corresponding to the training stage, the temperature parameter will change accordingly as the number of training iterations changes during the knowledge distillation process. The amount of information and the reliability of the information transmitted by the teacher model to the student model will also differ in each training iteration. Therefore, in this embodiment, during the knowledge distillation training process, the amount of information and the reliability of the information transmitted by the second speech recognition model to the first speech recognition model will change with different training stages, thereby improving the model accuracy and recognition accuracy of the trained first speech recognition model.In one example, the preset correlation between the temperature parameter and the number of iterations can be set according to the actual application scenario. For example, it can be set so that after the knowledge distillation training process starts, the temperature parameter increases with the increase of the number of iterations; or it can be set so that after the knowledge distillation training process starts, the temperature parameter first increases and then decreases with the increase of the number of iterations. In one example, the preset correlation between the temperature parameter and the number of iterations can be determined by a preset formula, which can include fixed parameters, such as the temperature change rate parameter, the number of iterations to stop, and the initial temperature parameter, etc., which can be set according to the actual application scenario. In one possible implementation, when the application scenario is a human-vehicle interaction scenario, the first speech recognition model is set on the vehicle side. The first speech recognition model outputs the corresponding recognition result based on the received speech signal to be recognized. The recognition result can include the text content corresponding to the speech signal. In one example, after obtaining the recognition result of the first speech recognition model, the recognition result can be displayed on the display device on the vehicle side (such as the vehicle screen). In another example, the obtained recognition result can also be stored in a preset storage device. In one possible implementation, the main architecture of the second speech recognition model can consist of a shared encoder, a CTC decoder, and an attention-based decoder. The shared encoder includes a basic encoder layer and a folded encoder layer, and a folded, self-adjusting intermediate decoder layer outputs the training parameters of the basic encoder layer and the upper layers within the shared encoder. The encoder of the first speech recognition model does not include the folded, self-adjusting intermediate decoder layer. The decoder architecture is consistent with that of the second speech recognition model, but the hyperparameters such as the number of layers, the number of attention heads, and the attention dimension differ. In one example, after obtaining the recognition result, the result is processed and displayed according to a preset presentation method. The processing of the recognition result may include, but is not limited to, filtering of interjections and detextification, converting the recognition result into data suitable for the application scenario.

[0080] In this embodiment of the disclosure, during the speech recognition process, a speech signal to be recognized is received and input into a first speech recognition model. The first speech recognition model outputs the recognition result. The first speech recognition model is obtained by knowledge distillation training through a second speech recognition model. During the knowledge distillation training process, there is a preset correlation between the temperature parameter and the number of iterations corresponding to the training stage. Therefore, during the knowledge distillation training process, the temperature parameter is adaptively adjusted as the number of iterations changes, thereby ensuring the simplicity of the structure of the first speech recognition model, improving the efficiency of speech recognition, and improving the recognition accuracy of the trained first speech recognition model. Because the temperature parameter changes adaptively during the training process, the information content provided by the second speech recognition model also changes adaptively in each iteration of training, ensuring training efficiency while improving the recognition accuracy of the first speech recognition model after multiple iterations of training.

[0081] In one embodiment, such as Figure 2 As shown, the first speech recognition model is obtained by knowledge distillation training using the second speech recognition model, and includes:

[0082] Step S210: Construct an initial first speech recognition model, wherein training parameters are set in the initial first speech recognition model;

[0083] Step S220: Input the training speech signal into the initial first speech recognition model, and output the initial first recognition result through the initial first speech recognition model;

[0084] Step S230: Input the training speech signal into the second speech recognition model, and output the initial second recognition result through the second speech recognition model;

[0085] Step S240: Process the initial first recognition result according to the temperature parameter matching the iteration number corresponding to the current training stage to obtain the first recognition result; process the initial second recognition result according to the temperature parameter to obtain the second recognition result.

[0086] Step S250: Determine the loss function value based on the difference between the first recognition result and the second recognition result, and iteratively adjust the initial first speech recognition model according to the loss function value until the preset conditions are met to obtain the first speech recognition model.

[0087] In this embodiment, the first speech recognition model is obtained by training a second speech recognition model using instruction distillation. Specifically, an initial first speech recognition model is constructed, which includes training parameters. This initial first speech recognition model comprises a model determined based on a preset deep learning algorithm. In one possible implementation, since the first speech recognition model is used for speech recognition, it can be configured to include an encoder and a decoder. In one example, the encoder can be configured to consist of multiple conformer modules, and the decoder can be configured to include a CTC (Connectivity-Temporal Classification) decoder and an Attention-based decoder. The training speech signal is input into the initial first speech recognition model, which outputs an initial first recognition result. The training speech signal is then input into the second speech recognition model, which outputs an initial second recognition result.

[0088] In one possible implementation, the training speech signal includes a speech signal obtained by processing audio data according to the actual application scenario. In one example, the training speech signal includes a training set of speech signals. When acquiring the training set, audio data can be collected first, and the audio data can be cleaned, valid audio segments extracted, and data labeled to obtain training audio segments and corresponding label pairs. Feature extraction is then performed on the processed data to determine audio features, resulting in the training set of the speech signal. The feature extraction process may include, but is not limited to, pre-emphasis, windowing, accelerated Fourier transform, Mel filtering, etc. Preferably, the extracted audio features may include F-bank (Filter Bank) features. Since the human ear's response to the sound spectrum is non-linear, F-bank feature extraction processes audio in a manner similar to the human ear, which can improve the performance of speech recognition. The general steps for obtaining the F-bank features of the speech signal are: pre-emphasis, framing, windowing, short-time Fourier transform (STFT), Mel filtering, etc. In one example, when the application scenario is a human-vehicle interaction scenario, when collecting audio data, user audio data can be collected for the actual vehicle. The audio data can include, but is not limited to, audio data from multiple interaction scenarios such as vehicle control, navigation, media, and question and answer.

[0089] In this embodiment, the second speech recognition model is a pre-trained speech recognition model that performs the same function as the first speech recognition model. Typically, the second speech recognition model has higher complexity and accuracy, but also higher computational cost and potentially lower recognition efficiency. Since there is a preset correlation between the temperature parameter and the number of iterations corresponding to the training stage, the temperature parameter corresponding to the current training stage is determined based on the number of iterations. The determined temperature parameter is used to process the initial first and second recognition results to obtain the first and second recognition results. In one example, when processing the recognition results using the temperature parameter, the initial first and second recognition results can be processed using a softmax function (normalized exponential function), where the temperature parameter in the softmax function is set to the temperature parameter corresponding to the current training stage. A loss function value is determined based on the obtained first and second recognition results. The initial first speech recognition model is iteratively adjusted based on the loss function value. In each iteration, training speech samples are input into the speech recognition model adjusted in the previous iteration to determine the new first and second recognition results and obtain the corresponding loss function value, thus iteratively adjusting the recognition model.

[0090] In one possible implementation, the loss function value can be determined using a preset loss function, which is typically determined based on the actual application scenario. In one example, the preset loss function can be set to be determined based on the first difference between the first and second recognition results, and the second difference between the initial first and initial second recognition results. It is understood that, depending on the actual application scenario and requirements, the loss function may also include other parameters, and this disclosure does not impose any restrictions on this. In one example, the loss function may also include loss weights. Different losses (e.g., differences between recognition results) correspond to different loss weights. The loss weights can be set according to the actual application scenario. The loss weights can be preset fixed weight values ​​that remain constant during the knowledge distillation iterative adjustment process. Alternatively, the loss weights can be set to adaptively change, and the loss weights corresponding to different training stages may differ. For example, a correlation can be set between the loss weights and the number of iterations corresponding to the training stage, with different iteration numbers corresponding to different loss weights.

[0091] In this embodiment of the disclosure, the iterative adjustment process ends when a preset condition is met, resulting in a first speech recognition model. The preset condition is typically determined based on the actual application scenario. In one example, the preset condition may include the number of iterations equaling a preset number of iterations. This preset number of iterations can be determined in advance based on the actual application scenario, or it can be determined by the performance of the speech recognition model on a preset set of speech signals. The preset condition may also include the loss function value no longer decreasing on a preset validation set of speech signals. In one example, the difference between recognition results can be obtained by calculating the Kullback-Leibler Divergence.

[0092] In this embodiment, a first speech recognition model is obtained by knowledge distillation training using a pre-trained second speech recognition model. During the knowledge distillation process, the initial recognition results of the first and second speech recognition models are processed using temperature parameters corresponding to the training phase. The loss function value is determined based on the difference between the processed results, thereby enabling adaptive changes according to different temperature parameters during the training phase. In the process of iteratively adjusting to obtain the first speech recognition model, the amount of information transmitted from the second speech recognition model to the first speech recognition model during training changes with the number of iterations. While ensuring the recognition efficiency of the first speech recognition model, the model accuracy and recognition precision of the trained first speech recognition model are improved. This approach can be applied to scenarios such as human-vehicle interaction, enhancing the user experience.

[0093] In one embodiment, such as Figure 3 As shown, determining the loss function value based on the difference between the first identification result and the second identification result includes:

[0094] Step S251: Obtain the initial first intermediate result output by the first preset intermediate layer of the initial first speech recognition model and the initial second intermediate result output by the second preset intermediate layer of the second speech recognition model;

[0095] Step S252: Process the initial first intermediate result according to the intermediate temperature parameter that matches the number of iterations corresponding to the current training stage to obtain the first intermediate result; process the initial second intermediate result according to the intermediate temperature parameter to obtain the second intermediate result.

[0096] Step S253: Determine the loss function value based on the first difference between the first identification result and the second identification result, and the second difference between the first intermediate result and the second intermediate result.

[0097] In this embodiment, the first speech recognition model and the second speech recognition model are configured as a multi-layer structure, with each layer corresponding to an output result. In this embodiment, the loss function value can also be determined based on the difference between the intermediate output results of the intermediate layers of the speech recognition model. Specifically, after inputting the training speech signal into the initial first speech recognition model, the initial first intermediate result output by the first preset intermediate layer of the initial first speech recognition model is obtained; after inputting the training speech signal into the second speech recognition model, the initial second intermediate result output by the second preset intermediate layer of the second speech recognition model is obtained. The first preset intermediate layer may include one or more first intermediate layers, and the second preset intermediate layer may include one or more second intermediate layers. The number of first intermediate layers is the same as the number of second intermediate layers, and there is a correspondence between the first preset intermediate layers and the second preset intermediate layers. When the first preset intermediate layer includes multiple first intermediate layers and the second preset intermediate layer includes multiple second intermediate layers, the number of first intermediate layers is the same as the number of second intermediate layers, and the multiple first intermediate layers and multiple second intermediate layers correspond one-to-one. For example, suppose the first preset intermediate layer includes two first intermediate layers, which correspond to the third and sixth layers of the initial first speech recognition model, and the second preset intermediate layer also includes two second intermediate layers, which correspond to the sixth and twelfth layers of the second speech recognition model. Then the third layer of the initial first speech recognition model corresponds to the sixth layer of the second speech recognition model, and the sixth layer of the initial first speech recognition model corresponds to the twelfth layer of the second speech recognition model. When the first preset intermediate layer and the second preset intermediate layer each include multiple intermediate layers, in this embodiment, the initial first intermediate result and the initial second intermediate result also include multiple intermediate results, with each intermediate layer corresponding to one intermediate result. At this time, the second difference also includes multiple difference values. When determining the second difference, multiple difference values ​​are obtained according to the one-to-one correspondence between the intermediate layers, and the loss function value is determined according to the multiple difference values ​​corresponding to the first difference and the second difference. In one example, the difference value can be calculated using the distribution difference of the output, i.e., the KL divergence. The first and second preset intermediate layers can be set according to the actual application scenario. In one example, due to the high complexity and large number of layers in the second speech recognition model, the layer number of each second intermediate layer in the second preset intermediate layer is greater than the layer number of the corresponding first intermediate layer in the first preset intermediate layer. In one possible implementation, the first and second preset intermediate layers can be determined based on the initial number of layers in the first and second speech recognition models. For example, the first and second preset intermediate layers can be selected proportionally to the number of layers.After obtaining the initial first intermediate result and the initial second intermediate result, further processing is required using the intermediate temperature parameter. In this embodiment, there is a preset correlation between the intermediate temperature parameter and the number of iterations corresponding to the training phase. In one example, the correlation between the intermediate temperature parameter and the number of iterations can be determined by a preset formula. The corresponding intermediate temperature parameter is determined based on the number of iterations corresponding to the current training phase. The initial first intermediate result and the initial second intermediate result are then processed using the determined intermediate temperature parameter to obtain the first intermediate result and the second intermediate result. The loss function value is determined based on the first difference between the first recognition result and the second recognition result, and the second difference between the first intermediate result and the second intermediate result. In one example, when determining the loss function value, the loss function also includes corresponding loss weights. For example, the first difference and the second difference can correspond to different loss weights. The loss weights can be set to a fixed value or can be set to adaptively change with different training phases.

[0098] In one possible implementation, the first speech recognition model includes an encoder, a first decoder (e.g., a CTC decoder), and a second decoder (e.g., an attention-based decoder). During knowledge distillation, a multi-level knowledge distillation method is used to train the first speech recognition model. Multi-level knowledge distillation refers to optimizing the first speech recognition model during training based on the differences between the outputs of intermediate and final layers. Specifically, a loss function value can be calculated based on the outputs of one or more intermediate layers, and the loss function value is determined by the loss function. In one example, the loss function includes the distribution difference of an intermediate layer output. The loss function can include four parts, namely the distribution difference between the last layer output of the second speech recognition model and the last layer output of the first speech recognition model, denoted as Loss. TS The difference in the distribution of the intermediate layer outputs of the second speech recognition model and the intermediate layer outputs of the first speech recognition model is denoted as Loss. Inter-TS The output loss of the first decoder is denoted as Loss. ctc The output loss of the second decoder is denoted as Loss. att At this point, the total optimization loss during training corresponds to Loss = α * Loss ctc +β*Loss att +γ*Loss Inter-TS +θ*Loss TS Where α, β, γ, and θ are hyperparameters, i.e., loss weights, used to measure the importance of different loss functions. Loss... TS and Loss Inter-TSThe output result is determined based on the softmax processing of the temperature parameter. In this embodiment, the temperature parameter has a preset correlation with the number of iterations. Therefore, the temperature parameter corresponding to the calculation of the loss function value in each iteration is not a constant value, but rather changes adaptively with the training process. In one example, Loss... TS The corresponding loss weights θ and Loss Inter-TS The corresponding loss weight γ can also be set to change adaptively as the training process progresses.

[0099] In this embodiment, during knowledge distillation, the loss function value is determined based on the output differences of the intermediate layers of the model, and the speech recognition model is iteratively adjusted. This achieves multi-level knowledge distillation training by normalizing the first part of the first speech recognition model and fitting the second speech recognition model using the differences in the output distributions of the first and second speech recognition models at different specific levels. This further improves the recognition accuracy of the trained first speech recognition model. Furthermore, when determining the loss function value, the intermediate temperature parameter is adaptively adjusted according to changes in the training phase. During the iterative adjustment to obtain the first speech recognition model, the amount of intermediate layer information passed from the intermediate layer of the second speech recognition model to the first speech recognition model during training changes with the number of iterations. This ensures the recognition efficiency of the first speech recognition model while improving the model accuracy and recognition precision of the trained first speech recognition model. This approach can be applied to scenarios such as human-vehicle interaction, enhancing the user experience.

[0100] In one embodiment, such as Figure 4 As shown, determining the loss function value based on the difference between the first identification result and the second identification result includes:

[0101] Step S254: Determine the loss weights that match the number of iterations corresponding to the current training phase;

[0102] Step S255: Determine the loss function value based on the difference between the first identification result and the second identification result and the loss weight.

[0103] In this embodiment of the disclosure, when determining the loss function value, the loss function value is determined based on the difference between the recognition results and the loss weights. The loss weights are set to have a preset correlation with the number of iterations corresponding to the training phase. The correlation between the loss weights and the number of iterations can be determined by a preset correlation formula. Specifically, a matching loss weight is determined based on the number of iterations corresponding to the current training phase. In one example, the number of iterations corresponding to the current training phase can be determined by a preset correlation formula. The difference between the first and second recognition results is determined based on the first and second recognition results, and the loss function value is determined by combining this difference with the determined loss weights. In one possible implementation, the loss function value is determined by a weighted sum of preset losses. The preset losses include, but are not limited to, the difference between the first and second recognition results. In one example, the loss function value also includes the loss and loss weights determined based on the decoder output of the speech recognition model. In one example, when determining the loss function value, there may be multiple loss weights. The values ​​of multiple loss weights can be determined according to the actual application scenario. For example, the loss weights can be set to a fixed value, or they can be set to adaptively change with the training phase. In one example, when the loss function value corresponds to multiple loss weights, the sum of the multiple loss weights can be set to a certain value.

[0104] In this embodiment, when determining the loss function value, the loss weight is set to be correlated with the number of iterations corresponding to the training stage. Therefore, the loss weight values ​​corresponding to different training stages will also be different, so that the amount of information passed from the second speech recognition model to the first speech recognition model changes with the training stage. This ensures the recognition efficiency of the first speech recognition model while further improving the recognition accuracy of the first speech recognition model obtained after knowledge distillation training. It can be applied to application scenarios such as human-vehicle interaction, taking into account both model complexity and recognition accuracy, and improving the user experience.

[0105] In one embodiment, such as Figure 5 As shown, the method for determining the temperature parameter includes:

[0106] Step S510: Obtain the correlation formula between temperature parameter and iteration number, wherein the correlation formula includes temperature change parameter value and initial temperature parameter value, wherein as the iteration number increases from the first preset value to the second preset value, the temperature parameter first increases and then decreases.

[0107] Step S520: Substitute the number of iterations corresponding to the current training stage into the correlation formula to obtain the temperature parameter corresponding to the current training stage.

[0108] In this embodiment, when determining the temperature parameter, a correlation formula between the temperature parameter and the number of iterations is obtained. In this embodiment, the correlation formula includes a temperature change parameter value and an initial temperature parameter value. The different temperature change parameter values ​​affect the rate at which the temperature parameter changes with the number of iterations. The initial temperature parameter value can be a suitable temperature parameter set according to the actual application scenario. In this embodiment, when setting the correlation formula, as the number of iterations increases from a first preset value to a second preset value, the temperature parameter first increases and then decreases. The first and second preset values ​​are set according to the actual application scenario. In one example, the first preset value can be set to the number of iterations at the start of the iteration, and the second preset value can be set to the number of iterations at the end of the iteration. In this case, throughout the entire knowledge distillation training process, the temperature parameter first increases and then decreases as the number of iterations increases. During each iteration training process, substituting the number of iterations corresponding to the training phase into the correlation formula allows the calculation of the corresponding temperature parameter.

[0109] As the number of iterations increases from the first preset value to the second preset value, the temperature parameter first increases and then decreases. Based on this change process, in one possible implementation, when setting the correlation formula, the correlation formula between the temperature parameter and the number of iterations can be set as a two-part linear relationship. In one part of the linear relationship, the temperature parameter increases with the number of iterations, and in the other part, the temperature parameter decreases with the number of iterations. In another possible implementation, the correlation formula can be set using the sine or cosine function based on the changing law of the sine and cosine functions. In one example, as shown in equation (1), the correlation formula between the temperature parameter and the number of iterations can be determined by the sine function.

[0110]

[0111] Where T0 represents the initial temperature parameter value, μ is the temperature change parameter value, representing the rate of change of the temperature parameter with the number of iterations, and p represents the number of iterations corresponding to the current training phase. stop p represents the number of iterations at which knowledge distillation training stops. stop The number of iterations can be directly specified, or it can be determined by the fact that the loss no longer decreases on the validation set. Correspondingly, the relationship between the loss weight and the number of iterations in this embodiment can be shown in equation (2).

[0112]

[0113] Where θ0 represents the initial loss weights, μ′ is the weight change parameter value, representing the rate of change of the loss weights with the number of iterations, and p represents the number of iterations corresponding to the current training phase. stopp represents the number of iterations at which knowledge distillation training stops. stop The number of iterations can be directly specified, or it can be determined by the fact that the loss no longer decreases on the validation set.

[0114] In this embodiment, the corresponding temperature parameter is determined by a preset correlation formula between the temperature parameter and the number of iterations. As the number of iterations increases from a first preset value to a second preset value, the temperature parameter first increases and then decreases. During the knowledge distillation training process, the amount of information transmitted by the second speech recognition model first increases and then decreases, while the information reliability first decreases and then increases. This allows the second speech recognition model to simulate a distillation process in real life while teaching the first speech recognition model, first "heating up and distilling" and then "cooling down and condensing," achieving adaptive knowledge distillation training. This enables the second speech recognition model to learn better, has better robustness, and improves the knowledge distillation training effect, thereby effectively improving the recognition accuracy of the trained first speech recognition model. It balances the recognition efficiency and recognition accuracy of the first speech recognition model, making it applicable to various application scenarios such as human-vehicle interaction and improving the user experience.

[0115] In one embodiment, such as Figure 6 As shown, the method for obtaining the second speech recognition model includes:

[0116] Step S610: Obtain a set of sample speech signals, the set including multiple sample speech signals labeled with speech tags;

[0117] Step S620: Construct an initial speech recognition model, wherein training parameters are set in the initial speech recognition model;

[0118] Step S630: Input the sample speech signal into the initial speech recognition model to generate a recognition result;

[0119] Step S640: Based on the difference between the recognition result and the labeled speech tags, the initial speech recognition model is iteratively adjusted until the difference meets the preset requirements, thereby obtaining the second speech recognition model.

[0120] In this embodiment of the disclosure, during the training of the second speech recognition model, a set of sample speech signals is acquired. This set includes multiple sample speech signals labeled with speech tags, where the speech tags may include text tags corresponding to the sample speech signals. In one example, the set of sample speech signals can be obtained by processing the acquired audio data. Specifically, audio data is acquired, the acquired audio data is cleaned, valid audio segments are extracted, and data is labeled to obtain training audio codes and corresponding tag pairs. Feature extraction is performed on the obtained data to obtain acoustic features, and the acoustic features and tags are used to construct the set of sample speech signals. In one example, the feature extraction process may include, but is not limited to, pre-emphasis, windowing, accelerated Fourier transform, Mel filtering, etc. Preferably, the extracted audio features may include F-bank (Filter Bank) features. Since the human ear's response to the sound spectrum is non-linear, F-bank feature extraction processes audio in a manner similar to the human ear, which can improve the performance of speech recognition. The general steps for obtaining the F-bank features of a speech signal are: pre-emphasis, framing, windowing, short-time Fourier transform (STFT), Mel filtering, etc. In one example, when the application scenario is a human-vehicle interaction scenario, audio data can be collected from the actual vehicle. This audio data can include, but is not limited to, audio data from multiple interaction scenarios such as vehicle control, navigation, media, and question-and-answer. An initial speech recognition model is constructed, which includes training parameters. This initial speech recognition model can be constructed according to the actual application scenario. In one example, the constructed initial speech recognition model can include an encoder and a decoder. The encoder can include, but is not limited to, a conformer encoder, and the decoder can include, but is not limited to, a CTC decoder, an attention-based decoder, etc., which can be selected according to the actual application scenario. The number of layers in the encoder and decoder can also be determined according to actual needs. In this embodiment, since the final trained second speech recognition model is used to train the first speech recognition model after knowledge distillation, the number of layers in the encoder and / or decoder can be set to be greater than the number of layers in the second speech recognition model when constructing the initial speech recognition model. This ensures the model accuracy and recognition accuracy of the second speech recognition model, thereby improving the recognition accuracy of the first speech recognition model. The sample speech signal is input into the initial speech recognition model to obtain the recognition result. Based on the difference between the recognition result and the labeled speech tag, the initial speech recognition model is iteratively adjusted until the difference between the output recognition result and the speech tag meets the preset requirements, thus obtaining the second speech recognition model. The preset requirements can be set in advance according to the actual application scenario; for example, the difference between the recognition result output by the adjusted initial speech recognition model and the speech tag is less than a preset threshold.In one example, the loss function value can be determined based on the difference between the recognition result and the speech label, combined with a preset loss function, and the initial speech recognition model can be iteratively adjusted based on the loss function value.

[0121] In this embodiment, a second speech recognition model is trained using a set of sample speech signals, which enables subsequent knowledge distillation training to obtain a first speech recognition model. This embodiment ensures the accuracy of the trained second speech recognition model, thereby ensuring the accuracy of the subsequently obtained first speech recognition model. It is applicable to various application scenarios such as human-vehicle interaction, improving the user experience.

[0122] In one embodiment, the initial speech recognition model includes an encoder, an intermediate decoder, a first decoder, and a second decoder. The step of inputting the sample speech signal into the initial speech recognition model to generate a recognition result includes:

[0123] The sample speech signal is input into the initial speech recognition model to generate a recognition result, wherein the recognition result includes a first sub-result output by the first decoder, a second sub-result output by the second decoder, and a third sub-result output by the intermediate decoder.

[0124] In this embodiment of the disclosure, when constructing the initial speech recognition model, the initial speech recognition model can be configured to include an encoder, an intermediate decoder, a first decoder, and a second decoder. The encoder can include multiple encoder layers, and the intermediate decoder is set within these layers. The output of the intermediate encoder layer is obtained through the intermediate decoder. In one example, to improve the recognition accuracy of the second speech recognition model, the first decoder and the second decoder can be configured to be based on different decoding principles. The model recognition result is obtained by combining the outputs of the first decoder and the second decoder. For example, the first decoder and the second decoder can be configured as a CTC decoder and an Attention-based decoder, respectively. In this embodiment of the disclosure, after the sample speech signal is input into the initial speech recognition model, a first sub-result, a second sub-result, and a third sub-result can be output through the first decoder, the second decoder, and the intermediate decoder. In subsequent iterative adjustments to the initial speech recognition model, parameters can be adjusted based on the differences between the first sub-result, the second sub-result, the third sub-result, and the speech label. In one possible implementation, the initial speech recognition model can be iteratively adjusted based on the first sub-difference between the first sub-result and the speech label, the second sub-difference between the second sub-result and the speech label, and the third sub-difference between the third sub-result and the speech label. In one example, a loss function can be set, which includes loss weights. The loss function value can be determined by the first sub-difference, the second sub-difference, the third sub-difference, and the corresponding loss weights.

[0125] In this embodiment of the disclosure, during the iterative adjustment process of training the second speech recognition model, the parameters are adjusted based on the output results of the intermediate decoder, the first decoder, and the second decoder. This allows for comprehensive consideration of the intermediate layer output and the final layer output when adjusting the model parameters. By regularizing the encoder layer training and upper layer parameters through the intermediate layer output of the encoder, the model's recognition performance is improved, thereby enhancing the model accuracy and precision of the second speech recognition model. This, in turn, improves the recognition accuracy of the first speech recognition model, making it suitable for various application scenarios such as human-vehicle interaction and enhancing the user experience.

[0126] In one embodiment, the encoder includes a basic encoder layer and a folded encoder layer, and the intermediate decoder is disposed between the basic encoder layer and the folded encoder layer. The folded encoder layer includes multiple folded encoders, which share parameters according to a preset division method, wherein the preset division method is determined based on a preset number of repetitions and the number of folded encoder layers.

[0127] In this embodiment, the encoder layer includes a basic encoder layer and a folded encoder layer, with an intermediate decoder positioned between them. The folded encoder layer is configured in parameter-sharing mode. Specifically, the folded encoder layer includes multiple folded encoders, with a preset number of layers sharing parameters. The preset number of layers can be determined by a preset partitioning method, which can be based on a preset number of repetitions and the number of folded encoder layers. In one example, the multiple folded encoder can be divided into multiple folded encoder modules based on the ratio between the number of folded encoder layers and the preset number of repetitions. Each module includes a preset number of folded encoder layers, and the folded encoders within each module share parameters. For instance, when the number of folded encoder layers is 9 and the preset number of repetitions is set to 3, the ratio is 3. In this case, the 9-layer folded encoder layer can be divided into 3 folded encoder modules, each module containing 3 folded encoders, with the 3 folded encoders within each module sharing parameters. During model iterative training, the parameters of the parameter-sharing folded encoders are simultaneously changed and adjusted. In one example, an intermediate decoder is positioned between the base encoder layer and the fold encoder layer. During data transmission, the parameters of the fold encoder layer can be adjusted based on the output of the intermediate decoder.

[0128] In this embodiment of the disclosure, by setting a basic encoder layer and a folded encoder layer, parameter sharing of the folded encoder layer can be achieved, reducing the number of model parameters. The parameters of the repeated layers are repeatedly standardized, reducing the amount of computation during model training. Thus, while ensuring the model training effect, the flexibility and efficiency of model training can be improved.

[0129] Figure 7 This is a schematic diagram illustrating the structure of a second speech recognition model according to an exemplary embodiment, with reference to... Figure 7 As shown, the main architecture of the second speech recognition model consists of three parts: a shared encoder, a CTC (Connectivity-Temporal Classification) decoder, and an Attention-based decoder. Figure 7 As shown, the linear layer's functions include data scaling. For example, when data passes through a convolutional module and the output is (32, 86, 4350), and the Conformer module in this embodiment requires 256 units of data as input, a linear layer can transform (32, 86, 4350) into (32, 86, 256). The linear layer is also used for feature enhancement, extracting more refined features. For example, in the intermediate CTC layer of the folded encoder, when the vocab size is 5000, a linear layer can transform 256 into 5000, and also transform 5000 back into 256. In one example, during training, the input sample speech signal is first subjected to speech enhancement processing, and then feature sampling is performed through a convolutional model. Since speech feature sequences are generally quite long, two convolutional operations with a stride of 2 are typically performed before feeding them into the model to reduce the sequence to 1 / 4 of its original length. The shared encoder primarily consists of n Conformer layers. In this embodiment, a folded, self-adjusting intermediate layer CTC output is used in the shared encoder to regulate the training of the base encoder layer and the parameters of the upper layers, enabling the model to perform better. In one example, the attention-based decoder can be composed of m identical Transformer or Bitransformer layers. In this embodiment, the shared encoder layer includes N layers. b The base encoder layer consists of multiple identical Conformer encoder layers, and N... f The fold encoder layer consists of layers of folded encoders, where b + f = n. The folded encoder layer is configured to share parameters, and the number of parameter sharing repetitions can be set to C times. For the input acoustic features...

[0130] The information flow of X(x1,x2,x...x(frame_len), where frame_len represents the frame length) in the encoder can be briefly described as follows: The input is X, which passes through a base encoder layer BaseEncoders to obtain an intermediate state M, and then passes through a fold encoder layer FoldEncoders with a self-adjusting intermediate layer CTC. Compared with BaseEncoders, FoldEncoders adds a self-adjusting intermediate layer CTC to regulate the lower layer parameters and adjust the features input to the upper layer, and finally obtains the output Z.

[0131] M = BaseEncoder(X) (3)

[0132] Z k =FoldEncoder(M) (4)

[0133] For the i-th Conformer Block (Conformer module) feature in BaseEncoders or FoldEncoders, input from layer i... Change to It can be represented as follows:

[0134]

[0135] X″ i =X′ i +MHSA(X′ i (6)

[0136] X″′ i =X″ i +Conv(X″ i ) (7)

[0138]

[0139] Where FFN represents the position feedforward module, MHSA represents the multi-head attention module, Conv represents the convolution module, and each layer is preceded by a normalization layer.

[0140] The features input from BaseEncoders to FoldEncoders, and the beginning of the repeating modules in FoldEncoders, pass through a self-adjusting intermediate layer (CTC). For the first Conformer Block input of FoldEncoder... It is expressed as follows:

[0141] I1 = Softmax(Linear) D→V′ (M)) (9)

[0142]

[0143] According to the model structure, in the process of training the second speech recognition model, the training loss is optimized by iteratively adjusting the parameters. The training loss to be optimized consists of three loss functions, including the loss function corresponding to the output of the CTC decoder as shown in Equation (11), the loss function corresponding to the output of the Attention-based decoder as shown in Equation (12), and the loss function corresponding to the output of the intermediate CTC layer as shown in Equation (13).

[0144] Loss ctc =log∑ A∈θ P ctc (A|X) (11)

[0145] Where θ represents the set of all tags that can be obtained by deduplication and removal of spaces;

[0146]

[0147] Where L is the total length of the decoded sequence, and X represents the high-level features output by the decoder;

[0148] (13)

[0150] Where C represents the number of repetitions.

[0151] Therefore, the training loss that needs to be optimized during the iterative adjustment process is shown in Equation (14).

[0152] Loss = α * Loss ctc +β*Loss att +γ*Loss Inter-ctc (14)

[0154] Here, α, β, and γ are hyperparameters, i.e., loss weights, used to measure the importance of different loss functions. In one example, α + β + γ = 1.

[0155] Figure 8 This is a schematic diagram illustrating the structure of a speech recognition system according to an exemplary embodiment, with reference to... Figure 8As shown, audio data is first acquired by a signal acquisition unit and then processed by a signal processing unit to obtain valid audio segments. Feature extraction is performed on the processed audio data. The feature-extracted audio data is then input as the speech signal to be recognized into a first speech recognition model. After speech recognition by the first speech recognition model, a recognition result is obtained. The recognition result is then displayed according to a preset presentation method. In one example, the recognition result can be the text corresponding to the speech signal to be recognized. When displaying the recognition result, it can be converted into a sound signal, image signal, etc., and presented on a preset terminal. The preset presentation method may include, but is not limited to, displaying text on a screen, or playing the sound signal corresponding to the recognition result on a voice terminal (such as a broadcasting device).

[0156] Figure 9 This is a schematic diagram illustrating the structure of a speech recognition system according to an exemplary embodiment, with reference to... Figure 9 As shown, the speech recognition method described in this disclosure can be applied to human-vehicle interaction scenarios. The first speech recognition model can be set up on the vehicle side for speech recognition. In this embodiment, the speech recognition system includes an external interaction module, a processor, a memory, and a display screen. The first speech recognition model is set in the processor. During the speech recognition process, the processor receives the speech signal input from the external interaction module, performs recognition, and obtains the recognition result in real time. The display screen receives and displays the recognition result. The memory is connected to the processor and is used to store the data generated in the speech recognition method. In human-vehicle interaction scenarios, specific areas may include, but are not limited to, vehicle control, media, video, navigation, children, and Chinese / English languages. In one example, as shown in Table 1, different recognition models exhibit varying recognition accuracies in human-vehicle interaction scenarios. Specifically, recognition model A is a complex and large-scale speech recognition model. Although it has high accuracy, its large computational load requires significant memory and is therefore unsuitable for vehicle-side applications. Recognition model X is a less complex and smaller-scale speech recognition model. While it consumes less memory, its recognition accuracy is poor. Recognition model Y is a speech recognition model obtained by training recognition model A using knowledge distillation with fixed temperature parameters. It has low complexity, but its recognition accuracy is not stable enough. Recognition model Z is a speech recognition model obtained by training recognition model A using knowledge distillation through the method described in this disclosure. Because the temperature parameters during the knowledge distillation training process adapt to the number of iterations, the resulting speech recognition model has high recognition accuracy, low complexity, small size, low computational load, and high recognition efficiency.

[0157] Table 1

[0158]

[0159] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0160] Based on the same inventive concept, this disclosure also provides a speech recognition apparatus for implementing the speech recognition method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, specific limitations in one or more speech recognition apparatus embodiments provided below can be found in the limitations of the speech recognition method described above, and will not be repeated here.

[0161] In one embodiment, such as Figure 10 As shown, a speech recognition device 1000 is provided, comprising:

[0162] The receiving module 1010 is used to receive the voice signal to be recognized;

[0163] The output module 1020 is used to input the speech signal to be recognized into the first speech recognition model and output the recognition result through the first speech recognition model. The first speech recognition model is obtained by knowledge distillation training through the second speech recognition model. There is a preset correlation between the temperature parameter in the knowledge distillation training process and the number of iterations corresponding to the training stage of the first speech recognition model.

[0164] In one embodiment, the module for acquiring the first speech recognition model includes:

[0165] The first construction module is used to construct an initial first speech recognition model, wherein the initial first speech recognition model is configured with training parameters;

[0166] The first input module is used to input the training speech signal into the initial first speech recognition model, and output the initial first recognition result through the initial first speech recognition model.

[0167] The second input module is used to input the training speech signal into the second speech recognition model, and output the initial second recognition result through the second speech recognition model;

[0168] The processing module is used to process the initial first recognition result according to the temperature parameter matching the number of iterations corresponding to the current training stage to obtain a first recognition result, and to process the initial second recognition result according to the temperature parameter to obtain a second recognition result;

[0169] The determination module is used to determine the loss function value based on the difference between the first recognition result and the second recognition result, and to iteratively adjust the initial first speech recognition model according to the loss function value until a preset condition is met to obtain the first speech recognition model.

[0170] In one embodiment, the determining module includes:

[0171] The acquisition submodule is used to acquire the initial first intermediate result output by the first preset intermediate layer of the initial first speech recognition model and the initial second intermediate result output by the second preset intermediate layer of the second speech recognition model.

[0172] The processing submodule is used to process the initial first intermediate result according to the intermediate temperature parameter that matches the number of iterations corresponding to the current training stage to obtain the first intermediate result, and to process the initial second intermediate result according to the intermediate temperature parameter to obtain the second intermediate result.

[0173] The first determining submodule is used to determine the loss function value based on the first difference between the first identification result and the second identification result, and the second difference between the first intermediate result and the second intermediate result.

[0174] In one embodiment, the determining module includes:

[0175] The second determination submodule is used to determine the loss weights that match the number of iterations corresponding to the current training phase;

[0176] The third determining submodule is used to determine the loss function value based on the difference between the first identification result and the second identification result and the loss weight.

[0177] In one embodiment, the temperature parameter determination module includes:

[0178] The first acquisition module is used to acquire the correlation formula between temperature parameters and iteration number, wherein the correlation formula includes temperature change parameter value and initial temperature parameter value, wherein as the iteration number increases from a first preset value to a second preset value, the temperature parameter first increases and then decreases.

[0179] The substitution module is used to substitute the number of iterations corresponding to the current training stage into the correlation formula to obtain the temperature parameters corresponding to the current training stage.

[0180] In one embodiment, the module for acquiring the second speech recognition model includes:

[0181] The second acquisition module is used to acquire a set of sample speech signals, the set including multiple sample speech signals labeled with speech tags;

[0182] The second construction module is used to construct an initial speech recognition model, in which training parameters are set;

[0183] The generation module is used to input the sample speech signal into the initial speech recognition model and generate a recognition result;

[0184] An adjustment module is used to iteratively adjust the initial speech recognition model based on the difference between the recognition result and the labeled speech tags until the difference meets the preset requirements, thereby obtaining a second speech recognition model.

[0185] In one embodiment, the initial speech recognition model includes an encoder, an intermediate decoder, a first decoder, and a second decoder; the generation module includes:

[0186] A generation submodule is used to input the sample speech signal into the initial speech recognition model and generate a recognition result, wherein the recognition result includes a first sub-result output by the first decoder, a second sub-result output by the second decoder, and a third sub-result output by the intermediate decoder.

[0187] In one embodiment, the encoder includes a basic encoder layer and a folded encoder layer, and the intermediate decoder is disposed between the basic encoder layer and the folded encoder layer. The folded encoder layer includes multiple folded encoders, which share parameters according to a preset division method, wherein the preset division method is determined based on a preset number of repetitions and the number of folded encoder layers.

[0188] The modules in the aforementioned speech recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0189] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 11As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data such as the speech signal to be recognized and the recognition results. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a speech recognition method.

[0190] Those skilled in the art will understand that Figure 11 The structures shown are merely block diagrams of some structures related to the embodiments of this disclosure and do not constitute a limitation on the computer devices on which the embodiments of this disclosure are applied. Specific computer devices may include more or fewer components than those shown in the figures, or combine certain components, or have different component arrangements.

[0191] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0192] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0193] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0194] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0195] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this disclosure may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this disclosure may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0196] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0197] The above-described embodiments are merely illustrative of several implementation methods of the present disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the patent for the embodiments of the present disclosure. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of the present disclosure, and these all fall within the protection scope of the embodiments of the present disclosure. Therefore, the protection scope of the embodiments of the present disclosure should be determined by the appended claims.

Claims

1. A method of speech recognition, characterized by, The method includes: Receive the voice signal to be recognized; The speech signal to be recognized is input into a first speech recognition model, which outputs a recognition result. The first speech recognition model is obtained by knowledge distillation training through a second speech recognition model. A preset correlation exists between the temperature parameter during the knowledge distillation training process and the number of iterations corresponding to the training phase of the first speech recognition model. After the knowledge distillation training process begins, the temperature parameter first increases and then decreases as the number of iterations increases. During knowledge distillation, the initial first recognition result output by the initial first speech recognition model and the initial second recognition result output by the second speech recognition model are processed using the temperature parameter corresponding to the training phase. A loss function value is determined based on the difference between the processed first and second recognition results, and the initial first speech recognition model is iteratively adjusted according to the loss function value.

2. The method of claim 1, wherein, The first speech recognition model is obtained by knowledge distillation training on the second speech recognition model, and includes: Construct an initial first speech recognition model, wherein training parameters are set in the initial first speech recognition model; The training speech signal is input into the initial first speech recognition model, and the initial first speech recognition model outputs the initial first recognition result. The training speech signal is input into the second speech recognition model, and the second speech recognition model outputs the initial second recognition result. The initial first recognition result is obtained by processing the temperature parameter that matches the number of iterations corresponding to the current training stage; the initial second recognition result is obtained by processing the temperature parameter. The loss function value is determined based on the difference between the first recognition result and the second recognition result. The initial first speech recognition model is iteratively adjusted according to the loss function value until the preset conditions are met, and the first speech recognition model is obtained.

3. The method of claim 2, wherein, The step of determining the loss function value based on the difference between the first identification result and the second identification result includes: Obtain the initial first intermediate result output by the first preset intermediate layer of the initial first speech recognition model and the initial second intermediate result output by the second preset intermediate layer of the second speech recognition model; The initial first intermediate result is obtained by processing the intermediate temperature parameter that matches the number of iterations corresponding to the current training stage; the second intermediate result is obtained by processing the initial second intermediate result according to the intermediate temperature parameter. The loss function value is determined based on the first difference between the first identification result and the second identification result, and the second difference between the first intermediate result and the second intermediate result.

4. The method of claim 2, wherein, The step of determining the loss function value based on the difference between the first identification result and the second identification result includes: Determine the loss weights that match the number of iterations corresponding to the current training phase; The loss function value is determined based on the difference between the first identification result and the second identification result, and the loss weight.

5. The method of claim 1, wherein, The method for determining the temperature parameter includes: Obtain the correlation formula between temperature parameter and iteration number, wherein the correlation formula includes temperature change parameter value and initial temperature parameter value, wherein as the iteration number increases from a first preset value to a second preset value, the temperature parameter first increases and then decreases; Substituting the number of iterations corresponding to the current training phase into the correlation formula yields the temperature parameter corresponding to the current training phase.

6. The method of claim 1, wherein, The methods for obtaining the second speech recognition model include: Obtain a set of sample speech signals, the set including multiple sample speech signals labeled with speech tags; Construct an initial speech recognition model, wherein training parameters are set in the initial speech recognition model; The sample speech signal is input into the initial speech recognition model to generate a recognition result; Based on the difference between the recognition result and the labeled speech tags, the initial speech recognition model is iteratively adjusted until the difference meets the preset requirements, thus obtaining the second speech recognition model.

7. The method of claim 6, wherein, The initial speech recognition model includes an encoder, an intermediate decoder, a first decoder, and a second decoder. The step of inputting the sample speech signal into the initial speech recognition model to generate a recognition result includes: The sample speech signal is input into the initial speech recognition model to generate a recognition result, wherein the recognition result includes a first sub-result output by the first decoder, a second sub-result output by the second decoder, and a third sub-result output by the intermediate decoder.

8. The method of claim 7, wherein, The encoder includes a basic encoder layer and a folded encoder layer. The intermediate decoder is located between the basic encoder layer and the folded encoder layer. The folded encoder layer includes multiple folded encoders. The multiple folded encoders share parameters according to a preset division method, wherein the preset division method is determined based on a preset number of repetitions and the number of folded encoder layers.

9. An apparatus for speech recognition, characterized by The device includes: The receiving module is used to receive the voice signal to be recognized; The output module is used to input the speech signal to be recognized into a first speech recognition model, and output a recognition result through the first speech recognition model. The first speech recognition model is obtained by knowledge distillation training through a second speech recognition model. There is a preset correlation between the temperature parameter in the knowledge distillation training process and the number of iterations corresponding to the training stage of the first speech recognition model. In the knowledge distillation process, the initial first recognition result output by the initial first speech recognition model and the initial second recognition result output by the second speech recognition model are processed by the temperature parameter corresponding to the training stage. The loss function value is determined based on the difference between the processed first recognition result and the second recognition result, and the initial first speech recognition model is iteratively adjusted according to the loss function value.

10. The apparatus of claim 9, wherein, The module for acquiring the first speech recognition model includes: The first construction module is used to construct an initial first speech recognition model, wherein the initial first speech recognition model is configured with training parameters; The first input module is used to input the training speech signal into the initial first speech recognition model, and output the initial first recognition result through the initial first speech recognition model. The second input module is used to input the training speech signal into the second speech recognition model, and output the initial second recognition result through the second speech recognition model; The processing module is used to process the initial first recognition result according to the temperature parameter matching the number of iterations corresponding to the current training stage to obtain a first recognition result, and to process the initial second recognition result according to the temperature parameter to obtain a second recognition result; The determination module is used to determine the loss function value based on the difference between the first recognition result and the second recognition result, and to iteratively adjust the initial first speech recognition model according to the loss function value until a preset condition is met to obtain the first speech recognition model.

11. The apparatus of claim 10, wherein, The determining module includes: The acquisition submodule is used to acquire the initial first intermediate result output by the first preset intermediate layer of the initial first speech recognition model and the initial second intermediate result output by the second preset intermediate layer of the second speech recognition model. The processing submodule is used to process the initial first intermediate result according to the intermediate temperature parameter matching the number of iterations corresponding to the current training stage to obtain the first intermediate result, and to process the initial second intermediate result according to the intermediate temperature parameter to obtain the second intermediate result. The first determining submodule is used to determine the loss function value based on the first difference between the first identification result and the second identification result, and the second difference between the first intermediate result and the second intermediate result.

12. The apparatus of claim 10, wherein, The determining module includes: The second determination submodule is used to determine the loss weights that match the number of iterations corresponding to the current training phase; The third determining submodule is used to determine the loss function value based on the difference between the first identification result and the second identification result and the loss weight.

13. The apparatus of claim 9, wherein, The module for determining the temperature parameter includes: The first acquisition module is used to acquire the correlation formula between temperature parameters and iteration number, wherein the correlation formula includes temperature change parameter value and initial temperature parameter value, wherein as the iteration number increases from a first preset value to a second preset value, the temperature parameter first increases and then decreases. The substitution module is used to substitute the number of iterations corresponding to the current training stage into the correlation formula to obtain the temperature parameters corresponding to the current training stage.

14. The apparatus of claim 9, wherein, The module for acquiring the second speech recognition model includes: The second acquisition module is used to acquire a set of sample speech signals, the set including multiple sample speech signals labeled with speech tags; The second construction module is used to construct an initial speech recognition model, in which training parameters are set; The generation module is used to input the sample speech signal into the initial speech recognition model and generate a recognition result; An adjustment module is used to iteratively adjust the initial speech recognition model based on the difference between the recognition result and the labeled speech tags until the difference meets the preset requirements, thereby obtaining a second speech recognition model.

15. The apparatus of claim 14, wherein, The initial speech recognition model includes an encoder, an intermediate decoder, a first decoder, and a second decoder. The generation module includes: A generation submodule is used to input the sample speech signal into the initial speech recognition model to generate a recognition result, wherein the recognition result includes a first sub-result output by the first decoder, a second sub-result output by the second decoder, and a third sub-result output by the intermediate decoder.

16. The apparatus of claim 15, wherein, The encoder includes a basic encoder layer and a folded encoder layer. The intermediate decoder is located between the basic encoder layer and the folded encoder layer. The folded encoder layer includes multiple folded encoders. The multiple folded encoders share parameters according to a preset division method, wherein the preset division method is determined based on a preset number of repetitions and the number of folded encoder layers. 17.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-16. When the processor executes the computer program, it implements the steps of the speech recognition method according to any one of claims 1 to 8.

18. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the speech recognition method according to any one of claims 1 to 8.

19. A computer program product comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the speech recognition method according to any one of claims 1 to 8.

Images