Speech recognition model training method, speech recognition method and equipment

Abstract

The invention provides a voice recognition model training method, a voice recognition method and equipment, and the method comprises the steps: training a first voice recognition model which comprises a CTC decoder and an attention decoder in combination with the sample voice data of an air traffic control field and corresponding annotation text information, according to the invention, the second speech recognition model adaptive to the air traffic control field is rapidly obtained, the efficiency of obtaining the second speech recognition model adaptive to the air traffic control field is improved, and the recognition accuracy and generalization ability of the obtained speech recognition model are improved.

Description

Technical Field

[0001] The present application relates to the field of air traffic control technology, and in particular to a training method for a speech recognition model, a speech recognition method and a device. Background Art

[0002] At present, in the field of air traffic control technology, a large amount of air traffic control voice data is usually involved in the process of air-to-ground communication. In related technologies, air traffic controllers usually manually determine the communication content of air traffic control voice data and perform subsequent processing based on the communication content. However, since manual determination of the communication content of air traffic control voice data has the disadvantages of low efficiency and prone to errors, how to obtain a speech recognition model adapted to the air traffic control field is very important for improving the processing efficiency and accuracy of air traffic control voice data in the air traffic control field. Summary of the invention

[0003] The present application proposes a speech recognition model training method, a speech recognition method and a device.

[0004] In one aspect, an embodiment of the present application proposes a method for training a speech recognition model, the method comprising: obtaining training data, wherein the training data comprises: first sample speech data in the field of air traffic control and corresponding first annotated text information; inputting the first sample speech data into a shared encoder in the first speech recognition model to obtain an encoding result of the first sample speech data; inputting the encoding result into a connection temporal classification (CTC) decoder in the first speech recognition model to obtain a plurality of candidate text sequences and respective corresponding first scores; inputting the plurality of candidate text sequences and the encoding result into an attention decoder in the first speech recognition model to obtain respective corresponding second scores of the candidate text sequences; inputting the plurality of candidate text sequences and the first scores and the second scores into an output layer of the first speech recognition model, so that the output layer determines a predicted text recognition result of the first sample speech data from the plurality of candidate text sequences; and training the first speech recognition model according to the first annotated text information and the predicted text recognition result to obtain a second speech recognition model adapted to the field of air traffic control.

[0005] The training method of the speech recognition model of the embodiment of the present application, after obtaining training data, inputs the first sample speech data into the shared encoder in the first speech recognition model to obtain the encoding result of the first sample speech data, and inputs the encoding result into the connection temporal classification CTC decoder in the first speech recognition model to obtain multiple candidate text sequences and their respective corresponding first scores; inputs the multiple candidate text sequences and the encoding results into the attention decoder in the first speech recognition model to obtain the second scores corresponding to the candidate text sequences, and inputs the multiple candidate text sequences and the first scores and the second scores into the output layer of the first speech recognition model, so that the output layer determines the predicted text recognition result of the first sample speech data from the multiple candidate text sequences, and trains the first speech recognition model according to the first annotated text information and the predicted text recognition result, so as to obtain a second speech recognition model adapted to the air traffic control field. Therefore, combined with the sample speech data in the air traffic control field and the corresponding annotated text information, the first speech recognition model including the CTC decoder and the attention decoder is trained, and a second speech recognition model adapted to the air traffic control field is quickly obtained. This improves the efficiency of obtaining the second speech recognition model adapted to the air traffic control field while improving the recognition accuracy and generalization ability of the obtained speech recognition model.

[0006] On the other hand, an embodiment of the present application proposes a speech recognition method, which includes: obtaining air traffic control speech data to be processed; inputting the air traffic control speech data into a second speech recognition model to obtain a target text recognition result of the air traffic control speech data, and the second speech recognition model is trained by the training method of the speech recognition model disclosed in the present application.

[0007] The speech recognition method of the embodiment of the present application processes the air traffic control speech data to be processed by a second speech recognition model adapted to the air traffic control field, thereby improving the efficiency and accuracy of the target text recognition results of the air traffic control speech data.

[0008] On the other hand, an embodiment of the present application proposes a training device for a speech recognition model, the device comprising: an acquisition module for acquiring training data, wherein the training data comprises: first sample speech data in the field of air traffic control and corresponding first annotated text information; an encoding module for inputting the first sample speech data into a shared encoder in the first speech recognition model to obtain an encoding result of the first sample speech data; a first decoding module for inputting the encoding result into a connection temporal classification (CTC) decoder in the first speech recognition model to obtain a plurality of candidate text sequences and respective corresponding first scores; a second decoding module for inputting the plurality of candidate text sequences and the encoding result into an attention decoder in the first speech recognition model to obtain respective corresponding second scores of the candidate text sequences; a recognition result determination module for inputting the plurality of candidate text sequences and the first scores and the second scores into the output layer of the first speech recognition model, so that the output layer determines the predicted text recognition result of the first sample speech data from the plurality of candidate text sequences; and a model training module for training the first speech recognition model according to the first annotated text information and the predicted text recognition result to obtain a second speech recognition model adapted to the field of air traffic control.

[0009] The training device of the speech recognition model of the embodiment of the present application, after acquiring training data, inputs the first sample speech data into the shared encoder in the first speech recognition model to obtain the encoding result of the first sample speech data, and inputs the encoding result into the connection temporal classification CTC decoder in the first speech recognition model to obtain multiple candidate text sequences and their respective corresponding first scores; inputs the multiple candidate text sequences and the encoding results into the attention decoder in the first speech recognition model to obtain the second scores corresponding to the candidate text sequences, and inputs the multiple candidate text sequences and the first scores and the second scores into the output layer of the first speech recognition model, so that the output layer determines the predicted text recognition result of the first sample speech data from the multiple candidate text sequences, and trains the first speech recognition model according to the first annotated text information and the predicted text recognition result, so as to obtain a second speech recognition model adapted to the air traffic control field. Therefore, combined with the sample speech data in the air traffic control field and the corresponding annotated text information, the first speech recognition model including the CTC decoder and the attention decoder is trained, and a second speech recognition model adapted to the air traffic control field is quickly obtained. This improves the efficiency of obtaining the second speech recognition model adapted to the air traffic control field while improving the recognition accuracy and generalization ability of the obtained speech recognition model.

[0010] On the other hand, an embodiment of the present application proposes a speech recognition device, which includes: an acquisition module for acquiring air traffic control speech data to be processed; a recognition result determination module for inputting the air traffic control speech data into a second speech recognition model to obtain a target text recognition result of the air traffic control speech data, and the second speech recognition model is trained by the training method of the speech recognition model disclosed in the embodiment of the present application.

[0011] The speech recognition device of the embodiment of the present application processes the air traffic control speech data to be processed by a second speech recognition model adapted to the air traffic control field, thereby improving the efficiency and accuracy of the target text recognition results of the air traffic control speech data.

[0012] On the other hand, an embodiment of the present application proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the training method of the speech recognition model of the embodiment of the present application or the speech recognition method is implemented.

[0013] On the other hand, an embodiment of the present application proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements the training method of the speech recognition model or the speech recognition method of the embodiment of the present application.

[0014] On the other hand, an embodiment of the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the training method of the speech recognition model or the speech recognition method of the embodiment of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0016] Figure 1 is a flowchart of a method for training a speech recognition model according to an embodiment of the present application;

[0017] Figure 2 is an example diagram of the internal structure of a first speech recognition model according to an embodiment of the present application;

[0018] Figure 3 is a flowchart of a method for training a speech recognition model according to another embodiment of the present application;

[0019] Figure 4 A flowchart of a method for training a speech recognition model according to another embodiment of the present application;

[0020] Figure 5 is a flow chart of a speech recognition method according to an embodiment of the present application;

[0021] Figure 6 is a structural schematic diagram of a training device for a speech recognition model according to an embodiment of the present application;

[0022] Figure 7 is a structural schematic diagram of a speech recognition device according to an embodiment of the present application;

[0023] Figure 8 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0025] The following describes the speech recognition model training method, speech recognition method, device, electronic device and storage medium of the embodiments of the present application with reference to the accompanying drawings.

[0026] Figure 1 It is a flowchart of a method for training a speech recognition model according to an embodiment of the present application.

[0027] It should be noted that the training method of the speech recognition model provided in this embodiment is executed by a training device for the speech recognition model. The training device for the speech recognition model in this embodiment can be implemented by software and / or hardware. The training device for the speech recognition model can be an electronic device, or can be configured in an electronic device.

[0028] The electronic devices in this example embodiment may include terminal devices, servers, etc., wherein the terminal devices may be PCs (Personal Computers), mobile devices, tablet computers, etc., and this embodiment does not specifically limit this.

[0029] like Figure 1 As shown, the training method of the speech recognition model may include:

[0030] Step 101, obtaining training data, wherein the training data includes: first sample voice data in the field of air traffic control and corresponding first annotated text information.

[0031] In one embodiment of the present disclosure, in order to improve the accuracy of the second speech recognition model trained subsequently, the first annotated text information of the first sample speech data in this embodiment corresponds to the first sample speech data, and can be obtained by manual text annotation of the first sample speech data.

[0032] Among them, the air traffic control field, namely the air traffic control field, is a vital link in the civil aviation industry. It is responsible for monitoring and managing the flight activities of aircraft to ensure safe and orderly air traffic.

[0033] It can be understood that the first sample voice data in the air traffic control field in this embodiment is the sample air traffic control voice data.

[0034] The sample air traffic control voice data is obtained from the historical voice data in the air traffic control field.

[0035] As an example, historical air-ground call voice data in the field of air traffic control can be used as the first sample voice data.

[0036] Step 102: input the first sample speech data into a shared encoder in a first speech recognition model to obtain an encoding result of the first sample speech data.

[0037] In this embodiment, the first sample speech data may be input into a shared encoder in the first speech recognition model, so that the first sample speech data may be encoded by the shared encoder to obtain an encoding result of the first sample speech data.

[0038] It can be understood that, in some embodiments, the second speech recognition model in this embodiment can be a speech recognition model that has not been pre-trained. That is, in some embodiments, the speech recognition model that has not been pre-trained can be directly trained based on the first sample speech data in the field of air traffic control and the corresponding first annotated text information to obtain the second speech recognition model that is suitable for the field of air traffic control.

[0039] In one embodiment of the present application, in order to improve the recognition accuracy and generalization ability of the second speech recognition model obtained subsequently, the first speech recognition model in this embodiment can be obtained by pre-training the initial speech recognition model based on the second sample speech data in the general field and the corresponding second annotated text information. That is to say, in some embodiments, the initial speech recognition model can be trained according to the second sample speech data in the general field and the corresponding second annotated text information to obtain the first speech recognition model, and the first speech recognition model can be trained in combination with the first sample speech data in the air traffic control field and the corresponding second annotated text information to obtain the second speech recognition model adapted to the air traffic control field. Thus, through this pre-training method, the first speech recognition model already has strong speech recognition ability and robustness, and then fine-tuning the first speech recognition model in combination with the first sample speech data in the air traffic control field and the corresponding first annotated text information can improve the efficiency of obtaining the second speech recognition model adapted to the air traffic control field, while effectively improving the recognition accuracy and generalization ability of the second speech recognition model.

[0040] Among them, it should be noted that the character error rate (CER) of the second speech recognition model obtained by directly training the non-pre-trained speech recognition model is greater than the CER of the second speech recognition model obtained by training the pre-trained speech recognition model based on the training data in the air traffic control field.

[0041] Step 103: Input the encoding result into a Connectionist Temporal Classification (CTC) decoder in the first speech recognition model to obtain a plurality of candidate text sequences and their respective corresponding first scores.

[0042] In this embodiment, the encoding result of the first sample speech data can be input into the CTC decoder. Correspondingly, the CTC decoder performs text sequence prediction on the encoding result based on the prefix beam search algorithm to obtain the best multiple candidate text sequences and the corresponding first scores.

[0043] The first score is obtained according to the probability that the candidate text sequence is the final text recognition result of the first sample speech data.

[0044] It can be understood that the higher the probability that a candidate text sequence is the final text recognition result of the first sample speech data, the higher the first score of the candidate text sequence, and vice versa.

[0045] Step 104, inputting the multiple candidate text sequences and the encoding results into the attention decoder in the first speech recognition model to obtain the second scores corresponding to each candidate text sequence.

[0046] In one embodiment of the present application, a plurality of candidate text sequences and encoding results are input into an attention decoder in a first speech recognition model to obtain a second score corresponding to each candidate text sequence. One possible implementation method may be: in the attention decoder in the first speech recognition model, based on the encoding results, determining the attention weights corresponding to each of the plurality of candidate text sequences; and determining the second scores corresponding to each of the plurality of candidate text sequences based on the attention weights corresponding to each of the plurality of candidate text sequences.

[0047] It can be understood that the higher the attention weight of a candidate text sequence, the higher the corresponding second score, and vice versa.

[0048] Step 105: input the multiple candidate text sequences and the first score and the second score into the output layer of the first speech recognition model, so that the output layer determines the predicted text recognition result of the first sample speech data from the multiple candidate text sequences.

[0049] In this embodiment, in order to accurately obtain the predicted text recognition result, a possible implementation method of step 105 is: in the output layer of the first speech recognition model, the first score and the second score of each candidate text sequence are weighted and summed to obtain the comprehensive score corresponding to each candidate text sequence; from multiple candidate text sequences, the text sequence with the highest comprehensive score is selected as the predicted text recognition result of the first sample speech data.

[0050] In one embodiment of the present application, a preset first weight may be used to weight the first score to obtain a weighted first score, and the second score and the weighted first score may be summed to obtain a comprehensive score.

[0051] The calculation formula for calculating the comprehensive score A is:

[0052] A=λ1S1+S2

[0053] Among them, A in the formula represents the comprehensive score; λ1 represents the first weight; S1 represents the first score, and S2 represents the second score.

[0054] The first weight in this embodiment is a weight preset according to actual needs, and this embodiment does not specifically limit the value of the first weight.

[0055] Step 106, training the first speech recognition model according to the first annotated text information and the predicted text recognition result to obtain a second speech recognition model adapted to the air traffic control field.

[0056] In order to clearly understand this application, Figure 2 The internal structure of the first speech recognition model shown in FIG. 1 is used to exemplarily describe the process of obtaining the predicted text recognition result of the first sample speech data through the first speech recognition model. Figure 2It can be seen that the first speech recognition model in this embodiment may include: a shared encoder, a CTC decoder connected to the shared encoder, an attention decoder connected to the shared encoder, and an output layer connected to the attention encoder. Correspondingly, the first sample speech data is input into the shared encoder to encode the first sample speech data through the shared encoder to obtain the encoding result of the first sample speech data, and the encoding result of the first sample speech data is input into the CTC decoder, and the CTC decoder performs text sequence prediction on the encoding result based on the prefix beam search algorithm to obtain the best multiple candidate text sequences and the corresponding first scores, and the multiple candidate text sequences are input into the attention encoder, and correspondingly, the attention encoder performs attention scores on the multiple candidate text sequences according to the encoding results to obtain the second scores corresponding to each candidate text sequence, and correspondingly, each candidate text sequence and the first score and the second score corresponding to each candidate text sequence are input into the output layer to obtain the predicted text recognition result of the first sample speech data.

[0057] In one embodiment of the present application, in order to further improve the recognition accuracy and generalization ability of the obtained second speech recognition model, correspondingly, the first speech recognition model is trained according to the first annotated text information and the predicted text recognition result to obtain a second speech recognition model adapted to the air traffic control field. A possible implementation method is: according to a first loss function, a first loss value of the first annotated text information and the predicted text recognition result is determined, wherein the first loss function is used to represent the loss function of a structure composed of a shared encoder and a CTC decoder; according to a second loss function, a second loss value of the first annotated text information and the predicted text recognition result is determined, wherein the second loss function is used to represent the loss function of a structure composed of a shared encoder and an attention decoder; the first loss value and the second loss value are weightedly summed to obtain a total loss value; according to the total loss value, the first speech recognition model is trained to obtain a second speech recognition model adapted to the air traffic control field.

[0058] It should be noted that the first loss function in this embodiment can also be called a CTC loss function, and the second loss function can also be called an attention-based encoder-decoder (AED) loss function.

[0059] In this embodiment, a possible implementation method of weighted summing the first loss value and the second loss value to obtain the total loss value is: weighting the first loss value with a second weight corresponding to the first loss function to obtain a weighted first loss value, and weighting the second loss value with a third weight corresponding to the second loss function to obtain a weighted second loss value, and then summing the weighted first loss value and the second loss value to obtain the total loss value, wherein the sum of the second weight and the third weight is equal to 1.

[0060] The formula for the total loss value L is: L = λ2L1 + λ3L2

[0061] In the formula, λ3 represents the second weight; λ4 represents the third weight, wherein λ2+λ3=1, L1 represents the first loss value; and L2 represents the second loss value.

[0062] It should be noted that the second weight and the third weight in this embodiment are weights preset according to actual needs, and this embodiment does not specifically limit the first weight and the second weight.

[0063] A possible implementation method of training the first speech recognition model according to the total loss value to obtain the second speech recognition model adapted to the air traffic control field is: judging whether the total loss value meets the preset model training end condition; if the total loss value cannot meet the preset model training end condition, adjusting the model parameters of the first speech recognition model according to the total loss value, and continuing to train the adjusted first speech recognition model until the total loss value meets the preset model training end condition to obtain the second speech recognition model adapted to the air traffic control field.

[0064] The preset model training end condition can be configured accordingly according to actual needs. For example, the total loss value can meet the model training end condition when the total loss value is less than the preset value, or when the change of the total loss value tends to be stable, that is, the difference between the total loss values ​​corresponding to two or more consecutive trainings is less than the set value, that is, the total loss value basically does not change.

[0065] Among them, it can be understood that in the process of training the first speech recognition model through training data, the model parameters of the first speech recognition model are continuously adjusted according to the total loss value of each training. For example, the model parameters of the first speech recognition model can be adjusted in the direction of reducing the total loss value. When the total loss value meets the preset conditions, the second speech recognition model adapted to the air traffic control field is obtained.

[0066] It should be noted that the second speech recognition model in this embodiment combines the advantages of the CTC decoder and the attention decoder, thereby improving the recognition accuracy and generalization ability of the second speech recognition model.

[0067] The training method of the speech recognition model of the embodiment of the present application, after obtaining training data, inputs the first sample speech data into the shared encoder in the first speech recognition model to obtain the encoding result of the first sample speech data, and inputs the encoding result into the connection temporal classification CTC decoder in the first speech recognition model to obtain multiple candidate text sequences and their respective corresponding first scores; inputs the multiple candidate text sequences and the encoding results into the attention decoder in the first speech recognition model to obtain the second scores corresponding to the candidate text sequences, and inputs the multiple candidate text sequences and the first scores and the second scores into the output layer of the first speech recognition model, so that the output layer determines the predicted text recognition result of the first sample speech data from the multiple candidate text sequences, and trains the first speech recognition model according to the first annotated text information and the predicted text recognition result, so as to obtain a second speech recognition model adapted to the air traffic control field. Therefore, combined with the sample speech data in the air traffic control field and the corresponding annotated text information, the first speech recognition model including the CTC decoder and the attention decoder is trained, and a second speech recognition model adapted to the air traffic control field is quickly obtained. This improves the efficiency of obtaining the second speech recognition model adapted to the air traffic control field while improving the recognition accuracy and generalization ability of the obtained speech recognition model.

[0068] Figure 3 It is a flowchart of a method for training a speech recognition model according to another embodiment of the present application.

[0069] like Figure 3 As shown, the training method of the speech recognition model may include:

[0070] Step 301, obtaining training data, wherein the training data includes: first sample voice data in the field of air traffic control and corresponding first annotated text information.

[0071] It should be noted that, for the specific description of step 301, reference may be made to the relevant descriptions in other embodiments, which will not be repeated here.

[0072] Step 302: denoise the first sample speech data to obtain denoised first sample speech data.

[0073] In one embodiment of the present application, a possible implementation method of denoising the first sample speech data to obtain the denoised first sample speech data is: performing frame processing on the first sample speech data to obtain multiple speech frames of the first sample speech data; performing Fourier transform on the multiple speech frames respectively to obtain the first spectrum corresponding to each of the multiple speech frames; extracting the noise speech segment containing only noise from the first sample speech data; for each speech frame, subtracting the second spectrum of the noise speech segment from the first spectrum of the speech frame to obtain the third spectrum of the speech frame; performing inverse Fourier transform on the third spectrum of each speech frame to obtain each denoised speech frame; performing speech reconstruction according to each denoised speech frame to obtain the denoised first sample speech data. Thus, by utilizing the different characteristics of noise and signal in the frequency domain, the noise is reduced by processing the spectrum of the first sample speech data in the air traffic control field, so that the denoised first sample speech data is clearer and the noise component is greatly reduced, providing a high-quality data foundation for subsequent speech recognition model training.

[0074] Step 303: input the denoised first sample speech data into a shared encoder in the first speech recognition model to obtain an encoding result of the denoised first sample speech data.

[0075] Step 304: input the encoding result into a connected temporal classification (CTC) decoder in the first speech recognition model to obtain a plurality of candidate text sequences and their respective corresponding first scores.

[0076] Step 305, inputting multiple candidate text sequences and encoding results into the attention decoder in the first speech recognition model to obtain second scores corresponding to each candidate text sequence.

[0077] Step 306: Input the multiple candidate text sequences and the first score and the second score into the output layer of the first speech recognition model, so that the output layer determines the predicted text recognition result of the first sample speech data from the multiple candidate text sequences.

[0078] Step 307: training the first speech recognition model according to the first annotated text information and the predicted text recognition result to obtain a second speech recognition model adapted to the air traffic control field.

[0079] It should be noted that, for the specific description of steps 303-307, reference may be made to the relevant description in other embodiments, which will not be repeated here.

[0080] In this embodiment, by denoising the first sample speech data in the air traffic control field, the denoised first sample speech data is made clearer and the noise component is greatly reduced, thereby providing a high-quality data foundation for subsequent speech recognition model training.

[0081] Based on any one of the above embodiments, in order to ensure that the recognition accuracy of the second speech recognition model meets the preset requirements, correspondingly, after the second speech recognition model is trained by training data, the second speech recognition model can also be tested by test data, and according to the test results, it is determined whether the character error rate (Character Error Rate, CER) of the second speech recognition model exceeds the preset threshold. If not, it is determined that the second speech recognition model meets the accuracy requirements. At this time, the second speech recognition model can be configured in a speech recognition system, wherein the speech recognition system is configured in an electronic device of the air traffic controller, so that the air traffic controller can perform speech recognition on the air traffic control voice data to be processed through the speech recognition system in the electronic device, thereby reducing the workload of the air traffic controller, and thus improving the efficiency of obtaining the speech recognition results of the air traffic control voice data to be processed.

[0082] It should be noted that, the lower the value of the character error rate CER is, the higher the recognition accuracy of the second speech recognition model for the characters is.

[0083] The preset threshold is a critical value of a word error rate set according to actual needs, and this embodiment does not specifically limit the value of the preset threshold.

[0084] Among them, one possible way to obtain training data and test data is: obtain an original voice data set in the air traffic control field, wherein the original voice data set includes multiple existing voice data and corresponding annotated text results, wherein the annotated text results are obtained by manually annotating the existing voice data, and the existing voice data set is divided into training data and test data according to a preset ratio.

[0085] The preset ratio is preset according to actual needs, and this embodiment does not specifically limit the value of the preset ratio.

[0086] Based on any of the above embodiments, before training the first speech recognition model based on the training data, it can also include: extracting the Cepstral Mean and Variance Normalization (CMVN) features in the first sample speech data, and correspondingly, inputting the CMVN features into the shared encoder to encode the CMVN features through the shared encoder to obtain the encoding result of the first sample speech data. Thus, the use of CMVN can effectively reduce the feature offset and scale changes caused by different recording devices and environmental changes, thereby improving the accuracy of speech recognition, and because the normalized feature data is more stable, this helps to speed up the convergence of the model and improve the generalization ability of the model.

[0087] In order to clearly understand this application, Figure 4 The training method of the speech recognition model of this embodiment is described exemplarily.

[0088] Figure 4 The figure is a flowchart of a method for training a speech recognition model according to another embodiment of the present application.

[0089] like Figure 4 As shown, the method may also include:

[0090] Step 401, obtaining training data, wherein the training data includes: first sample voice data in the air traffic control field and corresponding first annotated text information.

[0091] Step 402: denoise the first sample speech data to obtain denoised first sample speech data.

[0092] For the specific description of step 402, please refer to the relevant description in other embodiments, which will not be repeated here.

[0093] Step 403: extract features from the denoised first sample speech data to obtain CMVN features of the denoised first sample speech data.

[0094] Step 404: input the CMVN features into the shared encoder in the first speech recognition model to obtain an encoding result.

[0095] Step 405: Input the encoding result into a connected temporal classification (CTC) decoder in the first speech recognition model to obtain a plurality of candidate text sequences and their respective corresponding first scores.

[0096] Step 406, input the multiple candidate text sequences and the encoding results into the attention decoder in the first speech recognition model to obtain the second score corresponding to each candidate text sequence.

[0097] Step 407: input the multiple candidate text sequences and the first score and the second score into the output layer of the first speech recognition model, so that the output layer determines the predicted text recognition result of the first sample speech data from the multiple candidate text sequences.

[0098] Step 408: training the first speech recognition model according to the first annotated text information and the predicted text recognition result to obtain a second speech recognition model adapted to the air traffic control field.

[0099] In this embodiment, a good balance between performance and efficiency is achieved by combining the fast search of the CTC decoder and the precise scoring of the Attention decoder. The introduction of the CMVN feature significantly improves the robustness of the speech recognition model, especially in different recording environments. It can effectively reduce feature offset and scale changes, thereby improving the overall recognition effect.

[0100] Figure 5 1 is a flowchart of a speech recognition method according to an embodiment of the present application. It should be noted that the speech recognition method provided in this embodiment is performed by a speech recognition device. The speech recognition device in this embodiment can be implemented by software and / or hardware. The speech recognition device can be a speech recognition system, or can be configured in a speech recognition system. The speech recognition system can be configured in an electronic device.

[0101] The electronic devices in this example embodiment may include terminal devices, servers, etc., wherein the terminal devices may be PCs (Personal Computers), mobile devices, tablet computers, etc., and this embodiment does not specifically limit this.

[0102] like Figure 5 As shown, the speech recognition method may include:

[0103] Step 501, obtaining air traffic control voice data to be processed.

[0104] As an example, the air-ground conversation voice data to be processed in the air traffic control field can be used as the air traffic control voice data to be processed.

[0105] It should be noted that the above-mentioned ground-to-air call voice data is only an example of air traffic control voice data. The air traffic control voice data in this embodiment can also be other voice data involved in the air traffic control field. This embodiment does not specifically limit the air traffic control voice data.

[0106] Step 502: input the air traffic control speech data into a second speech recognition model to obtain a target text recognition result of the air traffic control speech data.

[0107] It should be noted that the second speech recognition model in this embodiment is trained using the training method disclosed in this application.

[0108] It should be noted that, regarding the process of training the second speech recognition model, reference can be made to the relevant description disclosed in this application, which will not be repeated here.

[0109] The speech recognition method provided in the embodiment of the present application processes the air traffic control speech data to be processed by a second speech recognition model adapted to the air traffic control field, thereby improving the efficiency and accuracy of the target text recognition results of the air traffic control speech data.

[0110] Based on the above embodiment, in order to improve the accuracy of the target text recognition result obtained by the second speech recognition model, correspondingly, before inputting the air traffic control voice data into the second speech recognition model to obtain the target text recognition result of the air traffic control voice data, it can also include: performing denoising on the air traffic control voice data to be processed to obtain denoised air traffic control voice data. Correspondingly, the denoised air traffic control voice data is input into the second speech recognition model to obtain the target text recognition result.

[0111] In one embodiment of the present application, a possible implementation method of performing denoising on the air traffic control voice data to be processed to obtain the denoised air traffic control voice data is as follows: performing frame processing on the air traffic control voice data to obtain multiple first voice frames of the air traffic control voice data; performing Fourier transform on the multiple first voice frames respectively to obtain the first spectra corresponding to the multiple first voice frames; extracting the target noise voice segment containing only noise from the air traffic control voice data; for each first voice frame, subtracting the second spectrum of the target noise voice segment from the first spectrum of the first voice frame to obtain the third spectrum of the first voice frame; performing inverse Fourier transform on the third spectrum of each first voice frame to obtain each denoised first voice frame; performing voice reconstruction according to each denoised first voice frame to obtain the denoised air traffic control voice data. Thus, by utilizing the different characteristics of noise and signal in the frequency domain, the noise is reduced by processing the spectrum of the air traffic control voice data, so that the denoised air traffic control voice data is clearer and the noise component is greatly reduced, providing a high-quality data basis for the subsequent training of the second voice recognition model, and then improving the text recognition result of the second voice recognition model.

[0112] In one embodiment of the present application, in order to further improve the accuracy of the text recognition results obtained, correspondingly, after the air traffic control speech data to be processed is denoised to obtain the denoised air traffic control speech data, feature extraction can also be performed on the denoised air traffic control speech data to obtain the CMVN features of the denoised air traffic control speech data, and the CMVN features are input into the second speech recognition model to obtain the target text recognition results.

[0113] Corresponding to the training methods of the speech recognition models provided in the above-mentioned embodiments, an embodiment of the present application also provides a training device for a speech recognition model. Since the training device for the speech recognition model provided in the embodiment of the present application corresponds to the training methods for the speech recognition models provided in the above-mentioned embodiments, the implementation method of the speech recognition model training method is also applicable to the training device for the speech recognition model of this embodiment and will not be described in detail in this embodiment.

[0114] Figure 6 It is a structural diagram of a training device for a speech recognition model according to an embodiment of the present application.

[0115] like Figure 6 As shown, the speech recognition model training device 600 includes: an acquisition module 601, an encoding module 602, a first decoding module 603, a second decoding module 604, a recognition result determination module 605 and a model training module 606, wherein:

[0116] The acquisition module 601 is used to acquire training data, wherein the training data includes: first sample voice data in the field of air traffic control and corresponding first annotated text information.

[0117] The encoding module 602 is used to input the first sample speech data into a shared encoder in the first speech recognition model to obtain an encoding result of the first sample speech data.

[0118] The first decoding module 603 is used to input the encoding result into the connection time classification CTC decoder in the first speech recognition model to obtain multiple candidate text sequences and their corresponding first scores.

[0119] The second decoding module 604 is used to input multiple candidate text sequences and encoding results into the attention decoder in the first speech recognition model to obtain second scores corresponding to each candidate text sequence.

[0120] The recognition result determination module 605 is used to input multiple candidate text sequences and the first score and the second score into the output layer of the first speech recognition model, so that the output layer determines the predicted text recognition result of the first sample speech data from the multiple candidate text sequences.

[0121] The model training module 606 is used to train the first speech recognition model according to the first annotated text information and the predicted text recognition result to obtain a second speech recognition model suitable for the air traffic control field.

[0122] In one embodiment of the present application, the model training module 606 is specifically used to: determine a first loss value of the first annotated text information and the predicted text recognition result according to a first loss function, wherein the first loss function is used to represent the loss function of a structure composed of a shared encoder and a CTC decoder; determine a second loss value of the first annotated text information and the predicted text recognition result according to a second loss function, wherein the second loss function is used to represent the loss function of a structure composed of a shared encoder and an attention decoder; perform weighted summation of the first loss value and the second loss value to obtain a total loss value; and train the first speech recognition model according to the total loss value to obtain a second speech recognition model adapted to the air traffic control field.

[0123] In one embodiment of the present application, the recognition result determination module 605 is specifically used to: in the output layer of the first speech recognition model, respectively perform weighted summation of the first score and the second score of each candidate text sequence to obtain a comprehensive score corresponding to each candidate text sequence; and select a text sequence with the highest comprehensive score from multiple candidate text sequences as the predicted text recognition result of the first sample speech data.

[0124] In one embodiment of the present application, the second decoding module 604 is specifically used to: determine the attention weights corresponding to each of the multiple candidate text sequences based on the encoding results in the attention decoder in the first speech recognition model; and determine the second scores corresponding to each of the multiple candidate text sequences based on the attention weights corresponding to each of the multiple candidate text sequences.

[0125] In one embodiment of the present application, the device may further include:

[0126] The denoising module is used to perform frame processing on the first sample voice data to obtain multiple voice frames of the first sample voice data; perform Fourier transform on the multiple voice frames respectively to obtain first frequency spectra corresponding to the multiple voice frames; extract a noisy voice segment containing only noise from the first sample voice data; for each voice frame, subtract the second frequency spectrum of the noisy voice segment from the first frequency spectrum of the voice frame to obtain a third frequency spectrum of the voice frame; perform inverse Fourier transform on the third frequency spectrum of each voice frame to obtain each denoised voice frame; perform voice reconstruction according to each denoised voice frame to obtain the denoised first sample voice data.

[0127] In one embodiment of the present application, the first speech recognition model is obtained by pre-training an initial speech recognition model based on second sample speech data of a general field and corresponding second annotated text information.

[0128] The training device based on the speech recognition model of the embodiment of the present application, after acquiring the training data, inputs the first sample speech data into the shared encoder in the first speech recognition model to obtain the encoding result of the first sample speech data, and inputs the encoding result into the connection temporal classification CTC decoder in the first speech recognition model to obtain multiple candidate text sequences and their respective corresponding first scores; inputs the multiple candidate text sequences and the encoding results into the attention decoder in the first speech recognition model to obtain the second scores corresponding to the candidate text sequences, and inputs the multiple candidate text sequences and the first scores and the second scores into the output layer of the first speech recognition model, so that the output layer determines the predicted text recognition result of the first sample speech data from the multiple candidate text sequences, and trains the first speech recognition model according to the first annotated text information and the predicted text recognition result, so as to obtain a second speech recognition model adapted to the air traffic control field. Therefore, combined with the sample speech data in the air traffic control field and the corresponding annotated text information, the first speech recognition model including the CTC decoder and the attention decoder is trained, and a second speech recognition model adapted to the air traffic control field is quickly obtained. This improves the efficiency of obtaining the second speech recognition model adapted to the air traffic control field while improving the recognition accuracy and generalization ability of the obtained speech recognition model.

[0129] Corresponding to the speech recognition methods provided in the above-mentioned embodiments, an embodiment of the present application also provides a speech recognition device. Since the speech recognition device provided in the embodiment of the present application corresponds to the speech recognition methods provided in the above-mentioned embodiments, the implementation method of the speech recognition method is also applicable to the speech recognition device of this embodiment and will not be described in detail in this embodiment.

[0130] Figure 7 It is a structural diagram of a speech recognition device according to an embodiment of the present application.

[0131] like Figure 7 As shown, the speech recognition device 700 includes: an acquisition module 701 and a recognition result determination module 702, wherein:

[0132] The acquisition module 701 is used to acquire the air traffic control voice data to be processed.

[0133] The recognition result determination module 702 is used to input the air traffic control speech data into the second speech recognition model to obtain the target text recognition result of the air traffic control speech data.

[0134] It should be noted that the second speech recognition model in this embodiment is trained by the speech recognition model training method proposed in the embodiment of the present application.

[0135] In one embodiment of the present application, the device may also include: a denoising module, which is used to perform frame processing on the air traffic control voice data to obtain multiple first voice frames of the air traffic control voice data; perform Fourier transform on the multiple first voice frames respectively to obtain first spectra corresponding to each of the multiple first voice frames; extract a target noise speech segment containing only noise from the air traffic control voice data; for each first voice frame, subtract the second spectrum of the target noise speech segment from the first spectrum of the first voice frame to obtain a third spectrum of the first voice frame; perform inverse Fourier transform on the third spectrum of each first voice frame to obtain each denoised first voice frame; and perform voice reconstruction based on each denoised first voice frame to obtain denoised air traffic control voice data.

[0136] The speech recognition device of the embodiment of the present application processes the air traffic control speech data to be processed by a second speech recognition model adapted to the air traffic control field, thereby improving the efficiency and accuracy of the target text recognition results of the air traffic control speech data.

[0137] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0138] Figure 8 It is a structural block diagram of an electronic device according to an embodiment of the present application.

[0139] like Figure 8 As shown, the electronic device 800 includes: a memory 810, a processor 820, and computer instructions stored in the memory 810 and executable on the processor 820.

[0140] When the processor 820 executes the instructions, the speech recognition method or the training method of the speech recognition model provided in the above embodiments is implemented.

[0141] Furthermore, the electronic device 800 further includes:

[0142] The communication interface 830 is used for communication between the memory 810 and the processor 820 .

[0143] The memory 810 is used to store computer instructions that can be executed on the processor 820 .

[0144] The memory 810 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0145] The processor 820 is used to implement the speech recognition method of the above embodiment, or the training method of the speech recognition model when executing the program.

[0146] If the memory 810, the processor 820 and the communication interface 830 are implemented independently, the communication interface 830, the memory 810 and the processor 820 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0147] Optionally, in a specific implementation, if the memory 810, the processor 820 and the communication interface 830 are integrated on a chip, the memory 810, the processor 820 and the communication interface 830 can communicate with each other through an internal interface.

[0148] The processor 820 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0149] On the other hand, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements any speech recognition method or speech recognition model training method of the embodiments of the present application.

[0150] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0151] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0152] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for training a speech recognition model, characterized in that: The method comprises: Acquire training data, wherein the training data includes: first sample voice data in the field of air traffic control and corresponding first annotated text information; Inputting the first sample speech data into a shared encoder in a first speech recognition model to obtain an encoding result of the first sample speech data; Inputting the encoding result into a connection temporal classification (CTC) decoder in the first speech recognition model to obtain a plurality of candidate text sequences and respective corresponding first scores; Inputting the multiple candidate text sequences and the encoding results into an attention decoder in the first speech recognition model to obtain a second score corresponding to each of the candidate text sequences; Inputting the plurality of candidate text sequences and the first score and the second score into an output layer of the first speech recognition model, so that the output layer determines a predicted text recognition result of the first sample speech data from the plurality of candidate text sequences; The first speech recognition model is trained according to the first annotated text information and the predicted text recognition result to obtain a second speech recognition model adapted to the air traffic control field.

2. The method according to claim 1, characterized in that The step of training the first speech recognition model according to the first annotated text information and the predicted text recognition result to obtain a second speech recognition model adapted to the air traffic control field includes: Determine a first loss value of the first annotated text information and the predicted text recognition result according to a first loss function, wherein the first loss function is used to represent a loss function of a structure composed of the shared encoder and the CTC decoder; Determining a second loss value of the first annotated text information and the predicted text recognition result according to a second loss function, wherein the second loss function is used to represent a loss function of a structure composed of the shared encoder and the attention decoder; Performing a weighted summation on the first loss value and the second loss value to obtain a total loss value; The first speech recognition model is trained according to the total loss value to obtain a second speech recognition model adapted to the air traffic control field.

3. The method according to claim 1, characterized in that The step of inputting the plurality of candidate text sequences and the first score and the second score into an output layer of the first speech recognition model so that the output layer determines a predicted text recognition result of the first sample speech data from the plurality of candidate text sequences comprises: In the output layer of the first speech recognition model, weighted summing is performed on the first score and the second score of each candidate text sequence to obtain a comprehensive score corresponding to each candidate text sequence; From the multiple candidate text sequences, the text sequence with the highest comprehensive score is selected as the predicted text recognition result of the first sample speech data.

4. The method according to claim 1, characterized in that The step of inputting the plurality of candidate text sequences and the encoding results into an attention decoder in the first speech recognition model to obtain second scores corresponding to the candidate text sequences respectively includes: In the attention decoder in the first speech recognition model, determining the attention weights corresponding to each of the plurality of candidate text sequences according to the encoding result; Determine second scores corresponding to each of the multiple candidate text sequences according to the attention weights corresponding to each of the multiple candidate text sequences.

5. The method according to claim 1, characterized in that Before inputting the first sample speech data into a shared encoder in a first speech recognition model to obtain an encoding result of the first sample speech data, the method further includes: Performing frame processing on the first sample voice data to obtain a plurality of voice frames of the first sample voice data; Performing Fourier transform on the multiple speech frames respectively to obtain first frequency spectra corresponding to the multiple speech frames respectively; Extracting a noise speech segment containing only noise from the first sample speech data; For each speech frame, subtract the second spectrum of the noisy speech segment from the first spectrum of the speech frame to obtain a third spectrum of the speech frame; Performing inverse Fourier transform on the third frequency spectrum of each of the speech frames to obtain each speech frame after denoising; Speech reconstruction is performed according to each denoised speech frame to obtain the denoised first sample speech data.

6. The method according to any one of claims 1 to 5, characterized in that The first speech recognition model is obtained by pre-training an initial speech recognition model based on second sample speech data in a general field and corresponding second annotated text information.

7. A speech recognition method, characterized in that: include: Obtain air traffic control voice data to be processed; The air traffic control voice data is input into a second speech recognition model to obtain a target text recognition result of the air traffic control voice data, wherein the second speech recognition model is trained by the method described in any one of claims 1-6.

8. The method according to claim 7, characterized in that Before inputting the air traffic control voice data into a second voice recognition model to obtain a target text recognition result of the air traffic control voice data, the method further includes: Performing frame processing on the air traffic control voice data to obtain a plurality of first voice frames of the air traffic control voice data; Performing Fourier transform on the plurality of first speech frames respectively to obtain first frequency spectra corresponding to the plurality of first speech frames respectively; Extracting a target noise speech segment containing only noise from the air traffic control speech data; For each first speech frame, subtract the second spectrum of the target noise speech segment from the first spectrum of the first speech frame to obtain a third spectrum of the first speech frame; Performing inverse Fourier transform on the third frequency spectrum of each of the first speech frames to obtain each first speech frame after denoising; Speech reconstruction is performed based on each first speech frame after denoising to obtain air traffic control speech data after denoising.

9. A training device for a speech recognition model, characterized in that: The device comprises: An acquisition module, used to acquire training data, wherein the training data includes: first sample voice data in the field of air traffic control and corresponding first annotated text information; an encoding module, used for inputting the first sample speech data into a shared encoder in a first speech recognition model to obtain an encoding result of the first sample speech data; A first decoding module, configured to input the encoding result into a connection temporal classification (CTC) decoder in the first speech recognition model to obtain a plurality of candidate text sequences and first scores corresponding to the respective candidate text sequences; A second decoding module, used for inputting the multiple candidate text sequences and the encoding results into an attention decoder in the first speech recognition model to obtain a second score corresponding to each of the candidate text sequences; a recognition result determination module, configured to input the plurality of candidate text sequences and the first score and the second score into an output layer of the first speech recognition model, so that the output layer determines a predicted text recognition result of the first sample speech data from the plurality of candidate text sequences; A model training module is used to train the first speech recognition model according to the first annotated text information and the predicted text recognition result to obtain a second speech recognition model suitable for the air traffic control field.

10. A speech recognition device, characterized in that: include: An acquisition module, used to acquire air traffic control voice data to be processed; A recognition result determination module is used to input the air traffic control voice data into a second speech recognition model to obtain a target text recognition result of the air traffic control voice data, wherein the second speech recognition model is trained by the method described in any one of claims 1-6.

11. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 6 or the method according to any one of claims 7 to 8 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 or the method according to any one of claims 7 to 8 is implemented.

13. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6, or the method according to any one of claims 7 to 8.

Images