Streaming speech recognition and model training method, device, and computer equipment

Abstract

The streaming speech recognition and model training method, device, and computer equipment provided by the present invention obtain a trained non-streaming speech recognition model; the recognition accuracy and output delay of the non-streaming speech recognition model both meet preset indicators; the model parameters of the non-streaming speech recognition model are fixed as the initial streaming speech recognition model; the non-streaming speech recognition model is used to perform distillation training on the streaming speech recognition model so that the output of the streaming speech recognition model approaches the output of the non-streaming speech recognition model; the training is stopped when the preset training conditions are met to obtain the trained streaming speech recognition model. The present invention utilizes an existing high-quality non-streaming speech recognition model and improves the effect of the streaming speech recognition model through the method of knowledge distillation. Compared with training directly from scratch, the streaming speech recognition model trained by the present invention can achieve low-latency streaming recognition and ensure recognition quality.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a streaming speech recognition and model training method, device, and computer equipment. Background Art

[0002] In a real-time muting system, the system needs to perform real-time recognition while the speaker is speaking. This requires the system to be able to recognize the content in real time and accurately, and correctly locate the illegal parts that need to be muted. However, if the recognition result is inaccurate, it may cause two problems: one is the wrong muting, which will affect the user's viewing experience; the other is the failure to mute the illegal content, which directly exposes the illegal content. At the same time, the recognition speed is also an important link. If the recognition delay is too large, it may not be possible to mute the illegal content in real time, and the illegal content will be directly exposed, thus creating content security risks.

[0003] Therefore, for the real-time silencing system, the streaming speech recognition technology used needs to meet two key conditions at the same time: first, the recognition delay needs to be low enough to achieve real-time recognition; second, the recognition accuracy needs to be high to reduce the possibility of misoperation. Only by taking these two aspects into account can the system truly achieve real-time automatic silencing and achieve the purpose of content security supervision. If the recognition effect is not immediate and accurate, it is likely to lead to poor user experience or the occurrence of content security incidents, which is not good for both the system and the user.

[0004] Current real-time noise cancellation systems usually use an end-to-end deep learning speech recognition model, such as the Wenet streaming speech recognition model. However, the conventional training method of the Wenet streaming speech recognition model still results in a high word error rate and output delay in the model, resulting in the speech recognition accuracy and real-time performance in the real-time noise cancellation system failing to meet the requirements. Summary of the invention

[0005] One of the purposes of the present invention is to provide a streaming speech recognition and model training method, apparatus, and computer equipment for improving the recognition accuracy of streaming speech recognition and reducing output delay. The embodiments of the present invention can be implemented as follows:

[0006] In a first aspect, the present invention provides a method for training a streaming speech recognition model, the method comprising: obtaining a trained non-streaming speech recognition model; the recognition accuracy and output delay of the non-streaming speech recognition model both meet preset indicators; fixing the model parameters of the non-streaming speech recognition model as an initial streaming speech recognition model; using the non-streaming speech recognition model to perform distillation training on the streaming speech recognition model so that the output of the streaming speech recognition model approaches the output of the non-streaming speech recognition model; stopping the training when the preset training conditions are met to obtain the trained streaming speech recognition model.

[0007] In an optional embodiment, the streaming speech recognition model includes a first CTC decoder and a first attention decoder; the non-streaming speech recognition model includes a second CTC decoder and a second attention decoder; the non-streaming speech recognition model is used to perform distillation training on the streaming speech recognition model so that the output of the streaming speech recognition model approaches the output of the non-streaming speech recognition model, including: during the training process, determining a first knowledge distillation loss between the respective outputs of the first CTC decoder and the second CTC decoder, a second knowledge distillation loss between the respective outputs of the first attention decoder and the second attention decoder, and a fusion loss between the first CTC decoder and the first attention decoder; and adjusting the model parameters of the streaming speech recognition model by combining the first knowledge distillation loss, the second knowledge distillation loss, and the fusion loss so that the output of the streaming speech recognition model approaches the output of the non-streaming speech recognition model.

[0008] In an optional embodiment, determining a first knowledge distillation loss between the respective outputs of the first CTC decoder and the second CTC decoder includes: determining an output delay offset between the first CTC decoder and the second CTC decoder; for each training audio sample, determining the first knowledge distillation loss based on the output delay offset and the probability of each character output by the first CTC decoder and the second CTC decoder.

[0009] In an optional embodiment, determining the output delay offset between the first CTC decoder and the second CTC decoder includes: for each training audio sample, calculating the character overlap ratio output by the first CTC decoder and the second CTC decoder under different offset preset values, and using the offset preset value corresponding to the maximum character overlap ratio as the output delay offset.

[0010] In an optional embodiment, determining a second knowledge distillation loss between respective outputs of the first attention decoder and the second attention decoder includes: for each training audio sample, calculating the second knowledge distillation loss based on the probability of each character contained in each output of the first attention decoder and the second attention decoder.

[0011] In an optional implementation, determining a fusion loss between the first CTC decoder and the first attention decoder includes: fusing the CTC loss of the first CTC decoder and the attention encoder-decoder loss of the first attention decoder according to a preset balance factor to obtain the fusion loss.

[0012] In a second aspect, the present invention provides a streaming speech recognition method, comprising: obtaining audio to be recognized; inputting the audio to be recognized into a trained streaming speech recognition model for speech recognition, and obtaining a recognition text corresponding to the audio to be recognized; wherein the streaming speech recognition model is obtained by the streaming speech recognition model training method as described in any of the aforementioned implementations; and the recognition accuracy and output delay of the recognized text meet preset indicators.

[0013] In a third aspect, the present invention provides a streaming speech recognition model training device, comprising: an acquisition module, used to acquire a trained non-streaming speech recognition model; the recognition accuracy and output delay of the non-streaming speech recognition model both meet preset speech recognition indicators; a determination module, used to fix the model parameters of the non-streaming speech recognition model as an initial streaming speech recognition model; a training module, used to perform distillation training on the streaming speech recognition model so that the output of the streaming speech recognition model approaches the output of the non-streaming speech recognition model; the training module is also used to stop training when a preset training condition is reached to obtain the trained streaming speech recognition model.

[0014] In a fourth aspect, the present invention provides a streaming speech recognition device, comprising: an acquisition module for acquiring audio to be recognized; a recognition module for inputting the audio to be recognized into a trained streaming speech recognition model for speech recognition, and obtaining a recognition text corresponding to the audio to be recognized; wherein the streaming speech recognition model is obtained by the streaming speech recognition model training method as described in any one of the aforementioned implementations; the recognition accuracy and output delay of the recognized text meet preset indicators.

[0015] In a fifth aspect, the present invention provides a computer device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the method described in the first and second aspects above.

[0016] The streaming speech recognition and model training method, device, and computer equipment provided by the present invention first obtain a trained non-streaming speech recognition model. Since the recognition accuracy and output delay of the model have met the requirements after training, it can be used as a teacher model for distillation training of the streaming speech recognition model. Then, the streaming speech recognition model is distilled and trained using the non-streaming speech recognition model. During the training process, optimization is performed by minimizing the difference between the outputs of the streaming speech recognition model and the non-streaming speech recognition model, so that the recognition result of the streaming speech recognition model is closer and closer to the non-streaming high-quality model. Once the preset training conditions are met, the distillation training process is terminated, thereby obtaining a streaming speech recognition model with high recognition accuracy and low output delay. The present invention utilizes an existing high-quality non-streaming speech recognition model and improves the effect of the streaming speech recognition model through the knowledge distillation method. Compared with training directly from scratch, while ensuring the real-time performance of the streaming speech recognition model, its recognition performance can also be greatly improved. The streaming speech recognition model trained in this way can not only achieve low-latency streaming recognition, but also ensure recognition quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 This is a system architecture diagram for a real-time voice scenario;

[0019] Figure 2 A schematic flow chart of a streaming speech recognition model training method provided by an embodiment of the present invention;

[0020] Figure 3 A schematic flow chart of step S203 provided in an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of the principle of a streaming speech recognition model training method provided by an embodiment of the present invention;

[0022] Figure 5 A schematic flow chart of a streaming speech recognition method provided by an embodiment of the present invention;

[0023] Figure 6 A functional module diagram of a streaming speech recognition model training device provided by an embodiment of the present invention;

[0024] Figure 7A functional module diagram of a streaming speech recognition device provided by an embodiment of the present invention;

[0025] Figure 8 A schematic structural block diagram of a computer device 800 provided in the present application is shown. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0027] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0029] In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear to indicate an orientation or position relationship, they are based on the orientation or position relationship shown in the accompanying drawings, or are the orientation or position relationship in which the product of the invention is usually placed when used. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0030] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0031] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0032] There are two main modes of speech recognition: streaming speech recognition and non-streaming speech recognition. Streaming speech recognition can process and recognize speech streams in real time. It can receive and recognize speech while returning the intermediate results of recognition. This is essential for application scenarios that require real-time response. For example, real-time subtitles display and output a number of words each time, instead of waiting for the speaker to finish a sentence and then output the result of the entire sentence. Another example is real-time muting. As long as there is illegal content in the part of the speech that has been recognized, it must be muted in time. Otherwise, the audience may miss the opportunity to mute the speech after the complete sentence is finished, and the illegal content has already been heard. Non-streaming speech recognition cannot give results in real time and is generally used for offline recognition of existing audio files.

[0033] In general, streaming recognition processes voice streams in real time and supports real-time output of intermediate results. This is effective for real-time voice interaction scenarios that require fast response and has become a necessary solution for such applications.

[0034] For example, see Figure 1 , Figure 1 This is a system architecture diagram for a real-time voice scenario. Figure 1 As shown, the system architecture includes: a terminal and a server. The server may be a server that provides speech recognition services, a server for a speech interaction system, etc., and may specifically be a server cluster deployed in the cloud. The server stores a streaming speech recognition model. Through the preset operation logic in the server, the server uses the streaming speech recognition model to perform speech recognition on the real-time input speech, obtains the recognition result, and can feed back the recognition result to the terminal in real time. In addition, the server may store the training audio samples required for training, implement the training process of the streaming speech recognition model based on the training audio samples, and obtain the trained streaming speech recognition model.

[0035] The terminal can be a hardware device with network communication function, computing function and information display function, including but not limited to smart phones, tablet computers, desktop computers, IoT devices, etc. Through communication interaction with the server, when the user uses the terminal to input the voice stream, the server can obtain the current input voice in real time, use the trained streaming voice recognition model to obtain the text information corresponding to the voice, and output the recognition result to the terminal according to the preset rules.

[0036] Figure 1The system architecture shown can be applied to the live broadcast field, especially the real-time muting scenario in the live broadcast field. In the real-time muting scenario, the streaming speech recognition technology used needs to meet two key conditions at the same time: first, the recognition delay needs to be low enough to achieve real-time recognition and timely muting; second, the recognition accuracy needs to be high to ensure that the illegal audio can be correctly identified and muted, and also reduce the possibility of normal audio being muted by mistake. Only by taking these two aspects into account can the system truly achieve real-time automatic muting and achieve the purpose of content security supervision. If the recognition effect is not immediate and accurate, it is likely to lead to poor user experience or the occurrence of content security incidents, which is not good for both the system and the user.

[0037] In order to meet the business requirements of accuracy and real-time performance of speech recognition in real-time silencing scenarios, an embodiment of the present invention provides a streaming speech recognition model training method. Compared with the common wenet streaming speech recognition model, the streaming speech recognition model obtained by the training method provided by the embodiment of the present invention has a lower word recognition error rate and lower output delay. This is very important for the accuracy and real-time performance of speech recognition in the silencing system.

[0038] See also Figure 2 , Figure 2 The schematic flow chart of the streaming speech recognition model training method provided by the embodiment of the present invention may include the following steps:

[0039] S201, obtaining a trained non-streaming speech recognition model;

[0040] S202, fixing the model parameters of the non-streaming speech recognition model as an initial streaming speech recognition model;

[0041] The non-streaming speech recognition model used in the embodiment of the present invention has two key indicators, recognition accuracy and output delay, when performing speech recognition, which both meet the pre-set level requirements. The model quality is reliable and suitable for use as a teacher model for distillation training. This can save the time cost of training a large teacher model.

[0042] In an optional implementation, the non-streaming speech recognition model used in the embodiment of the present invention may be the U2 model of the open source wenet. The model uses the Joint CTC / AED structure.

[0043] S203, using the non-streaming speech recognition model, performing distillation training on the streaming speech recognition model, so that the output of the streaming speech recognition model is close to the output of the non-streaming speech recognition model;

[0044] It is understandable that in the field of deep learning, distillation training is conducive to improving model recognition performance. Usually, a larger model is used to distill a smaller model, thereby improving the recognition performance of the small model. However, wenet's non-streaming speech recognition model does not perform dynamic block training during the training process. Compared with streaming speech recognition models of the same structure and size, it has a higher recognition rate and a smaller output delay, making it suitable for use as a teacher model for knowledge distillation. This can save the time spent on training a large teacher model, and it can also be used as the initial model of a streaming speech recognition model. Therefore, the same wenet non-streaming speech recognition model can be used to distill a streaming speech recognition model, thereby achieving the purpose of improving the recognition rate of the streaming speech recognition model and reducing its delay.

[0045] S204. When the preset training conditions are met, the training is stopped to obtain a trained streaming speech recognition model.

[0046] In an embodiment of the present invention, the preset training conditions may be reaching a preset number of iterations, convergence of the loss function, indicators of the streaming speech recognition model (recognition accuracy and output delay) reaching or exceeding preset targets, etc., which are not limited here.

[0047] In the technical solution of the above steps S201 to 204, it is considered that the recognition accuracy and output delay of the non-streaming speech recognition model meet the requirements of various indicators relative to the streaming speech recognition model. Therefore, the existing high-quality non-streaming speech recognition model can be used to improve the effect of the streaming speech recognition model through the knowledge distillation method. Specifically, first obtain a trained non-streaming speech recognition model. Since the recognition accuracy and output delay of the model have met the requirements after training, it can be used as a teacher model for distillation training of the streaming speech recognition model. Then, the streaming speech recognition model is distilled and trained using the non-streaming speech recognition model. During the training process, optimization is performed by minimizing the difference between the outputs of the streaming speech recognition model and the non-streaming speech recognition model, so that the recognition results of the streaming speech recognition model are closer and closer to the non-streaming high-quality model. Once the indicators of the streaming speech recognition model reach or exceed the preset target, or the number of iterations exceeds the limit, the distillation training process is terminated, thereby obtaining a streaming speech recognition model with high recognition accuracy and low output delay. Compared with training directly from scratch, this method can greatly improve the recognition performance of the streaming speech recognition model while ensuring the real-time performance of the streaming speech recognition model. The streaming speech recognition model trained in this way can not only achieve low-latency streaming recognition but also ensure recognition quality, and has good application prospects in real-time scenarios.

[0048] The above step S203 is described in detail below.

[0049] In step S203, the streaming speech recognition model and the non-streaming speech recognition model in the embodiment of the present invention have the same model structure, both of which have two decoders, namely, a CTC decoder and an attention decoder. Therefore, for the convenience of description, the decoder of the streaming speech recognition model in the embodiment of the present invention is represented by a first CTC decoder and a first attention decoder, and the decoder of the non-streaming speech recognition model is represented by a second CTC decoder and a second attention decoder. Therefore, for the above step S203, please refer to Figure 3 , Figure 3 The schematic flow chart of step S203 provided in the embodiment of the present invention may include:

[0050] S203-1, during the training process, determining a first knowledge distillation loss between the outputs of the first CTC decoder and the second CTC decoder, a second knowledge distillation loss between the outputs of the first attention decoder and the second attention decoder, and a fusion loss between the first CTC decoder and the first attention decoder;

[0051] S203-2. Combine the first knowledge distillation loss, the second knowledge distillation loss, and the fusion loss to adjust the model parameters of the streaming speech recognition model so that the output of the streaming speech recognition model approaches the output of the non-streaming speech recognition model.

[0052] In the embodiment of the present invention, the same training audio sample can be input into the encoders of the streaming speech recognition model and the non-streaming speech recognition model to obtain the text output by the respective decoders.

[0053] The following describes how to calculate the first knowledge distillation loss, the second knowledge distillation loss and the fusion loss in the embodiment of the present invention. For the convenience of description, the relevant symbols in the embodiment of the present invention are first explained.

[0054] Where p1 and p2 represent the output probabilities of the second attention decoder and the second CTC decoder, respectively, q1 and q2 represent the output probabilities of the first attention decoder and the first CTC decoder, respectively; y t,i Indicates the i-th token corresponding to the t-th training audio sample, y u,j represents the jth token of the uth output, and X represents the audio feature sequence of the input training audio sample.

[0055] First, the fusion loss includes the joint loss of the ctc loss of the first ctc decoder at the frame level and the aed loss of the first attention decoder at the label level. The expression of the fusion loss function is as follows:

[0056] L combine =a*L ctc (X,y ctc )+(1-a)*L aed (X,y aed )

[0057] Among them, y ctc is the output of the first ctc decoder; y aed is the output of the first attention decoder; a is the balance factor between ctcloss and aed loss.

[0058] Secondly, both the streaming speech recognition model and the non-streaming speech recognition model have two decoder outputs. The embodiment of the present invention adds a knowledge distillation loss (KD loss) to each of the CTC decoder and the attention decoder. Specifically, the embodiment of the present invention uses the KD loss between the CTC decoders of the streaming speech recognition model and the non-streaming speech recognition model as the first KD loss, and uses the KD loss between the attention decoders as the second KD loss.

[0059] For the first KD loss, since the CTC decoder outputs frame synchronously, it is uncertain in which frame normal characters are output instead of blank characters. Therefore, there may be position deviations in the outputs of the CTC decoders of the teacher model and the student model. It is unreasonable to force the output of each frame of the student model to be close to the output of the teacher model of the corresponding frame, and the training may be difficult to converge. Therefore, an embodiment of the present invention first determines an output delay offset to compensate for the above position deviation so that the output of the streaming speech recognition model is close to the output of the non-streaming speech recognition model. Therefore, in the process of determining the first knowledge distillation loss, the following steps may be included:

[0060] Step a1, determining the output delay offset between the first CTC decoder and the second CTC decoder;

[0061] The embodiment of the present invention sets up an adaptive mechanism, that is, for each training audio sample, the character overlap ratio output by the first CTC decoder and the second CTC decoder under different offset preset values ​​is calculated, only the token output with the largest output probability is taken for each frame, and the offset preset value corresponding to the maximum character overlap ratio is used as the output delay offset.

[0062] It is understandable that the above method of determining the output delay offset can dynamically select the most suitable delay offset according to different training audio samples, rather than using a fixed value. Taking the maximum value of character overlap rate as the standard, the offset with the best recognition effect can be selected, thereby optimizing the overall recognition performance. So in general, this method of dynamically selecting the optimal delay offset for each training audio sample can obtain a better recognition effect than a fixed delay value. It uses the recognition results of multiple decoders to assist in selecting the best delay, thereby improving the robustness of the selection.

[0063] For ease of understanding, assume that F t represents the output vector of the second ctc decoder at time t, F t ′ represents the output vector of the first ctc decoder at time t. First, define the function f(F1, F2) as:

[0064]

[0065] Among them, F1, F2 are two input vectors of function f, arg max i (F1(i)) represents the subscript of the maximum value among all dimensions of the input vector F1. arg max i (F2(i)) is the same. Based on the above f function, the calculation formula of the output delay offset (denoted as d) in the embodiment of the present invention can be expressed as:

[0066]

[0067] Wherein: τ is the preset offset value in the embodiment of the present invention, the value range of τ is [0, D], and D is a specified maximum preset offset value.

[0068] Step a2: For each training audio sample, determine the first knowledge distillation loss according to the output delay offset and the probability of each character output by the first CTC decoder and the second CTC decoder respectively.

[0069] In this embodiment of the present invention, the calculation expression of the first kd loss is:

[0070]

[0071] Where d is the offset; T is the total number of feature frames of the training audio sample; N is the total number of output tokens (also called output dimension or number of modeling units);

[0072] For the second kd loss, for each training audio sample, the second knowledge distillation loss can be calculated according to the probability of each character contained in each output of the first attention decoder and the second attention decoder. Specifically, the calculation formula of the second kd loss can be expressed as:

[0073]

[0074] Here, U represents the number of modeling units included in the training audio sample, or is called the sentence length.

[0075] By determining the first kd loss, the second kd loss and the fusion loss, the total loss function of the streaming speech recognition model can be obtained as:

[0076] L_final=b*(a*kd ctc +(1-a)*kd att )+(1-b)*L_combine

[0077] Among them, a and b are balance factors, and their values ​​are both (0, 1).

[0078] Based on the above total loss function, the parameters of the streaming speech recognition model are continuously adjusted during the training process through back propagation and gradient descent method, so that the output of the streaming speech recognition model is close to the output of the non-streaming speech recognition model.

[0079] In summary, the principle schematic diagram of the streaming speech recognition model training method provided by the embodiment of the present invention is as follows: Figure 4 As shown, the training audio sample speech is simultaneously input into the encoders of the streaming speech recognition model and the non-streaming speech recognition model, and the encoder inputs the encoding result into the CTC decoder and the attention decoder. The CTC decoder and the attention decoder of the streaming speech recognition model determine the CTC loss and the AED loss according to their respective outputs. At the same time, the first KD loss between the outputs of the CTC decoders of the streaming speech recognition model and the non-streaming speech recognition model and the second KD loss between the outputs of the attention decoders can also be determined. The model parameters of the streaming speech recognition model are adjusted by combining the CTC loss, the AED loss, the first KD loss and the second KD loss until the preset training conditions are reached.

[0080] The streaming speech recognition model obtained by the above training method of the embodiment of the present invention is between the non-streaming speech recognition model and the ordinary wenet streaming speech recognition model in terms of the two indicators of word recognition rate and output delay, and the effect is better than the ordinary wenet streaming speech recognition model.

[0081] Using the above-trained streaming speech recognition model, the embodiment of the present invention also provides a streaming speech recognition method, see Figure 5 , Figure 5 A schematic flow chart of a streaming speech recognition method provided by an embodiment of the present invention includes:

[0082] S501, obtaining audio to be recognized;

[0083] S502, inputting the audio to be recognized into the trained streaming speech recognition model for speech recognition, and obtaining a recognition text corresponding to the audio to be recognized;

[0084] Among them, the streaming speech recognition model is obtained through the streaming speech recognition model training method provided in any of the above embodiments; the recognition accuracy and output delay of the recognized text meet the preset indicators.

[0085] In the embodiment of the present invention, the audio to be recognized may be, but is not limited to, the audio of a user performing voice control on a smart home appliance, a mobile phone, a car computer, or other computer equipment. For example, in a live broadcast scenario, the audio to be recognized may be the voice of the anchor or his game teammates.

[0086] By using the streaming speech recognition model provided by the embodiment of the present invention to perform speech recognition, a recognition effect with high accuracy and low output delay can be obtained for various acoustic environments, such as live broadcast scenarios.

[0087] Based on the same inventive concept, the embodiments of the present invention are Figure 2 Based on this, a streaming speech recognition model training device is given, see Figure 6 , Figure 6 This is a functional module diagram of a streaming speech recognition model training device provided in an embodiment of the present invention. The streaming speech recognition model training device 600 includes: a first acquisition module 610, a determination module 620 and a training module 630.

[0088] The first acquisition module 610 is used to acquire a trained non-streaming speech recognition model; the recognition accuracy and output delay of the non-streaming speech recognition model both meet the preset speech recognition indicators;

[0089] A determination module 620, configured to fix the model parameters of the non-streaming speech recognition model as an initial streaming speech recognition model;

[0090] A training module 630, configured to perform distillation training on the streaming speech recognition model so that the output of the streaming speech recognition model approaches the output of the non-streaming speech recognition model;

[0091] The training module 630 is also used to stop the training when a preset training condition is reached to obtain a trained streaming speech recognition model.

[0092] It is understandable that the first acquisition module 610, the determination module 620 and the training module 630 can be executed in coordination. Figure 2 The various steps shown are used to achieve corresponding technical effects.

[0093] In an optional embodiment, the streaming speech recognition model includes a first CTC decoder and a first attention decoder; the non-streaming speech recognition model includes a second CTC decoder and a second attention decoder; the training module 630 is specifically used to: during the training process, determine the first knowledge distillation loss between the outputs of the first CTC decoder and the second CTC decoder, the second knowledge distillation loss between the outputs of the first attention decoder and the second attention decoder, and the fusion loss between the first CTC decoder and the first attention decoder; and adjust the model parameters of the streaming speech recognition model by combining the first knowledge distillation loss, the second knowledge distillation loss, and the fusion loss to make the output of the streaming speech recognition model close to the output of the non-streaming speech recognition model.

[0094] In an optional embodiment, the training module 630 is specifically used to: determine the output delay offset between the first CTC decoder and the second CTC decoder; for each training audio sample, determine the first knowledge distillation loss based on the output delay offset and the probability of each character output by the first CTC decoder and the second CTC decoder respectively.

[0095] In an optional implementation, the training module 630 is specifically used to: for each training audio sample, calculate the character overlap ratio output by the first CTC decoder and the second CTC decoder under different offset preset values, and use the offset preset value corresponding to the maximum character overlap ratio as the output delay offset.

[0096] In an optional implementation, the training module 630 is specifically used to: for each training audio sample, calculate the second knowledge distillation loss according to the probability of each character contained in each output of the first attention decoder and the second attention decoder.

[0097] In an optional implementation, the training module 630 is specifically used to: fuse the CTC loss of the first CTC decoder and the attention encoder-decoder loss of the first attention decoder according to a preset balance factor to obtain a fused loss.

[0098] Based on the same inventive concept, the embodiments of the present invention are Figure 2 Based on this, a streaming speech recognition device is given, see Figure 7 , Figure 7 The functional module diagram of the streaming speech recognition device provided by the embodiment of the present invention is as follows: The streaming speech recognition device 700 includes: a second acquisition module 710 and a recognition module 720 .

[0099] The second acquisition module 710 is used to acquire the audio to be recognized;

[0100] The recognition module 720 is used to input the audio to be recognized into the trained streaming speech recognition model for speech recognition to obtain the recognition text corresponding to the audio to be recognized; wherein the streaming speech recognition model is obtained by the streaming speech recognition model training method in any of the above embodiments.

[0101] It is understandable that the second acquisition module 710 and the identification module 720 can be executed in coordination. Figure 5 Each step in the process is performed to achieve the corresponding technical effect.

[0102] It should be noted that the division of modules in the above embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0103] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0104] See also Figure 8 , Figure 8 A schematic structural block diagram of a computer device 800 provided in the present application is shown. In some embodiments, the computer device 800 can be used as Figure 1In addition, in some embodiments, the computer device 800 may include a memory 801, a processor 802, and a communication interface 803, and the memory 801, the processor 802, and the communication interface 803 are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0105] The memory 801 can be used to store software programs and modules, such as the program instructions / modules corresponding to the streaming speech recognition model training device 600 or the streaming speech recognition device 700 provided in the present application. The processor 802 executes the software programs and modules stored in the memory 801 to perform various functional applications and data processing, thereby executing the steps of the streaming speech recognition and model training method provided in the present application. The communication interface 803 can be used to communicate signaling or data with other node devices.

[0106] Among them, the memory 801 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable programmable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0107] The processor 802 may be an integrated circuit chip with signal processing capability. The processor 802 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0108] Understandably, Figure 8 The structure shown is for illustration only. The computer device 800 may also include Figure 8More or fewer components as shown, or with Figure 8 Different configurations shown. Figure 8 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0109] Based on the above embodiments, the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a computer, the computer executes the streaming speech recognition model training method or the streaming speech recognition method provided in the above embodiments.

[0110] Based on the above embodiments, the embodiments of the present application further provide a computer program. When the computer program runs on a computer, the computer executes the streaming speech recognition model training method or the streaming speech recognition method provided in the above embodiments.

[0111] Based on the above embodiments, the embodiments of the present application further provide a chip, which is used to read a computer program stored in a memory and to execute the streaming speech recognition model training method or the streaming speech recognition method provided in the above embodiments.

[0112] A computer program product is also provided in an embodiment of the present application, including instructions, which, when executed on a computer, enable the computer to execute the streaming speech recognition model training method or the streaming speech recognition method provided in the above embodiments.

[0113] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by instructions. These instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0114] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0116] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A streaming speech recognition model training method, characterized in that: The method comprises: Acquire a trained non-streaming speech recognition model; the recognition accuracy and output delay of the non-streaming speech recognition model both meet preset indicators; The model parameters of the non-streaming speech recognition model are fixed as the initial streaming speech recognition model; the streaming speech recognition model includes a first CTC decoder and a first attention decoder; the non-streaming speech recognition model includes a second CTC decoder and a second attention decoder; Using the non-streaming speech recognition model, performing distillation training on the streaming speech recognition model so that the output of the streaming speech recognition model approaches the output of the non-streaming speech recognition model, including: during the training process, determining the fusion loss between the first CTC decoder and the first attention decoder, that is, fusing the CTC loss of the first CTC decoder and the attention encoder-decoder loss of the first attention decoder according to a preset balance factor to obtain the fusion loss; When the preset training conditions are met, the training is stopped to obtain the trained streaming speech recognition model.

2. The streaming speech recognition model training method according to claim 1, characterized in that: Using the non-streaming speech recognition model, performing distillation training on the streaming speech recognition model so that the output of the streaming speech recognition model approaches the output of the non-streaming speech recognition model, further comprising: During the training process, determining a first knowledge distillation loss between respective outputs of the first CTC decoder and the second CTC decoder, and a second knowledge distillation loss between respective outputs of the first attention decoder and the second attention decoder; The first knowledge distillation loss, the second knowledge distillation loss, and the fusion loss are combined to adjust the model parameters of the streaming speech recognition model so that the output of the streaming speech recognition model approaches the output of the non-streaming speech recognition model.

3. The streaming speech recognition model training method according to claim 2, characterized in that: Determining a first knowledge distillation loss between respective outputs of the first ctc decoder and the second ctc decoder, comprising: determining an output delay offset between the first ctc decoder and the second ctc decoder; For each training audio sample, the first knowledge distillation loss is determined according to the output delay offset and the probability of each character output by the first CTC decoder and the second CTC decoder respectively.

4. The streaming speech recognition model training method according to claim 3 is characterized in that: Determining an output delay offset between the first CTC decoder and the second CTC decoder, comprising: For each training audio sample, the character overlap ratio output by the first CTC decoder and the second CTC decoder under different preset offset values ​​is calculated, and the preset offset value corresponding to the maximum character overlap ratio is used as the output delay offset.

5. The streaming speech recognition model training method according to claim 2, characterized in that: Determining a second knowledge distillation loss between respective outputs of the first attention decoder and the second attention decoder, comprising: For each training audio sample, the second knowledge distillation loss is calculated according to the probability of each character contained in each output of the first attention decoder and the second attention decoder.

6. A streaming speech recognition method, characterized in that: include: Get the audio to be recognized; The audio to be recognized is input into the trained streaming speech recognition model for speech recognition to obtain a recognized text corresponding to the audio to be recognized; wherein the streaming speech recognition model is obtained by the streaming speech recognition model training method according to any one of claims 1 to 5; the recognition accuracy and output delay of the recognized text meet preset indicators.

7. A streaming speech recognition model training device, characterized in that: include: An acquisition module, used to acquire a trained non-streaming speech recognition model; the recognition accuracy and output delay of the non-streaming speech recognition model both meet preset speech recognition indicators; A determination module, used to fix the model parameters of the non-streaming speech recognition model as an initial streaming speech recognition model; the streaming speech recognition model includes a first CTC decoder and a first attention decoder; The non-streaming speech recognition model includes a second CTC decoder and a second attention decoder; A training module, used for performing distillation training on the streaming speech recognition model so that the output of the streaming speech recognition model approaches the output of the non-streaming speech recognition model, comprising: during the training process, determining the fusion loss between the first CTC decoder and the first attention decoder, that is, fusing the CTC loss of the first CTC decoder and the attention encoder-decoder loss of the first attention decoder according to a preset balance factor to obtain the fusion loss; The training module is also used to stop training when a preset training condition is reached to obtain the trained streaming speech recognition model.

8. A streaming speech recognition device, characterized in that: include: An acquisition module, used to acquire the audio to be recognized; A recognition module is used to input the audio to be recognized into a trained streaming speech recognition model for speech recognition, so as to obtain a recognition text corresponding to the audio to be recognized; wherein the streaming speech recognition model is obtained by the streaming speech recognition model training method as described in any one of claims 1 to 5; and the recognition accuracy and output delay of the recognition text meet preset indicators.

9. A computer device, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the method according to any one of claims 1 to 6.

Images