Voice recognition method and apparatus, and nonvolatile storage medium and electronic device

Abstract

Disclosed in the present application are a voice recognition method and apparatus, and a nonvolatile storage medium and an electronic device. The method comprises: continuously collecting an audio frame to be recognized, and inputting said collected audio frame into a target recognition model, wherein the target recognition model comprises a plurality of attention layers, which are sequentially connected, and the target recognition model is used for extracting features of said audio frame by means of the plurality of attention layers and outputting a recognition result corresponding to said audio frame; in the process of the target recognition model recognizing said audio frame, updating historical states of the attention layers in the target recognition model, so as to obtain an updated target recognition model; and on the basis of the updated target recognition model, determining the recognition result for said audio frame.

Description

Speech recognition method, device, non-volatile storage medium and electronic device

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 2023117763663, filed on December 21, 2023, entitled “Speech recognition method, device, non-volatile storage medium and electronic device,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of audio processing, and more specifically, to a speech recognition method, device, non-volatile storage medium, and electronic device. Background Art

[0004] In the related art, when recognizing audio, a common approach is to fill the audio to be recognized into pre-set blocks and then perform recognition on the audio to be recognized in each block. The problem with this approach is that if the pre-set blocks are not full of audio to be recognized, no recognition results will be output, resulting in an unsmooth output of the recognition results.

[0005] To address the above-mentioned problems, no effective solutions have been proposed so far.

[0006] Summary of the Invention

[0007] According to the first aspect of the embodiment of the present application, a speech recognition method is provided, including: continuously collecting audio frames to be recognized, and inputting the collected audio frames to be recognized into a target recognition model, wherein the target recognition model includes multiple attention layers connected in sequence, and the target recognition model is used to extract features of the audio frames to be recognized through the multiple attention layers and output recognition results corresponding to the audio frames to be recognized; in the process of the target recognition model recognizing the audio frames to be recognized, the historical states of each attention layer in the target recognition model are updated to obtain an updated target recognition model, wherein the historical states of the attention layers include the query vector, key vector and value vector of the attention layer at the target moment; and the recognition result of the audio frame to be recognized is determined based on the updated target recognition model.

[0008] Optionally, the step of determining the recognition result of the audio frame to be recognized based on the updated target recognition model includes: updating the decoding path of the target recognition model based on the updated historical status of each attention layer, wherein the decoding path is used to generate recognition results based on the features extracted by the attention layer; and determining the recognition result of the audio frame to be recognized through the updated decoding path.

[0009] Optionally, the step of determining the recognition result of the audio frame to be recognized through the updated decoding path includes: determining the recognition result of the audio frame to be recognized at the current moment through the updated decoding path; and determining the historical audio frames to be recognized input into the target recognition model within a preset time period; and determining the recognition results corresponding to the historical audio frames to be recognized again through the updated decoding path.

[0010] Optionally, in the process of the target recognition model identifying the audio frame to be recognized, the step of updating the historical status of each attention layer in the target recognition model includes: determining an update interval and an update step, wherein the update interval is the number of audio frames to be recognized input into the target recognition model when the historical status of each attention layer is updated twice adjacently, and the update step is the number of historical state sets of each attention layer that need to be updated, each historical state set corresponds to a frame of target audio frame to be recognized, and the historical state set includes the status of each attention layer corresponding to the target audio frame to be recognized; determining an update index based on the update step, wherein the update index is the serial number of the audio frame to be recognized, and the number of audio frames to be recognized between two adjacent update indexes is equal to the number of historical state sets indicated by the update step; updating the historical status of each attention layer based on the update index, update interval and update step.

[0011] Optionally, the step of updating the historical state of each attention layer based on the update index, update interval and update step size includes: when the serial number of the audio frame to be recognized input into the target recognition model is greater than the first update index and less than the second update index, when the number of audio frames to be recognized in the first target audio frame set to be recognized is equal to the update interval, updating the historical state set corresponding to the first target audio frame set to be recognized based on the first target audio frame set to be recognized, wherein the first target audio frame to be recognized includes the audio frames to be recognized input into the target recognition model starting from the audio frame to be recognized with the serial number of the first update index; when the serial number of the audio frame to be recognized input into the target recognition model is equal to the second update index, updating the historical state set corresponding to the second target audio frame set to be recognized based on the second target audio frame set to be recognized, wherein the second target audio set to be recognized includes audio frames to be recognized with serial numbers not less than the first update index and not greater than the second update index.

[0012] Optionally, the target recognition model is trained in the following manner: obtaining a preset mask matrix, wherein the preset mask matrix includes multiple update blocks determined according to the update step size, and there is an overlapping area between adjacent update blocks; obtaining a causal mask matrix; in each training process, randomly selecting the preset mask matrix or the causal mask matrix according to a preset probability to train the target recognition model.

[0013] Optionally, the step of updating the historical state of each attention layer in the target recognition model includes: determining the input features of the previous attention layer of the target attention layer, wherein the target attention layer is any attention layer except the first layer, and the first layer is the attention layer whose input is the audio frame to be recognized; updating the historical state of the target attention layer based on the historical state of the target attention layer and the input features of the previous attention layer.

[0014] According to the second aspect of the embodiment of the present application, a speech recognition device is also provided, including: a first processing module, used to continuously collect audio frames to be recognized, and input the collected audio frames to be recognized into a target recognition model, wherein the target recognition model includes multiple attention layers connected in sequence, and the target recognition model is used to extract the features of the audio frames to be recognized through multiple attention layers and output recognition results corresponding to the audio frames to be recognized; a second processing module, used to update the historical status of each attention layer in the target recognition model during the process of the target recognition model recognizing the audio frames to be recognized, to obtain an updated target recognition model, wherein the historical status of the attention layer includes the query vector, key vector and value vector of the attention layer at the target moment; a third processing module, used to determine the recognition result of the audio frame to be recognized based on the updated target recognition model.

[0015] According to a third aspect of an embodiment of the present application, a non-volatile storage medium is further provided, in which a program is stored. When the program is running, the device where the non-volatile storage medium is located is controlled to execute a speech recognition method.

[0016] According to a fourth aspect of an embodiment of the present application, an electronic device is further provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the speech recognition method is executed when the program is run.

[0017] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] FIG1 is a schematic structural diagram of a computer terminal (mobile terminal) provided according to an embodiment of the present application;

[0020] FIG2 is a flow chart of a speech recognition method according to an embodiment of the present application;

[0021] FIG3 is a schematic diagram of a training mask matrix provided according to an embodiment of the present application;

[0022] FIG4 is a schematic diagram of a model training result provided according to an embodiment of the present application;

[0023] FIG5 is a flow chart of a speech recognition method according to another embodiment of the present application;

[0024] FIG6 is a schematic structural diagram of a speech recognition device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0028] Stability: In streaming speech recognition, some intermediate output recognition results change due to the continuous changes in the decoding path. Models with high stability have fewer historical output results that change, making them easier for users to understand or integrate with downstream tasks.

[0029] Causal model: A model that relies only on the input speech frames during calculation and does not rely on any future speech information.

[0030] Theoretical Encoder Latency (EIL): This is the time delay caused by the model encoder structure and training settings. This delay is unavoidable, meaning that the latency of the streaming speech recognition system will ultimately approach this theoretical latency. This is often referred to simply as latency at the model algorithm level.

[0031] Streaming speech recognition involves converting audio input into corresponding text and outputting it in real time. Streaming speech recognition has been widely used in various scenarios, such as intelligent customer service and real-time captioning. Unlike traditional speech recognition, which requires a complete audio input before returning the corresponding text, streaming speech recognition requires real-time feedback with low latency and high accuracy.

[0032] Most of the current streaming speech recognition models in related technologies are based on a chunk-based attention mechanism model. They need to wait for the input audio to fill a chunk before performing calculations and outputting the recognition results in chunks. The time delay of such models in actual applications depends on the size of the set chunk. Smaller chunks mean faster response times, but the recognition effect is often worse than larger chunks. To ensure recognition accuracy, the chunk size is usually not less than 500ms, so there will be noticeable pauses in the output. In addition, because the model tends to wait for more audio input to achieve better performance during training, the actual output of the recognition result is delayed, and neither real-time nor smoothness can be guaranteed.

[0033] Related technologies offer two solutions to this problem. One involves shifting the timestamp of the model's output forward through training, or by using a strategy to pad the input during decoding to shift the output bytes forward. However, this approach only allows a small number of recognition results to appear in the previous chunk. While this reduces average character latency, the output is still in chunks, and pauses can still occur. Another approach involves constructing a system consisting of a fast and slow model, where the fast model is a causal model and the slow model is a standard streaming model. The system calculates frame by frame as the audio input accumulates. After a certain period of time, it calls the standard streaming model to recalculate the previous input and modify the previous output. This approach ensures real-time, uninterrupted results, but using two models makes deployment inconvenient. Furthermore, because the causal model's recognition results are significantly inferior to those of the standard streaming model, the output results are constantly changing (for example, the result of the 5th second causes the result of the 1st second to be adjusted, requiring updates between 1 and 5 seconds). This makes the previous output unusable and leads to poor stability.

[0034] In order to solve the problem that the streaming speech recognition solution provided in the related art cannot meet both the requirements of low response time and the accuracy and stability of real-time output results, a relevant solution is provided in the embodiments of the present application, which is described in detail below.

[0035] According to an embodiment of the present application, a method embodiment of a speech recognition method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a speech recognition method. As shown in Figure 1, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used in the figure to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that the structure shown in Figure 1 is only for illustration and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may also include more or fewer components than those shown in Figure 1, or have a configuration different from that shown in Figure 1.

[0037] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0038] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the speech recognition method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned speech recognition method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0039] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0040] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0041] In the above operating environment, an embodiment of the present application provides a speech recognition method, as shown in FIG2 , which includes the following steps:

[0042] Step S202: continuously collect audio frames to be recognized and input the collected audio frames to be recognized into a target recognition model, wherein the target recognition model includes multiple attention layers connected in sequence, and the target recognition model is used to extract features of the audio frames to be recognized through the multiple attention layers and output recognition results corresponding to the audio frames to be recognized;

[0043] In the technical solution provided in step S202, the target recognition model can obtain the audio frames to be recognized in real time and extract the features of each audio frame to be recognized input into the target recognition model in a streaming manner, and calculate and decode the features of the audio frames to be recognized frame by frame, thereby obtaining the recognition result of the current frame.

[0044] As an optional implementation, the above-mentioned target recognition model can be any model based on the attention mechanism, such as a Transformer model based on the attention mechanism.

[0045] Step S204: During the process of the target recognition model identifying the audio frame to be recognized, the historical state of each attention layer in the target recognition model is updated to obtain an updated target recognition model, wherein the historical state of the attention layer includes the query vector, key vector, and value vector of the attention layer at the target time.

[0046] Specifically, the above query vector can be expressed as The above key vector can be expressed as The above value vector can be expressed as in, represents the real number field, d q d k and d v Both represent dimensions, and d q =d k The superscript l represents the number of the corresponding attention layer, and the subscript t represents the target time corresponding to the vector or the sequence number of the corresponding audio frame to be recognized. When t represents the target time, the target time refers to the time when any audio frame to be recognized is input into the model. When t represents a sequence number, the sequence number can be the sequence number of any audio frame to be recognized input into the model.

[0047] In addition, in the target recognition model, the result calculated by the l-1 attention layer at time t-1 is sent to the l-1 layer for calculation. That is, the result obtained at time t of the l-1 layer depends on the results of the l-1 layer at time t-1 and before and the result of the l-1 layer input at time t. When calculating the result at time t, it is necessary to concatenate the previous state and the new input after passing through the linear layer K and V, and then calculate the attention score. K is the key vector of the current layer at the current moment, and V is the value vector of the current layer at the current moment. The above attention score can be used to reflect the weight of each feature in the identified audio frame to be recognized, that is, the importance of the feature, so that the model pays more attention to important features and improves the accuracy of the recognition results.

[0048] For the convenience of representation, the calculation result of the lth layer at the tth moment can be calculated according to the softmax (normalization) calculation formula Write the following formula:

[0049] In the above formula, h represents the number of saved historical states, and T represents the matrix transpose. As can be seen from the above formula, the output result is consistent with the historical state of each layer. as well as And the calculation result of the previous layer at the current moment as the input of this layer There is a significant correlation. That is, if the historical state is inaccurate, the output result is likely to be problematic. Furthermore, historical information will be reused repeatedly, making the results even more inaccurate in subsequent calculations, leading to accumulated errors. The calculation of historical state is related to both previous history and the future information currently involved in the calculation. The more future information there is, the more accurate the state calculation. Therefore, if the chunk size is blindly reduced to reduce theoretical latency, the calculation of the historical state at each moment and at each layer will lack more future information than when the chunk size is larger, resulting in even more inaccurate calculated states and a significant decrease in model recognition accuracy.

[0050] In view of this situation, the present application provides a method for The method of adjusting the dependent states K and V is used to correct the inaccurate states caused by reducing the chunk size. Specifically, assuming t r is the current index of the correction, t r ≥th,. That is, t needs to be corrected r The previous part, t r The part at time t remains unchanged. The adjusted K, V and Q as the output of the current layer can be expressed as the following formula:

[0051] In the above formula, uppercase K and V represent the state before correction, and lowercase k, v, and q represent the state after correction.

[0052] In the technical solution provided in step S204, in the process of the target recognition model identifying the audio frame to be recognized, the step of updating the historical status of each attention layer in the target recognition model includes: determining the update interval and the update step, wherein the update interval is the number of audio frames to be recognized input into the target recognition model when the historical status of each attention layer is updated twice adjacently, and the update step is the number of historical state sets of each attention layer that need to be updated, each historical state set corresponds to a frame of target audio frame to be recognized, and the historical state set includes the status of each attention layer corresponding to the target audio frame to be recognized; determining the update index according to the update step, wherein the update index is the serial number of the audio frame to be recognized, and the number of audio frames to be recognized between two adjacent update indexes is equal to the number of historical state sets indicated by the update step; updating the historical status of each attention layer according to the update index, the update interval and the update step.

[0053] The historical states of each attention layer corresponding to the target audio frame to be recognized are the states corresponding to the target audio frame to be recognized based on the set historical state index. In addition, the difference between the historical state sets corresponding to two adjacent audio frames to be recognized is only the state of each attention layer obtained when calculating the later input audio frame to be recognized.

[0054] In some embodiments of the present application, the step of updating the historical state of each attention layer in the target recognition model includes: determining the input features of the previous attention layer of the target attention layer, wherein the target attention layer is any attention layer except the first layer, and the first layer is the attention layer whose input is the audio frame to be recognized; updating the historical state of the target attention layer based on the historical state of the target attention layer and the input features of the previous attention layer.

[0055] Specifically, when updating the historical states of each attention layer in the target recognition model, the state of the first layer No adjustments will be made, and starting from the second layer, the correction will depend on the history of the current layer and the input of the previous layer. The output of each layer can be expressed as the following formula:

[0056] As an optional embodiment, the step of updating the historical state of each attention layer according to the update index, update interval and update step size includes: when the serial number of the audio frame to be recognized input into the target recognition model is greater than the first update index and less than the second update index, when the number of audio frames to be recognized in the first target audio frame set to be recognized is equal to the update interval, the historical state set corresponding to the first target audio frame set to be recognized is updated according to the first target audio frame set to be recognized, wherein the first target audio frame to be recognized includes the audio frames to be recognized input into the target recognition model starting from the audio frame to be recognized with the serial number of the first update index; when the serial number of the audio frame to be recognized input into the target recognition model is equal to the second update index, the historical state set corresponding to the second target audio frame set to be recognized is updated according to the second target audio frame set to be recognized, wherein the second target audio set to be recognized includes audio frames to be recognized with serial numbers not less than the first update index and not greater than the second update index.

[0057] Specifically, for ease of calculation, we can define the correction interval ν as the time between two corrections. During this interval, the model calculates and outputs the recognition result in a causal manner. The correction step size σ is defined as the number of encoder states that need to be corrected.

[0058] The historical state before the correction step index is determined as the corrected information and will only be used for subsequent calculations without further correction. To ensure the accuracy of the real-time output and minimize the difference between the states before and after the first correction, before the input reaches the σth frame (that is, when the current input frame number is not the update index number), all current states can be corrected every ν using all inputs smaller than the step size. When the input audio reaches the σth frame, all previous historical states can be uniformly modified again. The state is then corrected at each correction interval using the correction step information.

[0059] Assume t r is the sequence number of the audio frame to be identified corresponding to the time point in the current correction interval, then at t r -σ+1 to t r -1, the historical state corresponding to the audio frame to be recognized input will be adjusted. r The state before -σ+1 remains unchanged. m ∈[t r ,t r +ν) represents the sequence number of the audio frame to be recognized corresponding to the time point after one correction but not yet the next correction. Similarly, the state of the first layer Without making any adjustments, starting from the second layer, the correction is based on the history of the current layer and the input of the previous layer. The final output is calculated as follows:

[0060] In some embodiments of the present application, the target recognition model is trained in the following manner: obtaining a preset mask matrix, wherein the preset mask matrix includes multiple update blocks determined according to the update step size, and there is an overlapping area between adjacent update blocks; obtaining a causal mask matrix, wherein the causal mask matrix is ​​shown in Figure 3; in each training process, the preset mask matrix or the causal mask matrix is ​​randomly selected according to a preset probability to train the target recognition model.

[0061] Specifically, in order to improve training efficiency through parallel training, an attention mask matrix (attention mask) for the encoder as shown in Figure 3 is also provided in the embodiment of the present application to control the attention calculation range. The gray part of each mask matrix in Figure 3 represents the part involved in the calculation. The left side of Figure 3 is the causal model mask matrix calculated frame by frame. The Q of each frame can only be calculated with K in the absence of any future frames. The middle of Figure 3 is an ordinary streaming model, which allows Q to consider the information in the current chunk and the historical chunk information when calculating with K. The right side of Figure 3 is the training matrix provided in the embodiment of the present application. The matrix range where each correction step is located is called a revision block, which is the update block mentioned above. Each correction block can overlap, that is, the frame at the correction interval index will also be corrected, and future information will be added in subsequent correction operations. Compared with the original causal model attention calculation range, additional information can be introduced in training through the attention mask matrix, which is equivalent to introducing overlapping parts between blocks.

[0062] During training, the attention mask matrix shown on the right side of Figure 3 can be converted into a 0-1 Boolean matrix and applied after the QK dot product to limit the calculation range of the QK dot product and V when calculating the attention score.

[0063] Furthermore, during decoding, the causal output of the model trained solely on the right side of Figure 3 may not match the neurons used during training, which can lead to poor results. To ensure the stability of the output and reduce the time required to adjust the decoding path, dynamic mask training can be used. During model training, either the causal mask matrix on the left side of Figure 3 or the modified mask matrix on the right side of Figure 3 is randomly selected with a probability of 30% or 70% to ensure that the receptive fields of the neurons used during decoding and training are the same.

[0064] Step S206 , determining the recognition result of the audio frame to be recognized based on the updated target recognition model.

[0065] It should be noted that in the embodiments of this application, the recognition result corresponding to each audio frame to be recognized will be output. In the target recognition model provided in this application, after the audio frame to be recognized is initially input into the target recognition model, the target recognition model will output the recognition result based on the input audio frame to be recognized. In other words, the target recognition model will continuously output the recognition result corresponding to each audio frame to be recognized while the audio frame to be recognized is continuously input.

[0066] In the technical solution provided in step S206, the step of determining the recognition result of the audio frame to be recognized based on the updated target recognition model includes: updating the decoding path of the target recognition model based on the updated historical status of each attention layer, wherein the decoding path is used to generate the recognition result based on the features extracted by the attention layer; and determining the recognition result of the audio frame to be recognized through the updated decoding path.

[0067] As an optional implementation, the step of determining the recognition result of the audio frame to be recognized through the updated decoding path includes: determining the recognition result of the audio frame to be recognized at the current moment through the updated decoding path; and determining the historical audio frames to be recognized input into the target recognition model within a preset time period; and determining the recognition results corresponding to the historical audio frames to be recognized again through the updated decoding path.

[0068] Specifically, during the recognition process, as the audio frames to be recognized are input, the amount of historical information that has been recognized will continue to increase. In order to save computing time and consumption, the previously calculated output can be cached as a state and used in the next calculation. The state is usually represented in the form of a tensor. Assuming that the length of the historical information used is h, then after the calculation at time t-1 of the l-1 layer, only the calculation results from time t to time t-1 will be saved. As cached information, the part before th is discarded to reduce computational cost and time. When calculating the model at layer l, its input is the output of layer l-1, and the calculation of the output of layer l-1 requires the use of previously cached information. The mapping f(·) represents the projection, residual, and other operations required in the calculation, and c represents the length of future information accessible at time t. The formula for calculation using cached information is as follows:

[0069] In addition, for the target recognition model based on the attention mechanism, the state of the l-th layer cache after each calculation is as well as For the speech recognition method provided in the embodiment of the present application, additional buffering is required Used to update the historical state of each attention layer. That is, the state calculated at time t-1 will be cached and maintained in a queue, always saving the state from time t to time t-1.

[0070] When updating the historical state, the cached Q can be used to update the historical moments K and V, and replace the state originally cached. In this way, during the next calculation, the cached state in the queue can be queried and spliced ​​with the current result on the tensor before calculation. Finally, it can be seen from Figure 4 that, compared with the existing streaming speech recognition method, the streaming speech recognition method provided in the embodiment of the present application can ensure a processing speed close to the causal model processing speed while ensuring a higher recognition rate.

[0071] The embodiment of the present application further provides a streaming speech recognition process as shown in FIG5 , including the following steps:

[0072] Step S502: acquiring the audio to be recognized in real time and extracting the features of the audio to be recognized in a streaming manner;

[0073] Step S504: Calculate and decode the features of the audio to be recognized frame by frame and output the recognition result of the current frame;

[0074] Step S506, determining whether the serial number of the audio frame to be recognized input into the model is a modified index signal, if not, jump to step S502, if yes, jump to step S508;

[0075] Step S508, using the historical state cached by the model to update the historical state within the correction interval;

[0076] Step S510, adjusting the current decoding path of the target recognition model according to the updated state;

[0077] Step S512: updating the previously output recognition result using the updated model, and outputting a new recognition result using the updated model.

[0078] By continuously collecting audio frames to be recognized and inputting the collected audio frames to be recognized into a target recognition model, wherein the target recognition model includes multiple attention layers connected in sequence, and the target recognition model is used to extract features of the audio frames to be recognized through multiple attention layers and output recognition results corresponding to the audio frames to be recognized; in the process of the target recognition model recognizing the audio frames to be recognized, the historical states of each attention layer in the target recognition model are updated to obtain an updated target recognition model, wherein the historical state of the attention layer includes the query vector, key vector and value vector of the attention layer at the target moment; a method of determining the recognition result of the audio frame to be recognized based on the updated target recognition model, and outputting the recognition result of each audio frame to be recognized by the target recognition model that updates the historical state of the attention layer in real time during the recognition process, thereby achieving the purpose of outputting recognition results frame by frame and avoiding the output interval of adjacent recognition results being too long, thereby achieving the technical effect of improving the output fluency of the recognition result, and then solving the technical problem of non-smooth output of the recognition result caused by the block processing method adopted in the related technology when recognizing audio.

[0079] An embodiment of the present application provides a speech recognition device, and Figure 6 is a structural schematic diagram of the device. As can be seen from Figure 6, the device includes: a first processing module 60, which is used to continuously collect audio frames to be recognized and input the collected audio frames to be recognized into a target recognition model, wherein the target recognition model includes multiple attention layers connected in sequence, and the target recognition model is used to extract features of the audio frames to be recognized through multiple attention layers and output recognition results corresponding to the audio frames to be recognized; a second processing module 62, which is used to update the historical states of each attention layer in the target recognition model during the process of the target recognition model recognizing the audio frames to be recognized, to obtain an updated target recognition model, wherein the historical states of the attention layers include the query vector, key vector and value vector of the attention layer at the target moment; a third processing module 64, which is used to determine the recognition results of the audio frames to be recognized based on the updated target recognition model.

[0080] In some embodiments of the present application, the second processing module 62 is also used to: determine the update interval and update step, wherein the update interval is the number of audio frames to be identified input into the target recognition model when the historical states of each attention layer are updated twice adjacently, and the update step is the number of historical state sets of each attention layer that need to be updated, each historical state set corresponds to a frame of target audio frame to be identified, and the historical state set includes the states of each attention layer corresponding to the target audio frame to be identified; determine the update index based on the update step, wherein the update index is the serial number of the audio frame to be identified, and the number of audio frames to be identified between two adjacent update indexes is equal to the number of historical state sets indicated by the update step; update the historical state of each attention layer based on the update index, update interval and update step.

[0081] In some embodiments of the present application, the second processing module 62 is also used to: when the serial number of the audio frame to be identified input into the target recognition model is greater than the first update index and less than the second update index, when the number of audio frames to be identified in the first target audio frame set to be identified is equal to the update interval, update the historical status set corresponding to the first target audio frame set to be identified according to the first target audio frame set to be identified, wherein the first target audio frame to be identified includes the audio frames to be identified input into the target recognition model starting from the audio frame to be identified with the serial number of the first update index; when the serial number of the audio frame to be identified input into the target recognition model is equal to the second update index, update the historical status set corresponding to the second target audio frame set to be identified according to the second target audio frame set to be identified, wherein the second target audio set to be identified includes audio frames to be identified with serial numbers not less than the first update index and not greater than the second update index.

[0082] In some embodiments of the present application, the second processing module 62 is also used to: obtain a preset mask matrix, wherein the preset mask matrix includes multiple update blocks determined according to the update step size, and there is an overlapping area between adjacent update blocks; obtain a causal mask matrix; and in each training process, randomly select a preset mask matrix or a causal mask matrix according to a preset probability to train the target recognition model.

[0083] In some embodiments of the present application, the second processing module 62 is also used to: determine the input features of the previous attention layer of the target attention layer, wherein the target attention layer is any attention layer except the first layer, and the first layer is the attention layer whose input is the audio frame to be identified; update the historical state of the target attention layer based on the historical state of the target attention layer and the input features of the previous attention layer.

[0084] In some embodiments of the present application, the third processing module 64 is also used to: update the decoding path of the target recognition model based on the updated status of each attention layer, wherein the decoding path is used to generate recognition results based on the features extracted by the attention layer; and determine the recognition results of the audio frame to be recognized through the updated decoding path.

[0085] In some embodiments of the present application, the third processing module 64 is also used to: determine the recognition result of the audio frame to be recognized at the current moment through the updated decoding path; and determine the historical audio frames to be recognized input into the target recognition model within a preset time period; and determine the recognition results corresponding to the historical audio frames to be recognized again through the updated decoding path.

[0086] It should be noted that the various modules in the above-mentioned speech recognition device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0087] According to another aspect of an embodiment of the present application, a non-volatile storage medium is also provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute a speech recognition method: continuously collecting audio frames to be recognized, and inputting the collected audio frames to be recognized into a target recognition model, wherein the target recognition model includes multiple attention layers connected in sequence, and the target recognition model is used to extract features of the audio frames to be recognized through the multiple attention layers and output recognition results corresponding to the audio frames to be recognized; in the process of the target recognition model recognizing the audio frames to be recognized, the historical states of each attention layer in the target recognition model are updated to obtain an updated target recognition model, wherein the historical states of the attention layers include the query vector, key vector and value vector of the attention layer at the target moment; and the recognition results of the audio frames to be recognized are determined based on the updated target recognition model.

[0088] According to another aspect of an embodiment of the present application, an electronic device is also provided, including: a memory and a processor, the processor being used to run a program stored in the memory, wherein the following speech recognition method is executed when the program is running: continuously collecting audio frames to be recognized, and inputting the collected audio frames to be recognized into a target recognition model, wherein the target recognition model includes multiple attention layers connected in sequence, and the target recognition model is used to extract features of the audio frames to be recognized through multiple attention layers and output recognition results corresponding to the audio frames to be recognized; in the process of the target recognition model recognizing the audio frames to be recognized, the historical states of each attention layer in the target recognition model are updated to obtain an updated target recognition model, wherein the historical states of the attention layers include the query vector, key vector and value vector of the attention layer at the target moment; and the recognition results of the audio frames to be recognized are determined based on the updated target recognition model.

[0089] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0091] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0092] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0093] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0094] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A speech recognition method, comprising: Continuously collecting audio frames to be recognized and inputting the collected audio frames to be recognized into a target recognition model, wherein the target recognition model includes a plurality of sequentially connected attention layers, and the target recognition model is used to extract features of the audio frames to be recognized through the plurality of attention layers and output a recognition result corresponding to the audio frames to be recognized; During the process of the target recognition model recognizing the audio frame to be recognized, updating the historical state of each attention layer in the target recognition model to obtain an updated target recognition model, wherein the historical state of the attention layer includes the query vector, key vector, and value vector of the attention layer at the target time; The recognition result of the audio frame to be recognized is determined according to the updated target recognition model.

2. The speech recognition method according to claim 1, wherein: The step of determining the recognition result of the audio frame to be recognized based on the updated target recognition model includes: updating a decoding path of the target recognition model according to the updated historical states of each of the attention layers, wherein the decoding path is used to generate the recognition result according to the features extracted by the attention layers; The recognition result of the audio frame to be recognized is determined through the updated decoding path.

3. The speech recognition method according to claim 2, wherein: The step of determining the recognition result of the to-be-recognized audio frame through the updated decoding path includes: Determining the recognition result of the audio frame to be recognized at the current moment through the updated decoding path; and Determining historical audio frames to be recognized that are input into the target recognition model within a preset time period; The recognition result corresponding to the historical audio frame to be recognized is determined again through the updated decoding path.

4. The speech recognition method according to claim 1, wherein: In the process of the target recognition model recognizing the audio frame to be recognized, the step of updating the historical state of each attention layer in the target recognition model includes: Determine an update interval and an update step size, wherein the update interval is the number of audio frames to be recognized input into the target recognition model when the historical states of each of the attention layers are updated twice in a row, and the update step size is the number of historical state sets of each of the attention layers that need to be updated, each of the historical state sets corresponding to a frame of target audio frame to be recognized, and the historical state sets include the states of each of the attention layers corresponding to the target audio frame to be recognized; Determining an update index according to the update step size, wherein the update index is a sequence number of the audio frame to be identified, and the number of audio frames to be identified between two adjacent update indexes is equal to the number of the historical state sets indicated by the update step size; The historical state of each of the attention layers is updated according to the update index, the update interval and the update step.

5. The speech recognition method according to claim 4, wherein: The step of updating the historical state of each of the attention layers according to the update index, the update interval, and the update step size includes: In a case where the sequence number of the audio frame to be recognized input into the target recognition model is greater than the first update index and less than the second update index, when the number of audio frames to be recognized in the first target audio frame set to be recognized is equal to the update interval, updating the historical state set corresponding to the first target audio frame set to be recognized according to the first target audio frame set to be recognized, wherein the first target audio frame to be recognized includes the audio frames to be recognized input into the target recognition model starting from the audio frame to be recognized whose sequence number is the first update index; When the serial number of the audio frame to be identified input into the target recognition model is equal to the second update index, the historical state set corresponding to the second target audio frame set to be identified is updated according to the second target audio frame set to be identified, wherein the second target audio set to be identified includes audio frames to be identified whose serial numbers are not less than the first update index and not greater than the second update index. The speech recognition method according to claim 4 , wherein: The object recognition model is trained in the following way: Obtaining a preset mask matrix, wherein the preset mask matrix includes a plurality of update blocks determined according to the update step size, and there are overlapping areas between adjacent update blocks; Get the causal mask matrix; In each training process, the preset mask matrix or the causal mask matrix is ​​randomly selected according to a preset probability to train the target recognition model.

7. The speech recognition method according to claim 1, wherein: The step of updating the historical state of each of the attention layers in the target recognition model includes: Determine the input features of the previous attention layer of the target attention layer, wherein the target attention layer is any attention layer except the first layer, and the first layer is the attention layer whose input is the audio frame to be recognized; The historical state of the target attention layer is updated according to the historical state of the target attention layer and the input features of the previous attention layer.

8. A speech recognition device comprising: a first processing module, configured to continuously collect audio frames to be recognized and input the collected audio frames to be recognized into a target recognition model, wherein the target recognition model includes a plurality of sequentially connected attention layers, and the target recognition model is configured to extract features of the audio frames to be recognized through the plurality of attention layers and output a recognition result corresponding to the audio frames to be recognized; A second processing module is configured to update the historical state of each attention layer in the target recognition model during the process of the target recognition model recognizing the audio frame to be recognized, so as to obtain an updated target recognition model, wherein the historical state of the attention layer includes the query vector, key vector, and value vector of the attention layer at the target time; The third processing module is used to determine the recognition result of the audio frame to be recognized based on the updated target recognition model.

9. A non-volatile storage medium having a program stored therein, wherein: When the program is running, the device where the non-volatile storage medium is located is controlled to execute the speech recognition method according to any one of claims 1 to 7.

10. An electronic device comprising: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the speech recognition method according to any one of claims 1 to 7 when running.

Images