Systems and methods for speech recognition

Abstract

Systems and methods for speech recognition. A computer-implemented method for speech recognition, the method comprising: receiving a frame of speech audio; encoding the received frame; determining a context vector from the encoding of the received frame; deriving an action from the context vector; responsive to the action satisfying a predetermined condition, deriving a token from the context vector; and performing a function based on the token, wherein the function comprises at least one of a text output or a command execution.

Description

Technical Field

[0001] The embodiments described herein relate to systems and methods for speech recognition. Background Technology

[0002] Speech recognition methods and systems receive speech audio and recognize its content, such as text content. Speech recognition systems include hybrid systems and may incorporate an acoustic model (AM), a pronunciation lexicon, and a language model (LM) to determine the content of the speech audio, such as decoding speech. Early hybrid systems utilized Hidden Markov Models (HMMs) or similar statistical methods for the acoustic and / or language models. Later hybrid systems utilized neural networks for at least one of the acoustic and / or language models. These systems can be referred to as deep speech recognition systems.

[0003] Speech recognition systems with end-to-end architectures have also been introduced. In these systems, a single neural network can be used, in which an acoustic model, a pronunciation dictionary, and a language model can be implicitly integrated. This single neural network can be a recurrent neural network. Recently, transformer models have been used for speech recognition systems. Transformer models can use a self-attention mechanism for speech recognition, capturing dependencies regardless of their distance. Transformer models can employ an encoder-decoder framework. Attached Figure Description

[0004] The system and method according to a non-limiting example will now be described with reference to the accompanying drawings, in which:

[0005] Figure 1A This is an example of a voice assistant system according to the example embodiment;

[0006] Figure 1B This is an example of a speech transcription system according to an exemplary embodiment;

[0007] Figure 1C This is a flowchart of a method for providing voice assistance according to an example embodiment;

[0008] Figure 1D This is a flowchart of a method for performing speech transcription according to an example embodiment;

[0009] Figure 2 A schematic diagram illustrating a method for speech recognition according to an embodiment;

[0010] Figure 3A This is a schematic diagram of a method for speech recognition according to an embodiment;

[0011] Figure 3B This is a schematic diagram of a method for speech recognition according to an embodiment;

[0012] Figure 4 This is a schematic diagram of the components of a speech recognition method.

[0013] Figure 5 This is a schematic diagram of a system for performing speech recognition according to an example embodiment;

[0014] Figure 6 This is a flowchart of a method for preprocessing speech audio;

[0015] Figure 7 This is a flowchart of a method for training a speech recognition system;

[0016] Figure 8 This is a flowchart of a method for training a speech recognition system according to an embodiment;

[0017] Figure 9 A graph representing the loss during the training of a system used for speech recognition;

[0018] Figure 10 This is a flowchart of the speech recognition method;

[0019] Figure 11 This is a schematic diagram of method 1300;

[0020] Figure 12A This is a schematic diagram of a system for performing speech recognition according to an example embodiment;

[0021] Figure 12B This is a flowchart of a method for training a speech recognition system;

[0022] Figure 13 This represents the halting decision of different heads in the ASR system;

[0023] Figure 14A This indicates the ASR system's stop decision;

[0024] Figure 14B This indicates the stop decision of the ASR system according to the example embodiment;

[0025] Figure 15 This is a schematic diagram of the hardware used to implement the methods and systems according to the example embodiments. Detailed Implementation

[0026] According to a first aspect, a method for speech recognition is provided. The method includes: receiving a speech audio frame; encoding the received frame; determining a context vector from the encoding of the received frame; deriving an action from the context vector via an agent, wherein the agent includes a trained model; deriving a token from the context vector in response to the action satisfying the predetermined condition; and executing a function based on the token, wherein the function includes at least one of text output or command execution.

[0027] The disclosed method reduces the latency and computational cost of speech recognition. This reduction in latency and computational cost is achieved while maintaining the quality (e.g., accuracy) of speech recognition. In the method, speech audio frames are received and subsequently encoded. A context vector is determined from the encoding. The context vector is associated with acoustic information present in the received frames and with historical acoustic information. The context vector is computed on-the-fly; for example, it is determined before a step of checking if predetermined conditions are met. The context vector can also be referred to as an interim context vector.

[0028] Actions are derived from the context vector. Actions can correspond to variables indicating whether a token should be exported. A token is exported from the context vector when a predetermined condition is met. For example, actions can be exported from a delegate or a stop selector.

[0029] The function is executed based on the derived token. For example, the function may include outputting text or executing a command. The time step (or position) at which the action satisfies a predetermined condition can be understood as the stopping position. An improved stopping position can be obtained by considering when the action satisfies the predetermined condition. An improved stopping position means a smaller stopping position while maintaining the quality (e.g., accuracy) of the derived token. In particular, the stopping position may be smaller (or earlier) than methods that rely on arbitrary conditions rather than using actions derived from context vectors. Subsequent steps based on token-based function execution are also performed earlier. Therefore, the disclosed method can perform speech recognition with reduced latency (attributed to the earlier export of the token). Computational costs can also be reduced because less computation is required before exporting the token.

[0030] For example, an action is derived from a context vector by an agent. The agent comprises a trained model. The agent is trained so that it can determine from the context vector when a token should likely be derived. The agent outputs an action. The action is one of the possible actions the agent can take. The action may correspond to a variable indicating whether a token should be derived. A token is derived when a predetermined condition is met.

[0031] The agent can be trained to improve the stopping position. An earlier stopping position means deriving the token from the context vector earlier.

[0032] For example, an action can have one of two values. For example, an action can be a binary value. For example, an action can be "0" or "1", or "true" or "false", or "yes" or "no".

[0033] The action is compared with a predetermined condition. For example, the predetermined condition is met when the action equals "1".

[0034] In an embodiment, deriving an action via a proxy includes: inputting a context vector into a trained model; determining an action probability (also called a first probability) from the trained model; comparing the action probability with a predetermined threshold; and setting an action based on the comparison. The action probability (first probability) is determined based on the context vector. The action probability indicates the confidence level at which a token can be derived at that time step. For example, a high confidence level may mean that the derived token is likely accurate, thus maintaining the quality of speech recognition. When the action probability is greater than the predetermined threshold, the proxy outputs an action that satisfies a predetermined condition. For example, when the action probability is greater than or equal to 0.5, the action is set to a value that satisfies the predetermined condition.

[0035] In one embodiment, determining the action probability (first probability) from the trained model includes: adding the output of the trained model to a predetermined bias; and applying a sigmoid function to the sum to obtain the action probability. Adding the predetermined bias reduces the likelihood that the agent will set actions that satisfy predetermined conditions in the early stages of training. This helps the speech recognition method achieve sufficient accuracy more quickly during training.

[0036] In one embodiment, the agent comprises a deep neural network.

[0037] In other examples, the agent may include a trained model. A trained model may include recurrent neural networks (RNNs) or convolutional neural networks (CNNs), etc.

[0038] In one embodiment, determining the context vector includes using a context vector determined from previous speech audio frames. By calculating from previous frames, the context vector carries historical information.

[0039] In this embodiment, determining the context vector includes determining a second probability corresponding to the attention weights of the self-attention decoder layer. The second probability represents the confidence level of the derived context vector. In other words, the second probability represents the probability that the determined context vector is acceptable.

[0040] The second probability is the likelihood that an ASR output (token) should be derived. This is a result of the context vector being accepted.

[0041] In one embodiment, determining the context vector includes multiplying a second probability by an encoder state derived from received speech audio frames.

[0042] The received frames can be encoded by a self-attention encoder. The self-attention encoder includes self-attention encoder layers. The self-attention encoder layers can include a stack of self-attention encoder layers. The encoder state can be computed using the self-attention encoder layers of the encoder neural network.

[0043] In this embodiment, the second probability is calculated using the self-attention decoder layer of the decoder neural network.

[0044] The second probability can be derived from the attention energy. The attention energy is calculated using the self-attention decoder layer of the decoder neural network.

[0045] In an embodiment, the self-attention decoder layer is a multi-head self-attention decoder layer comprising multiple attention heads, wherein the second probability is calculated using the attention heads among the multiple attention heads.

[0046] In an embodiment, determining the context vector includes determining a first context vector from each of the plurality of attention heads and concatenating the determined first context vectors to obtain the context vector.

[0047] The self-attention encoder layer can be a multi-head self-attention encoder layer. The number of attention heads corresponds to the number of heads in the multi-head self-attention decoder layer. The encoder state can be calculated using the attention heads among the multiple attention heads of the multi-head self-attention encoder layer.

[0048] In one embodiment, the method further includes adding the determined second probability to an accumulator variable, wherein a token is derived from the context vector in response to the accumulator variable satisfying a second condition; and then a function is executed based on the token, wherein the function includes at least one of text output or command execution. The second condition differs from the predetermined condition.

[0049] For example, the second condition includes a comparison of an accumulator variable with a second predetermined threshold. Satisfaction of the second condition provides an additional condition for triggering token derivation. For example, the second predetermined threshold could be the number of attention heads. In some cases, received speech audio frames may lead to actions that do not meet the predetermined conditions. In these cases, satisfaction of the second condition can instead trigger token derivation. Therefore, this ensures token derivation and command execution, potentially reducing latency in speech recognition methods.

[0050] The second condition can be used to assist the agent. However, the second condition may not guarantee that decoding execution will stop. The third condition (maximum lookahead order) can instead force a stop.

[0051] In an alternative embodiment, the action corresponds to a probability. This probability may be referred to as a third probability (or a stopping probability). The action is derived from the context vector by a module. This module may also be referred to as a stop selector.

[0052] The stop selector is configured to determine whether to trigger the ASR output at each time step. The stop selector is trained so that it can determine from the context vector when a token should likely be derived from the context vector. The output of the stop selector corresponds to the stopping probability.

[0053] For example, the third probability represents the likelihood that the i-th decoding will stop at some encoder time step j.

[0054] The stop selector can include a trained model. For example, a trained model includes a deep neural network (DNN).

[0055] According to a second aspect, a computer-implemented method for training a speech recognition system is provided, the method comprising, for training data including speech audio frames and training tokens,

[0056] Encode the frame;

[0057] The context vector is determined from the encoding of the frame;

[0058] Action probabilities are derived from the context vector using an agent that includes a trainable model;

[0059] In response to the action probability satisfying a predetermined condition, a predicted token is derived from the context vector;

[0060] The correctness of a prediction is determined by comparing the predicted token with the training token, where the correctness of a prediction indicates that the prediction is correct when the predicted token corresponds to the training token.

[0061] The stop position is obtained based on the predicted token, wherein the stop position represents the time step in which the predetermined condition is met;

[0062] The reward is determined based on the accuracy of the prediction and the stopping position obtained; and

[0063] The agent's weight is updated based on the determined reward.

[0064] The disclosed method for training speech recognition relies on the behavior of the speech recognition system. It does not rely on supervised learning; instead, it relies on reinforcement learning. A trainable agent is trained based on prediction correctness and rewards for obtaining stopping positions, enabling the trainable agent to learn improved stopping positions while maintaining prediction correctness (prediction accuracy).

[0065] In this embodiment, determining the reward includes:

[0066] Obtain the first prediction accuracy and the first stopping position from the first training epoch;

[0067] The second prediction accuracy and the second stopping position are obtained from the second period, where the first period is before the second period;

[0068] Compare the accuracy of the first prediction with the accuracy of the second prediction, and compare the first stopping position with the second stopping position.

[0069] In this embodiment, the determined reward has a first value under the following conditions:

[0070] The second stop position is greater than the first stop position;

[0071] The second stopping position is equal to the first stopping position, and the accuracy of the first prediction is different from the accuracy of the second prediction; and

[0072] The second stop position is smaller than the first stop position, and the second prediction correctness indicates that the prediction is correct;

[0073] The determined reward has a second value in the following situations:

[0074] The second stopping position is equal to the first stopping position, and the first prediction accuracy is equal to the second prediction accuracy; and

[0075] The second stopping position is smaller than the first stopping position, and the first prediction accuracy and the second prediction accuracy are equal but lower; and

[0076] When the second stop position is less than the first stop position and the second prediction accuracy is less than the first prediction accuracy, the determined reward has a third value.

[0077] The first value is greater than the second value, and the second value is greater than the third value.

[0078] Low prediction accuracy indicates that the prediction is incorrect. For example, low prediction accuracy means that the prediction accuracy is lower than the expected value.

[0079] In this embodiment, the first value is 0, and the second and third values ​​are negative.

[0080] In the example, determining the context vector includes:

[0081] Determine the second probability, where the second probability corresponds to the attention weight of the self-attention decoder layer.

[0082] In one embodiment, the method includes adding the determined second probability to an accumulator variable, wherein the predetermined condition includes comparing the accumulator variable with a predetermined threshold.

[0083] In an embodiment, the predetermined condition is satisfied when the accumulator variable is less than a predetermined threshold.

[0084] According to another aspect, a system for speech recognition is provided, the system including a processor configured to:

[0085] Receive voice audio frames;

[0086] Encode the received frames;

[0087] The context vector is determined from the encoding of the received frame;

[0088] Derive the action from the context vector;

[0089] In response to the action satisfying a predetermined condition, a token is derived from the context vector; and

[0090] The token is used to execute functions, wherein the functions include at least one of text output or command execution.

[0091] According to another example, a computer-implemented method for speech recognition is provided, the method comprising: receiving a speech audio frame; encoding the received frame; determining a context vector from the encoding of the received frame; deriving a third probability from the context vector; deriving a token from the context vector in response to the third probability satisfying a predetermined condition; and performing a function based on the token, wherein the function includes at least one of text output or command execution.

[0092] The method improves the accuracy of speech recognition and reduces its latency. In this method, speech audio frames are received and subsequently encoded. A context vector is determined from the encoding. The context vector is a vector associated with acoustic information present in the received frames and with historical acoustic information. The context vector is computed on the fly; for example, it is determined before a step checking if predetermined conditions are met. The context vector can also be referred to as a provisional context vector.

[0093] The third probability, also known as the stopping probability, represents the likelihood of deriving a token from the context vector of the received speech audio frame. The third probability differs from the second probability described above. A token is derived from the context vector when the stopping probability meets a predetermined condition.

[0094] For example, the third probability represents the likelihood that the i-th decoding will stop at some encoder time step j.

[0095] The stopping probability is derived from the context vector through a stopping selector. The stopping selector comprises the trained model. The stopping selector is configured to provide stopping probabilities. The stopping selector takes the context vector as input. The stopping selector enables the determination of robust stopping positions.

[0096] The received frames can be encoded by a self-attention encoder. A self-attention encoder consists of self-attention encoder layers. A self-attention encoder layer can include a stack of self-attention encoder layers. The encoder state can be computed by the self-attention encoder layers of the encoder neural network. The encoder state is a vector.

[0097] For example, the context vector is derived using the encoder state.

[0098] For example, the context vector is computed using the self-attention decoder layer of the decoder neural network.

[0099] A self-attention decoder layer can be a multi-head self-attention decoder layer comprising multiple attention heads. The context vector is derived from one of the multiple attention heads.

[0100] A self-attention decoder layer can consist of a stack of self-attention decoder layers. The number of self-attention decoder layers can correspond to the number of self-attention encoder layers.

[0101] In the example, one of the self-attention decoder layers in the stack of self-attention decoder layers includes a stop selector. This decoder layer could be, for example, the last decoder layer.

[0102] According to another example, a computer-implemented method for training a speech recognition system is provided, the method comprising, for training data including speech audio frames and training tokens,

[0103] Encode the frame;

[0104] Determine the expected stopping probability;

[0105] Export the expected context vector;

[0106] Derive the predicted token from the expected context vector;

[0107] Compare the predicted tokens with the training tokens;

[0108] The weights of the speech recognition system are updated based on the comparison.

[0109] The expected stopping probability is obtained by considering all possible stopping positions (encoder time steps) and their stopping probabilities (third probabilities).

[0110] The expected context vector is derived from the expected stopping probability and the context vector derived from the encoding of the frame.

[0111] The comparison between the predicted token and the training token involves determining the difference between the predicted token and the training token according to the loss function.

[0112] These methods are computer-implemented methods. Since some methods according to the embodiments can be implemented in software, some embodiments include computer code provided to a general-purpose computer on any suitable carrier medium. The carrier medium may include any storage medium, such as a floppy disk, CD-ROM, magnetic device, or programmable memory device, or any transient medium, such as any signal, such as an electrical signal, optical signal, or microwave signal. The carrier medium may include a non-transitory computer-readable storage medium. According to another aspect, a carrier medium is provided that includes computer-readable code configured to cause a computer to perform any of the methods described above.

[0113] For illustrative purposes, regarding Figure 1A-1D The examples illustrate the possible contexts in which thematic innovations can be applied. However, it should be understood that these are exemplary and that these thematic innovations can be applied in any suitable context, such as any context in which speech recognition is applicable.

[0114] Voice assistant system

[0115] Figure 1A This is an example of a voice assistant system 120 according to an exemplary embodiment.

[0116] User 110 can speak commands 112, 114, and 116 to the voice assistant system 120. In response to user 110 speaking commands 112, 114, and 116, the voice assistant system executes the commands, which may include outputting an audible response.

[0117] To receive spoken commands 112, 114, and 116, the voice assistant system 120 includes or is connected to a microphone. To output an audible response, the voice assistant system 120 includes or is connected to a speaker. The voice assistant system 120 may include functionality, such as software and / or hardware, suitable for recognizing spoken commands, executing commands, or causing commands to be executed, and / or causing appropriate audible responses to be output. Alternatively or additionally, the voice assistant system 120 may be connected via a network, such as via the Internet and / or a local area network, to one or more other systems suitable for recognizing spoken commands and causing commands to be executed, such as cloud computing systems and / or local servers. A first portion of this functionality may be executed by the hardware and / or software of the voice assistant system 120, while a second portion of this functionality may be executed by the one or more other systems. In some examples, the functionality, or a large portion thereof, may be provided by the one or more other systems that are accessible via a network, but when, for example, the voice assistant system 120 is disconnected from the network and / or the one or more other systems are unavailable via the network due to a malfunction, the functionality may be provided by the voice assistant system 120. In these examples, the voice assistant system 120 may be able to utilize the greater computing resources and data availability of the one or more other systems, for example, to execute a wider range of commands to improve the quality of speech recognition and / or the quality of audible output, while still being able to operate without connection to the one or more other systems.

[0118] For example, in command 112, user 110 asks, “What is X?”. This command 112 can be interpreted by the voice assistant system 120 as a spoken command providing a definition of term X. In response to this command, the voice assistant system 120 can query a knowledge source, such as a local database, a remote database, or another type of local or remote index, to obtain the definition of term X. Term X can be any term for which a definition is available. For example, term X can be a dictionary term, such as a noun, verb, or adjective; or an entity name, such as the name of a person or business. When the definition is obtained from the knowledge source, it can be synthesized into a sentence, such as a sentence of the form “X is [definition]”. The sentence can then be converted into audible output 112, for example, using the text-to-speech function of the voice assistant system 120, and output using a speaker included in or connected to the voice assistant system 120.

[0119] As another example, in command 114, user 110 says "turn off the lights." Command 114 can be interpreted by the voice assistant system as a spoken command to turn off one or more lights. Command 114 can be interpreted by the voice assistant system 120 in a context-sensitive manner. For example, the voice assistant system 120 may know the room it is in and specifically turn off the lights in that room. In response to the command, the voice assistant system 120 may turn off one or more lights, for example, by turning off one or more smart bulbs. The voice assistant system 120 may turn off one or more lights directly, for example, via a wireless connection (such as Bluetooth) between the voice assistant system and the lights; or indirectly by interacting with the lights, for example, by sending one or more "turn off the lights" messages to a smart home hub or cloud smart home control server. The voice assistant system 120 can also generate an audible response 124, such as saying "lights off" to confirm to the user that the voice assistant system 120 has heard and understood the command.

[0120] As another example, in command 116, user 110 says "play music". Command 116 can be interpreted by the voice assistant system as a spoken command to play music. In response to this command, voice assistant system 120 can: access a music source, such as a local music file or a music streaming service, stream music from the music source, and output the streamed music 126 from a speaker included in or connected to voice assistant system 120. The music 126 output by voice assistant system 120 can be personalized for user 110. For example, voice assistant system 120 can identify user 110, for example, by the nature of user 110's voice, or it can statically associate with user 110 and then restore music previously played by user 110 or play a playlist personalized for user 110.

[0121] Speech transcription system

[0122] Figure 1B This is an example of a speech transcription system according to an exemplary embodiment.

[0123] User 130 can speak to computer 140. In response to the user's speech, computer 140 produces text output 142 representing the content of speech 132.

[0124] To receive speech, computer 140 includes or is connected to a microphone. Computer 140 may include software suitable for recognizing the content of the speech audio and outputting text representing the content of the speech, such as transcribing the speech content. Alternatively or additionally, computer 140 may be connected via a network, such as via the Internet and / or a local area network, to one or more other systems suitable for recognizing the content of the speech audio and outputting text representing the content of the speech. A first portion of this functionality may be performed by the hardware and / or software of computer 140, while a second portion of this functionality may be performed by the one or more other systems. In some examples, the functionality, or a large portion thereof, may be provided by the one or more other systems, which may be accessible via a network, but may be provided by computer 140 when, for example, the one or more other systems are disconnected from the network and / or malfunction of the one or more other systems, they are inaccessible via the network. In these examples, computer 140 may be able to utilize the greater computing resources and data availability of the one or more other systems, for example, to improve the quality of speech transcription, while still being able to operate without being connected to the one or more other systems.

[0125] The output text 142 can be displayed on a monitor included in or connected to computer 140. The output text can be entered into one or more computer programs running on computer 140, such as word processing programs or web browser programs.

[0126] Voice Assistance Methods

[0127] Figure 1C This is a flowchart of a method 150 for voice assistance according to an example embodiment. Optional steps are indicated by dashed lines. Example method 150 can be implemented by one or more computing devices, for example, regarding Figure 7 The hardware 700 described executes one or more computer-executable instructions. The one or more computing devices may be or include a voice assistant system, such as voice assistant system 120, and / or may be integrated into a multi-purpose computing device such as a desktop computer, laptop computer, smartphone, smart TV, or game console.

[0128] In step 152, a microphone, such as the microphone of a voice assistant system or a microphone integrated into or connected to a multipurpose computing device, is used to receive voice audio. When voice audio is received, it can be cached in memory, such as the memory of the voice assistant system or the multipurpose computing device.

[0129] In step 154, the content of the speech audio is identified. The content of the speech audio can be determined using the methods described herein, for example... Figure 2 Method 200 Figure 3A Method 300A or Figure 3B Method 300B is used for identification. Before using this method, for example, [the following methods can be used]. Figure 6 Method 600 or its various aspects are used to preprocess audio. The content of the identified speech audio can be text, grammatical content, and / or semantic content. The identified content can be represented using one or more vectors. Alternatively, for example, after further processing, or alternatively, the identified content can be represented using one or more tokens. In the case that the identified content is text, each token and / or vector can represent a character, phoneme, morpheme or other morphological unit, word portion, or word.

[0130] In step 156, a command is executed based on the content of the speech audio. The command executed may be, but is not limited to, commands related to... Figure 1A Any of the commands 112, 114, and 116 described herein, and can be executed in the described manner. The command to be executed can be determined by matching the identified content against one or more command phrases or command patterns. The matching can be approximate. For example, for the command 114 to turn off the light, the command can be matched with phrases containing the words "light" and "off," such as "turn off the light" or "lights off." Command 114 can also be matched with phrases that semantically approximate "turn off the light," such as "turn off the light" or "lights on."

[0131] In step 158, for example, using a speaker included in or connected to a voice assistant system or multipurpose computing device, an audible response is output based on the voice audio content. The audible response may be about... Figure 1A The audible response may be any one of 122, 124, or 126, and may be generated in the same or similar manner as described. The audible response may be a spoken sentence, word, or phrase; music; or other sound, such as a sound effect or alarm. The audible response may be based on the content of the spoken audio itself and / or may be indirectly based on the content of the spoken audio, such as based on an executed command, which itself is based on the content of the spoken audio.

[0132] When the audible response is a spoken sentence, phrase, or word, the output audible response may include converting the text, vector, or token representation of the sentence, phrase, or word into spoken audio corresponding to the sentence, phrase, or word using a text-to-speech function. The representation of the sentence or phrase may be synthesized based on the content of the spoken audio and / or based on the executed command. For example, in the case where the command is a definition retrieval command of the form “What is X?”, the content of the spoken audio includes X, and the command causes the retrieval of the definition [def] from a knowledge source. A sentence of the form “X is [def]” is synthesized, where X comes from the content of the spoken audio, and [def] is the content retrieved from the knowledge source by the executed command.

[0133] As another example, in the case where the command is a command to make a smart device perform a certain function, such as a light-off command to turn off one or more smart bulbs, the audible response can be a sound effect indicating that the function has been performed or is being performed.

[0134] As shown by the dotted lines in the diagram, the step of generating an audible response is optional and may not occur for some commands and / or in some implementations. For example, in the case of a command that causes a smart device to perform a certain function, the function can be performed without outputting an audible response. An audible response may not be output because the user has other feedback indicating that the command has been successfully completed, such as a light being turned off.

[0135] Speech transcription methods

[0136] Figure 1D This is a flowchart of a method 160 for performing speech transcription according to an example embodiment. Example method 160 can be implemented by one or more computing devices, for example, regarding... Figure 7 The hardware 700 described executes one or more computer-executable instructions. The one or more computing devices may be computing devices such as desktop computers, laptop computers, smartphones, smart TVs, or game consoles.

[0137] In step 162, a microphone, such as one integrated into or connected to a computing device, is used to receive voice audio. When voice audio is received, it can be cached in memory, such as the memory of the computing device.

[0138] In step 164, the content of the speech audio is identified. The content of the speech audio can be identified using the methods described herein, for example... Figure 2 Method 200 Figure 3A Method 300A or Figure 3B Method 300B is used for identification. Before using this method, for example, [the following methods can be used]. Figure 6Method 600 or its various aspects are used to preprocess audio. The content of the identified speech audio can be text content, grammatical content, and / or semantic content. The identified content can be represented using one or more vectors. Alternatively, for example, after further processing, or alternatively, the identified content can be represented using one or more tokens. In the case that the identified content is text, each token and / or vector can represent a character, phoneme, morpheme or other morphological unit, word portion, or word.

[0139] In step 166, text is output based on the content of the speech audio. If the recognized speech audio content is text content, the output text can be that text content, or it can be derived from the recognized text content. For example, the text content can be represented using one or more tokens, and the output text can be derived by converting the tokens into the characters, phonemes, morphemes, or other morphological units, word parts, or words they represent. If the recognized speech audio content is or includes semantic content, output text with a meaning corresponding to the semantic content can be derived. If the recognized speech audio content is or includes grammatical content, output text with a structure corresponding to the grammatical content, such as a grammatical structure, can be derived.

[0140] The output text can be displayed. The output text can be input into one or more computer programs, such as a word processor or web browser. Further processing can be performed on the output text. For example, spelling and grammatical errors in the output text can be highlighted or corrected. In another example, a machine translation system can be used to translate the output text.

[0141] Figure 2 -Flowchart of the method

[0142] Figure 2 This is a flowchart of a speech recognition method according to an embodiment. In step 210, a speech audio frame is received. This speech audio frame is derived from an audio signal containing speech. For example, the speech audio contains the audio of a user's voice. The speech audio frame can be preprocessed before reception. How the speech audio frame is preprocessed and how it is obtained from the audio signal will be described later. The audio signal can be obtained from a sound capture device such as a microphone. The speech audio frame can be received as part of a plurality of speech audio frames. The speech audio frame can be referred to as a window of speech audio. For example, the window can have a duration of 25 ms. The plurality of speech audio frames can be referred to as a speech audio block.

[0143] In step 230, the speech audio frame is encoded. The encoded speech audio frame represents the speech audio frame. The speech audio frame can be encoded using an encoder neural network, such as a self-attention encoder neural network and / or other types of machine learning encoders, such as kernel method encoders like decision tree encoders or Gaussian process encoders. Alternatively or additionally, the speech audio frame can be encoded using a programmable encoder and / or encoding method. For example, an algorithm such as the Fast Fourier Transform algorithm can be used to encode the speech audio frame.

[0144] Specifically, a speech audio frame can be encoded as a part of multiple speech audio frames, such as as part of a speech audio block. Each frame in the multiple speech audio frames can be encoded simultaneously, for example, each frame in a speech audio block can be encoded simultaneously. The speech audio block can be encoded using a self-attention encoder as described below.

[0145] In step 240, the context vector c is determined (step 240-a), the action (a) is derived (240-b), and it is determined whether the obtained action satisfies the predetermined conditions (250).

[0146] In step 240-a, a context vector is determined based on the encoding of the speech audio frame in step 230. The determination of the context vector is explained in more detail below. Briefly, the context vector is based on the acoustic information of the encoded frame. It may also carry historical acoustic information (e.g., from previous speech audio frames). The context vector is also referred to as a temporary context vector. As described later, a temporary context vector can be obtained for each time step (frame) in an autoregressive manner (i.e., based on its past values).

[0147] In step 240-b, the action is derived from the context vector. For example, the action is derived using an agent. The agent consists of a trained neural network. The agent implementation is represented as π. θ The agent's strategy involves parameters θ, which are trained parameters. Parameters θ can also be referred to as weights. For example, a neural network might be implemented as a two-layer DNN. The agent takes a context vector as input. The agent's policy πθ maps the context vector to an action (a).

[0148] An agent can be understood as a module or algorithm that takes a certain action based on its interaction with its surrounding environment. In this case, the environment is the ASR system. The agent interacts with the ASR system. The agent takes a state as input. The state is the configuration in which the agent finds itself. For example, the state is represented by a context vector. The agent takes the context vector as input. An action is one of the possible actions that the agent can take. There can be a finite number of possible actions; for example, the agent can emit an action that is 1 or 0. Policy π θ It is a strategy implemented by the proxy to determine the next action based on the current state.

[0149] The agent's trained neural network can be implemented as a two-layer DNN. For example, a DNN consists of two fully connected layers, where the first layer has input and output dimensions of {D, D}, and the second layer has input and output dimensions of {D, 1}, where D is the dimension of the context vector. Alternatively, the agent comprises a neural network consisting of two layers of a recurrent neural network (RNN) with the same dimensions as the DNN.

[0150] In step 250, it is determined whether the obtained action meets a predetermined condition. If the condition is met, the method proceeds to step 260. When the condition is met, a decision to stop is made. The decision to stop refers to stopping step 240.

[0151] As described below, step 240 can be performed at the decoder end of the transformer ASR. For example, step 240 can be performed by the decoder layer of the decoder neural network. Step 240 can be referred to as the decoding step.

[0152] In step 260, a token is derived. If the condition is not met, the method returns to step 240-a, where a temporary context vector and action probability are determined. Step 240 is then repeated for subsequent speech audio frames.

[0153] Optionally, satisfying a predetermined condition includes determining whether the action is equal to a predetermined value. For example, if the probability of the action is equal to "1", then the condition is considered satisfied.

[0154] In step 260, a token is derived. In the speech recognition method, when speech audio is recognized as text, the token can represent a character, phoneme, morpheme or other morphological unit, word portion or word.

[0155] In step 270, a function is executed based on the exported token. The executed function includes at least one of command execution and / or text output.

[0156] Additionally and optionally, as described with respect to the following figures, step 240 may include computing more context vectors and computing combined context vectors. Step 240 may also include processing the context vectors or combined context vectors using one or more linear layers. Step 240 may also include processing the context vectors, combined context vectors, or the outputs of the one or more linear layers using a classification layer. The output of the classification layer may be a plurality of probabilities or scores, such as a vector of probabilities or scores, where each probability or score indicates the likelihood of a given token being correct. The classification layer may be a cross-entropy layer, a softmax layer, or a variant or equivalent thereof. The derived token may be the token with the highest corresponding probability or score, or it may be selected probabilistically based on the probability or score, for example, a token with a higher probability or score is more likely to be selected. The probabilities or scores may be used with a joint decoder, which also utilizes a language model that outputs the probability or score of each token based on the previously output tokens. Such a joint decoder may compute joint probabilities or scores based on the language model probabilities or scores and the classification layer probabilities or scores. Tokens can be derived as the tokens with the highest combination probability or score, and can be selected probabilistically based on combination probability or score; for example, tokens with higher combination probability or score are more likely to be selected.

[0157] Inference using DRL-HS-DACS – Variant 1

[0158] Figure 3A This is a schematic diagram of method 300A according to an embodiment. Method 300A can be used to implement, for example, regarding... Figure 2 Step 240 is explained.

[0159] Method 300A involves a decoder layer of a decoder neural network. The decoder layer may be referred to as a self-attention decoder layer. The decoder neural network may be referred to as a self-attention decoder. The decoder layer and the decoder neural network will be further described below. The decoder neural network may include one or more decoder layers (l).

[0160] Method 300A includes an initialization step 330A. In step 330A, the context vector is initialized. The context vector is represented as c. l,h i,j Here, h represents the attention head number. “i” represents the decoder step, and “j” represents the encoder time step (and corresponds to the speech audio frame under consideration). The above statements will be explained in more detail below.

[0161] The terms "context vector," "attention head," "decoder step," and "encoder time step" are related to the converter ASR, which has been streamlined through a decoder-side adaptive computation step (DACS). The converter ASR is further streamlined to include combining attention weights across individual heads (when more than one head is included) and triggering the ASR to output for all heads at the same time step. This is called head-synchronized decoder-side adaptive computation step (HS-DACS).

[0162] Method 300A can be referred to as Deep Reinforcement Learning Head-Synchronous Decoder-End Adaptive Computation Step (DRL-HS-DACS). Here, DRL refers to the use of the surrogate described in this paper. HS refers to combining attention weights across the individual heads (when more than one head is involved) and triggering ASR output for all heads at the same time step. DACS refers to the accumulation of attention during the decoding step. The DRL-HS-DACS algorithm belongs to the monotonic attention family, which allows attention weights to be computed strictly from left to right immediately from each encoder state. As will be explained below, the DRL-HS-DACS sublayer in the self-attention decoder layer can implement Method 300A.

[0163] In method 300A, each decoder layer (l) is taken separately. In other words, the steps of method 300A are performed for each decoder layer (l) of the decoder neural network.

[0164] In step 330A, the context vector is initialized. For example, the context vector is initialized to 0. l,h i,0 =0.

[0165] For each audio frame (j) until the condition is met, perform the following steps.

[0166] In step 340A, the probability is determined. The probability is represented as p. h,l i,j To obtain this probability, the attention energy is first calculated. For example, taking the decoding step i of the l-th decoder layer, for a certain head h, the attention energy e at time step j is... l,h i,j Calculated as:

[0167]

[0168] In Equation 1, q and k are the decoder state and encoder state, respectively. Attention energy e l,h i,j It is then passed to the sigmoid unit to generate probabilities:

[0169]

[0170] probability (p) l,h i,j Physically, it represents the head. h The confidence level at which the current decoding process ends is obtained at time step j. The probability of Equation (2) can be called the second probability. The probability of Equation (2) is calculated using the attention head. The probability of Equation (2) can also be called the attention weight or the stopping probability. The second probability is different from the stopping probability described with respect to Equation (11). The stopping probability of Equation (11) can be called the third probability.

[0171] In step 350A, the temporary context vector (c) is calculated. l,h i,j For example, for each time step, the temporary context vector is suggested in an autoregressive manner as follows:

[0172]

[0173] In equation (3), c l,h i,j-1 v represents the temporary context vector from the previous encoder step size j-1. l,h j This represents the encoder state at time step j. When j = 1, term c l,h i,j-1 Deprecated (e.g., set to 0).

[0174] Note that the temporary context vector is computed on the fly and carries all acoustic information from the past. The context vector is computed first, and then the context vector leads to a stopping decision (which will be explained below).

[0175] For each attention head h, proceed with steps 340A and 350A.

[0176] After considering all self-attention points h, proceed to step 360A.

[0177] In step 360A, the temporary context vectors obtained for each attention head h are concatenated (c l i,j It's important to understand that in an arrangement with only one attention head (h=1), step 360A is optional. The context vector from step 350A can then be used instead of the linked vector.

[0178] In step 370A, the action probability (p) is derived. l a i,j The linked context vector is immediately sent to the agent model where the action probabilities are derived. Note that the temporary context vector is computed up to time step j. Action probability (p la i,j The probability (p) is calculated by the proxy model. l a i,j This can be referred to as the first probability. As will be discussed later, the action probability (p) l a i,j This is used to derive actions that indicate a stop action.

[0179] The following obtains the action probability (p) l a i,j ):

[0180]

[0181] Proxy Implementation Strategy π θ In equation (4), π l θ Let represent the policy of the agent in the l-th decoder layer. Given the parameters θ of the agent model, this policy maps the context vector c to the probability p of the action. l a i,j .

[0182] The bias term 'b' added to the DNN's output prevents the agent from stalling during the early stages of training, allowing the ASR module to access the complete speech and reach a sufficient level of accuracy as quickly as possible. For example, the bias term 'b' can be any negative value. In a non-restrictive example, 'b' is -4. Note that even though the action is sampled in a single time step, it is still subject to all previous frames.

[0183] In step 380A, from the probability p of the action l a i,j Calculate action a l i,j For example, action a l i,j It has a binary value. For example, action a l i,j It has a value of 0 or 1. The action can be obtained as follows. If the policy provides a probability less than 0.5, the 'Continue' action is selected (a = 0); otherwise, the 'Stop' action is performed on the decoding (a = 1). This action can be obtained as follows:

[0184]

[0185] Steps 370A and 380A are performed by the agent.

[0186] In step 390A, check action a l i,j Does the condition (first condition) meet? For example, the condition is: "Action value = 1 (a l i,j =1)?" If al i,j If the condition is 1, then a stop action is performed, and method 300A ends for each decoder layer l. If this condition is not met, method 300A is repeated for subsequent time steps (i.e., subsequent values ​​of j).

[0187] Inference using DRL-HS-DACS – Variant 1 (with accumulator)

[0188] Figure 3B This is a schematic diagram of method 300B according to another embodiment. Method 300B can be used to implement, for example, regarding... Figure 2 Step 240 is explained.

[0189] Method 300B is similar to Figure 3A Method 300A. Method 300B also involves a decoder layer. Method 300B can also be referred to as DRL-HS-DACS.

[0190] For each decoder layer l, the following steps are performed.

[0191] Perform initialization step 330B. In addition, initialize the accumulator variable acc. l i Except for step 330B, which is similar to step 330A. For example, the accumulator variable is initialized to 0.

[0192] For each audio frame (j) until the condition is met, perform the following steps.

[0193] Perform step 340B to obtain the second probability. In step 350B, calculate the temporary context vector (c l,h i,j Steps 340B and 350B are related to... Figure 3A Steps 340A and 350A are the same. Additionally, in method 300B, an additional step 345B is performed to add the second probability to the accumulator variable. Step 345B is executed for each self-attention head. Step 345B can be executed concurrently with step 350B.

[0194] Steps 340B, 345B, and 350B are performed for each attention head h. After considering all self-attention heads h, step 360B is performed.

[0195] Then proceed to step 360B. Step 360B is the same as step 360A.

[0196] Then proceed to step 370B. Step 370B is the same as step 370A.

[0197] Then proceed to step 380B. Step 380B is the same as step 380A.

[0198] Then proceed to step 390B. Step 390B is similar to step 390A, except that it checks action a. l i,j In addition to checking whether the condition regarding step 390A is met (which may be referred to as the first condition), it is also checked whether the accumulator variable meets a certain condition (which may be referred to as the second condition). For example, it is checked whether the accumulator variable exceeds a predetermined threshold (the second threshold). In this example, the predetermined threshold is the number of self-attention heads (H).

[0199] When action a l i,j Method 300B terminates when the condition (first condition) is met and / or the accumulator variable meets a certain condition (second condition).

[0200] The second condition has the following effect: it normalizes the scale of attention weights used in context vector computation by preventing the model from overfitting due to the unrestricted size of the context vector.

[0201] The purpose of the second condition is to serve as a backup criterion for determining the stopping context vectors 240 and 250 and for exporting the token. The agent (as described with respect to steps 370B-380B) can issue a stop action triggering method 300B according to the first condition. The agent can issue this action, and the first condition can be satisfied earlier than the accumulator variable condition. In other words, the first condition can be satisfied before the second condition is satisfied.

[0202] In some cases, the second condition is satisfied before the first condition is satisfied. Therefore, having the second condition allows step 240 to stop earlier and the token to be exported earlier (step 260). Thus, latency can be reduced.

[0203] DRL-HS-DACS Workflow

[0204] Figure 4 This is a schematic diagram of component 400 of the speech recognition method. Figure 4 Examples are shown of the possible values ​​that some components (states or variables) described in methods 300A and 300B may have. Specifically, Figure 4 The diagram illustrates snapshots of the components at different encoder time steps (j). Figure 4 It involves a single attention head. Figure 4 This illustrates the evolution of the method as the encoder time step increases from j=1 to j=6 (from left to right in the figure).

[0205] The encoder 430 states include e1, e2, e3, e4, e5, and e6.j This represents the encoder state e at encoder time step j. The encoder state is generated by the encoder layer, for example, regarding... Figure 2 As described in step 230. The encoder state can correspond to the encoder state v described with respect to Equation 3 and S350B. l j .

[0206] For each time step, a stopping probability 440 is calculated. The stopping probability is obtained as described with respect to step 340B. The calculated stopping probabilities 440 are accumulated in accumulator variable 445. At each time step j, the stopping probability for time step j is added to the accumulator variable. As j increases, the value stored in the accumulator variable also increases. For example, as... Figure 4 As shown, the calculated stopping probabilities from j=1 to j=6 are 0.1, 0.2, 0.3, 0.2, 0.2, and 0.1, respectively, and the accumulator values ​​from j=1 to j=5 are 0.1, 0.3, 0.6, 0.8, and 1.0.

[0207] The temporary context vector 450 is calculated from the encoder state. The context vector 450 is calculated using equation (3). For example, using probability 440 and encoder state 430: at j=1, c1=0.1×e1; at j=2, c2=c1+0.2×e2; at j=3, c3=c2+0.3×e3; at j=4, c4=c3+0.2×e4.

[0208] Temporary context vector 450 is fed into agent 470. Agent 470 corresponds to the agent specified with respect to 240, 370B, and / or 380B.

[0209] For example, as described with respect to step 380B, the derived action 480 for each time step is obtained. The derived actions from j=1 to j=4 can be 0, 0, 0, and 1.

[0210] exist Figure 4 In the example, at time step j=4, the action value a is obtained. l i,j =1. Conversely, at time step j=5, the accumulator value acc is obtained. l i =H=1. In this example, there is only one attention head, so H=1. Therefore, referring to the condition described in step 390B or 390A, the first condition is satisfied earlier (j=4) than the second condition (which is only satisfied when j=5). Therefore, the first condition in step 390A or 390B allows the computation to be stopped earlier. The token can then be derived earlier. The computation of the temporary context vector can stop at j=4. Therefore, the first condition in 390A or 390B allows for a reduction in the number of computation steps.

[0211] In other words, in this example, the calculation stops at time step 4 (j=4) when agent 470 takes action 480'1'. The accumulated stopping probability 445 up to the stopping position (j=4) is 0.8, which is below the threshold of 1 reached at time step 5. This results in a reduction in the number of calculation steps.

[0212] The speech recognition method described above can be represented by the following pseudocode:

[0213]

[0214] In the pseudocode (Algorithm 1), <sos>It is a token indicating the beginning of a sentence. <eos>This is a token indicating the end of a sentence. The other variables and parameters shown in Algorithm 1 correspond to those described with respect to methods 300A and 300B. In lines 4 and 5, initialization step 330B is performed. In lines 8-10, for each attention head (for h = 1 to H), the stopping probability is determined (step 340B), the accumulator is updated (step 345B), and the temporary context vector is computed (step 350B). In line 12, the temporary context vector is concatenated (step 360B). In lines 13 and 14, for each encoder time step (j), the action probability (step 370B) and action (step 380B) are derived. Line 14 of the algorithm corresponds to Equation 5. In line 16, "break" indicates the stopping action derived from the stopping context vector. In line 19, the context vector c is obtained. l i The context vector for decoder layer l is set to the temporary context vector computed so far. In line 20, t i This represents the time step in which the algorithm stops. In other words, t i Provides an indication of when the stopping decision was made (i.e., the stopping position). In line 20(t) i =max(t) i ,j)),t i Set as t i The larger of the current value (e.g., from a different decoder layer) or j (which is the stopping position of the current decoder layer). In line 22, for each decoder step (i), the token y is derived. i The token is from the last decoder layer l=N. d context vector c Nd i Exported. In line 23, increment the decoder step variable (i). Reference Figure 5 In system 500, the OutputLayer(.) in line 22 can correspond to the linear and classification layers 528, the joint online decoder 530, and / or the language model 540.

[0215] In Algorithm 1, there are three measures that can stop the decoding step. These measures include the first and second conditions described above. Another condition (condition 3 or the third condition) is shown in line 6 of Algorithm 1. The third condition is that the encoder time step (j) reaches an upper limit, which is t. i-1 +M or the smaller of T. T represents the length of the speech audio block. M represents the maximum look-ahead order. M represents an arbitrary upper limit used to constrain the calculation. As mentioned earlier, the first condition can provide an earlier stopping position. The second and third conditions are provided as backup criteria for obtaining the stopping position.

[0216] Figure 5 Architecture

[0217] Figure 5 This is a schematic diagram of a system for performing speech recognition according to an example embodiment. System 500 can be implemented using one or more computing devices, such as the hardware described below, and one or more computer-executable instructions. System 500 may also be referred to as an ASR system, an ASR module, or a speech recognition system or speech recognition module.

[0218] System 500 can use one or more computing devices, such as those related to computing. Figure 15 This is implemented using one or more computer-executable instructions on the hardware 900 described.

[0219] System 500 is based on a converter architecture.

[0220] The transducer architecture is an encoder-decoder framework consisting of stacked layers at both ends based on a pure attention mechanism. In a transducer-based ASR system, a convolutional neural network (CNN)-based front end is used to enhance feature extraction and subsampling. Each encoder layer comprises a cascade of two sub-layers: a multi-head self-attention module and a pointwise feedforward network (FFN). Layer normalization is also applied to the layer outputs to facilitate model convergence. For the decoder layers, in addition to the sub-layers present on the encoder side, a third multi-head cross-attention module exists between them to produce speech-to-text alignment. In this example of an ASR system, the cross-attention sub-layers can reduce the likelihood of in-line decoding. This is reflected in the dot-product attention mechanism employed by the transducer:

[0221]

[0222] Where K and The same matrix represents the encoder states with T time steps. This is a vector of decoder states at a certain decoding step. It is also called the attention energy, the product of Q and K, scaled down by the square root of dk, where dk represents the dimension of the aforementioned states. The softmax function applies global normalization. The above requires the complete input sequence, which may not be suitable for online ASR systems. In System 500, different sublayers are used.

[0223] The system 500 according to an embodiment of the present invention is described below.

[0224] System 500 includes a self-attention encoder 510. The self-attention encoder 510 can be a block-by-block self-attention encoder. The self-attention encoder receives speech blocks. Each speech audio block comprises multiple audio frames. The self-attention encoder 510 encodes the multiple frames in the speech audio block to produce corresponding codes for the multiple speech audio frames. The encoding of the speech audio frames in the block can be substantially simultaneous. The number of speech audio frames in the block can be referred to as N. c A speech audio block can be composed of N speech audio segments. l The left frame (e.g., the frame received before the frame in the chunk) and N of the speech audio r A right frame (e.g., a frame received after a frame in a chunk) is used as a supplement. Left and right frames provide context for encoding the speech audio chunk, but the encoding of the speech audio frames within the chunk is used during decoding. The encoding latency can be limited to the number of right frames of the speech audio, for example, N. r The length of the audio represented by a frame. N c N l and N r They can be equal. In the implementation, N c N l and N r It can be 64.

[0225] The self-attention encoder 510 includes one or more convolutional neural network layers 512. The use of one or more convolutional neural network layers 512 can improve the extraction of acoustic features from speech audio frames in a block and can subsample frames within the speech audio block. In an implementation, the one or more convolutional neural network layers 512 can be two identical convolutional neural network layers of size 3×3 with 256 convolutional kernels and a stride of 2×2 that reduces the frame rate by a factor of 2. These one or more convolutional neural network layers produce outputs for subsequent layers.

[0226] The self-attention encoder 510 also includes a position encoding layer 514. The position encoding layer 514 uses sequence order information (e.g., information indicating the position of the CNN output corresponding to a given frame compared to other frame encoders) to enhance the output of one or more convolutional neural network (CNN) layers.

[0227] The self-attention encoder 510 also includes one or more self-attention encoder layers 516. Each self-attention encoder layer may have two sub-layers: a self-attention sub-layer and a position-forward sub-layer. The self-attention layer may be a multi-head attention layer. Each self-attention encoder layer may use residual connections and layer normalization after each sub-layer. The input of the first self-attention encoder layer in the one or more self-attention encoder layers 516 is the output of the position encoding layer 514. The input of subsequent layers in the one or more self-attention encoder layers is the output of the previous self-attention encoder layer. In an implementation, the one or more self-attention encoder layers 516 may be 12 self-attention encoder layers.

[0228] The encoding of a speech audio frame represents the speech audio frame. For example, each self-attention encoder layer 516 generates an encoding of the speech audio frame that contains information about the speech audio frame. For example, the encoding contains information about which parts of the input are related to each other.

[0229] Each self-attention encoder layer 516 can generate the encoding as follows: From the vectors input to the self-attention encoder layer, query (Q), key (K), and value (V) matrices are obtained. The Q, K, and V matrices are obtained by multiplying the input with the corresponding trained weight matrices. Then, the score (or attention energy) is obtained by multiplying Q and K. The score is normalized (e.g., by dividing by d). k The square root of d k (This represents the dimension of the aforementioned states). The normalized score is then multiplied by the V matrix. The result is the output of the self-attention encoder layer.

[0230] System 500 includes a self-attention decoder 520. The self-attention decoder 520 receives previous outputs, such as tokens from previous outputs, as input.

[0231] The self-attention decoder 520 includes one or more embedding layers 522. The one or more embedding layers 522 process the output token into an embedding of the output token, such as a vector embedding. In the implementation, this embedding has a dimension of 256, for example, a vector with 256 elements.

[0232] The self-attention decoder 520 includes a positional encoding layer 524. The positional encoding layer 524 enhances the embedding by utilizing sequence order information (e.g., information indicating the position of the token in the output sequence of tokens, such as a phrase or sentence).

[0233] The self-attention decoder 520 includes one or more self-attention decoder layers 526. Each self-attention decoder layer may have three sub-layers: a self-attention sub-layer; a Deep Reinforcement Learning, Head-Synchronization, Decoder Adaptive Computation Steps (DRL-HS-DACS) sub-layer; and a positional feedforward sub-layer. Each self-attention decoder layer may have three sub-layers, with the DRL-HS-DACS layer located within a cross-attention sub-layer. The DRL-HS-DAC sub-layer includes an agent and performs steps 240, 300A, or 300B as described herein. Each self-attention decoder layer may use residual connections and layer normalization after each sub-layer. The first self-attention decoder layer in the one or more self-attention decoder layers receives the enhanced embedding as input, i.e., the output of the positional encoding layer. Subsequent self-attention decoder layers receive the output of the previous self-attention decoder layer. Through the attention mechanism, the self-attention decoder layers also access the encoding of each frame generated by the self-attention encoder 510. In the implementation, one or more self-attention decoder layers 526 can be 6 self-attention decoder layers, the dimension of the self-attention sub-layer and DRL-HS-DACS sub-layer of each self-attention decoder layer can be 256, the position feedforward sub-layer of each self-attention decoder layer can have 2048 units, and each self-attention decoder layer can include 4 attention heads.

[0234] The self-attention decoder 520 includes linear and classification layers 528. The linear and classification layers process the output of the self-attention decoder layers to determine multiple probabilities or scores, such as a vector of probabilities or scores, each indicating the likelihood that a given token is correct. The classification layer can be a cross-entropy layer, a softmax layer, or variations or equivalents thereof.

[0235] System 500 includes a joint online decoder 530. The joint online decoder receives multiple probabilities or scores from the linear and classification layers 528 of the self-attention decoder 520. The joint decoder 530 also utilizes a language model 540 to output the probability or score of each token based on previously output tokens. Such a joint decoder can compute joint probabilities or scores based on both the language model probabilities or scores and the probabilities or scores received from the linear and classification layers of the self-attention decoder. The joint probabilities or scores can be weighted sums of the language model probabilities or scores and the received probabilities or scores, respectively. The joint decoder can also utilize Connection Temporal Classification (CTC) scores for decoding. The joint probabilities or scores can be further based on CTC scores, and each joint probability or score can be a weighted sum of the language model probabilities or scores, the received probabilities or scores, and the CTC scores. The joint score can be defined as λ. CTC s CTC +λ t s r +λ lm s lm , where s CTC It is the CTC score, s r It is the probability or fraction of acceptance, s lm It is the language model probability or score, λ CTC , λ r and λ lm These are weighting parameters. Each weighting parameter can be between 0 and 1. The weighting parameters can be summed to 1. These joint probabilities or scores can be used to derive tokens through any of the methods described with respect to step 260, such as selecting the token with the highest probability or score or selecting a token based on probability. A beam search can also be used to derive tokens.

[0236] System 500 includes a language model 540. The language model can be a deep language model implemented using neural networks. For example, a long short-term memory network can be used to implement the language model.

[0237] Obtain audio frames

[0238] Figure 6 This is a flowchart of a method 600 for preprocessing speech audio, such as processing speech audio before using the processed speech audio for speech recognition and / or training a speech recognition system 600. Optional steps are indicated by dashed lines. Example method 600 can be implemented by one or more computing devices, for example, regarding... Figure 7 The hardware 700 described executes one or more computer-executable instructions.

[0239] In step 610, the speech audio is divided into multiple frames. Each frame can be a window of speech audio. In the implementation, each frame can be a window of speech audio of 25ms with a 10ms offset.

[0240] In step 620, acoustic features are extracted for each speech audio frame. A filter bank can be used to extract the acoustic features. In this implementation, the acoustic features can be an 80-dimensional filter bank and 3-dimensional pitch-related information.

[0241] At 630, cepstral mean and variance normalization are performed on the acoustic features. Cepstral mean and variance normalization are optional. Cepstral mean and variance normalization can be applied to the acoustic features during training.

[0242] In step 640, N c The acoustic features of the frames are spliced ​​together to form speech audio chunks, where N c This refers to the chunk size. The chunk size can be defined as the number of frames of speech audio in each chunk.

[0243] In step 650, using N l Acoustic features of the previous frame and N r Acoustic features from subsequent frames supplement speech audio chunks.

[0244] train

[0245] Figure 7 This is a flowchart of method 700 for training a speech recognition system. The speech recognition system can be... Figure 5 The speech recognition system 500. As described above, system 500 includes agent 500b. Agent 500b corresponds to the agent described above with respect to 470, 380B, 370B, 380A, 370A, and 240. Agent 500b includes its own parameter θ.

[0246] The ASR system 500 and agent 500b are trained together. System 500 and agent 500b are trained using the same training data. The parameters of system 500 and agent 500b are updated together. However, the way agent 500b's parameters θ are updated differs from the way the ASR network parameters are updated. The ASR network parameters are updated via ASR training step 720, while the agent parameters θ are updated via policy gradient training step 730. The ASR system 500 and agent 500b are trained using different loss functions. For ease of understanding, ASR training 720 will be explained first, separate from policy gradient training 730. Policy gradient training 730 will be discussed below regarding... Figure 8 Please provide an explanation.

[0247] In step 720, the weights of the speech recognition system 500 are updated. System 500 includes multiple weights. The self-attention decoder layer of the speech recognition system 500 includes the first batch of multiple weights among many others.

[0248] The weight update described in step 720 involves the parameters of system 500, but is independent of the proxy parameter θ. The proxy parameter θ is updated using a different method 730, which is further described below.

[0249] The following steps are performed on each training pair in the training set. Each training pair 710 includes multiple speech audio frames and a training token sequence. The training sequence of tokens can also be described as a gold standard sequence or a reference sequence. The training sequence of tokens can be an indication of the speech content corresponding to the multiple speech audio frames or can be derived from the indication. For example, a token sequence representing words, phrases, or sentences.

[0250] A token sequence is derived from multiple speech audio frames. For each frame, the token can be derived using steps 230 to 260 of method 200, steps 330A to 390A of method 300A, or steps 330B to 390B of method 300B. These steps can be implemented by the speech recognition system 500.

[0251] When the token sequence includes multiple tokens, more tokens in the token sequence can be obtained by applying the above steps to the remaining frames in the multiple voice audio frames.

[0252] Exporting the token sequence and updating the weights of the speech recognition system involves the following steps. Two variations can be used, and each variation will be explained separately.

[0253] The first variant involves training the system used to implement method 300A.

[0254] First variant – Training a speech recognition network

[0255] i. Encode each frame as described with respect to step 230.

[0256] ii. For each self-attention decoder layer (l) of the decoder neural network, perform an initialization step. The initialization step corresponds to step 330A.

[0257] iii. For each frame, each self-attention head (h) is considered in turn, the stopping probability is determined as described with respect to step 340A, and the temporary context vector is calculated as described with respect to step 350A.

[0258] iv. Then connect the computed temporary context vector as described with respect to step 360A.

[0259] v. Then, as described regarding step 370A, derive the action probability p. l a i,j .

[0260] vi. Derive action a from the action probability as described with respect to step 380A. l i,j .

[0261] vii. Then check action a as described regarding step 390A. l i,j Check if the conditions are met. If the conditions are met, stop the action and complete the decoding steps of the self-attention decoder layer (l).

[0262] viii. Repeat steps ii. to vii. for each decoder layer until the final decoder layer (l = N) is reached. d Up to this point. For each decoder layer (l), obtain the context vector c. l i In the final decoder layer, the context vector c of the final decoder layer is obtained. Nd i .

[0263] ix. Then, as in step 260 and regarding Figure 5 The exported token. For example, as described in Algorithm 1 above, from the context vector c of the final decoder layer. Nd i Export token.

[0264] x. Then, for each decoder layer, the parameters are updated 720 times. The update is performed according to the loss function, based on the difference between the derived token sequence and the training sequence of the tokens. The first batch of multiple weights is updated according to the loss function, based on the difference between the derived token sequence and the training sequence of the tokens. Other weights among the more multiple weights can also be updated. For example, the weights of one or more other self-attention decoder layers or other layer types such as linear layers can also be updated. Other weights among the more multiple weights related to other parts of the speech recognition system can also be updated. For example, an encoder including a second batch of multiple weights from the more multiple weights can be used to generate the encoding for each speech audio frame. The second batch of multiple weights can also be updated according to the loss function based on the difference between the derived token sequence and the training sequence of the tokens. The weights can also be updated using backpropagation.

[0265] The loss function can be a cross-entropy loss function. The loss function can include a connection-time classification (CTC) component. The loss function can be a multi-objective loss function that includes both a cross-entropy component and a CTC component. The cross-entropy loss function of a multi-objective loss function can be used to update the first batch of multiple weights and other weights in the decoder. The cross-entropy loss function of a multi-objective loss function can be used to update the second batch of multiple weights in the encoder. The CTC loss function of a multi-objective loss function can be used to update the second batch of multiple weights in the encoder.

[0266] For ease of explanation, update step 720 is described as occurring for each training pair. However, it should be noted that in some variations, update step 720 may occur only for some training pairs, such as every Nth training pair. In these cases, the weight update is based on the difference between each of the N derived token sequences since the last update and the corresponding training sequence of the token, and can be computed according to a loss function. For example, the update may be based on the average of the losses computed for each of the derived token sequences and the corresponding training sequence of the token. Such methods for updating weights can be referred to as batch updates or mini-batch methods.

[0267] The second variant involves training the system used to implement method 300B.

[0268] Second variant – Training a speech recognition network

[0269] i. Encode each frame as described with respect to step 230.

[0270] ii. For each self-attention decoder layer (l) of the decoder neural network, an initialization step is performed. This initialization step corresponds to step 330B.

[0271] iii. For each frame, each self-attention head (h) is considered sequentially, and the stopping probability is determined as described with respect to step 340B, and the temporary context vector is calculated as described with respect to step 350B. The stopping probability is also accumulated in the accumulator variable as described with respect to step 345B.

[0272] iv. Then connect the computed temporary context vector as described with respect to step 360B.

[0273] v. Then, as described regarding step 370B, derive the action probability p. l a i,j .

[0274] vi. Derive action a from the action probability as described with respect to step 380B. l i,j .

[0275] vii. Then check action a as described regarding step 390B. l i,j Whether the conditions are met. The conditions may include a first condition and a second condition. If the conditions are met, a stop action is performed, and the decoding steps of the self-attention decoder layer (l) are completed.

[0276] The effect of using the second condition is to normalize the scale of the attention weights used in the context vector computation. For example, the effect of the second condition is to normalize the scale of the attention weights in the bottom decoder layer. This reduces the model's performance due to c. l i,j The size of the context vector is unrestricted, increasing the chance of overfitting. For example, the stopping probability used in context vector computation can have a value between 0 and 1. This value is accumulated in the accumulator variable. As the frame moves from left to right (i.e., as j increments), the accumulator variable can become very large. This can happen in the lower decoder layers (i.e., small values ​​of l). This is attributed to the modeling capabilities of the transformer model, which also occurs in non-streaming systems. In particular, the attention in the lower decoder layers may be ineffective; that is, they may not properly attend to the input frame. The stopping probability is multiplied by the encoder state, so these ineffective attentions may be added to the temporary context vector, causing decoding problems. By having a second condition, the computation of the context vector can be stopped, reducing the effect of ineffective attention. Furthermore, this can reduce the potential decrease in ASR accuracy. The effect of using the second condition will be related to... Figure 9 Further explanation.

[0277] viii. Repeat steps ii. to vii. for each decoder layer until the final decoder layer (l = N) is reached. d Up to this point. For each decoder layer (l), obtain the context vector c. l i In the final decoder layer, the context vector c of the final decoder layer is obtained. Nd i .

[0278] ix. Then, as in step 260 and regarding Figure 5 The exported token. For example, as described in Algorithm 1 above, from the context vector c of the final decoder layer. Nd i Export token.

[0279] x. Then, for each decoder layer, the parameters are updated 720. The update 720 is as described in step (x.) of the first variant.

[0280] Policy gradient training

[0281] Figure 8 This is a flowchart illustrating a method 800 for training a speech recognition system. Specifically, method 800 relates to training an agent 500b of the speech recognition system 500. Method 800 can be used in conjunction with... Figure 7 The first or second variant is used together.

[0282] In method 800, the training system 500 and agent 500b are sustained for at least two consecutive periods E-1 and E. Period E-1 may be referred to as the first period, and period E may be referred to as the second period. E-1 precedes E. Period E is the current period, and period E-1 is the previous period.

[0283] The number of periods is the number of times the learning algorithm passes through the entire training dataset. Within a period, the internal model parameters are updated for each sample in the training dataset. As mentioned above, in one arrangement, the model parameters are updated only after every N training pairs (microbatch method, batch size N). In this case, a period comprises one or more batches of N training pairs. N is the number of training pairs in a microbatch. N is also known as the microbatch size.

[0284] The ASR system 500 can be a decoder neural network that includes one or more decoder layers (l). Each decoder layer may include an agent.

[0285] Each microbatch simultaneously updates one of the agents in both the ASR system 500 and the decoder layer. Since there is more than one agent in the decoder, each agent is trained sequentially and continuously. Micro-batch; that is, continuous Train one agent in micro-batch, then train another agent. For example, if If the value is 3, then the entire system is trained as follows: {(ASR, Agent 0), (ASR, Agent 0), (ASR, Agent 0), (ASR, Agent 1), (ASR, Agent 1), (ASR, Agent 1), (ASR, Agent 1), (ASR, Agent 2), (ASR, Agent 2), (ASR, Agent 2), ...}. In other words, ASR system 500 and the first agent (Agent 0) are trained together for three consecutive microbatches; ASR system 500 and the second agent (Agent 1) are trained together for the next three consecutive microbatches; ASR system 500 and the third agent (Agent 2) are trained together for the next three consecutive microbatches. Each microbat trains the ASR network, but each... Each micro-batch transfers the agent during training.

[0286] D is the number of micro-batches updated together with the ASR module by the same agent. D is the number of tokens (decoding steps) in each micro-batch. D is a dynamic value that varies with each training pair in a given micro-batch, since each training pair may have a different number of tokens.

[0287] In it In the example, the micro-batch used and the agent in training can follow the following sequence:

[0288] {(micro-batch 1, proxy 1), (micro-batch 2, proxy 1), (micro-batch 3, proxy 1), (micro-batch 4, proxy 1)},

[0289] {(micro-batch 5, agent 2), (micro-batch 6, agent 2), (micro-batch 7, agent 2), (micro-batch 8, agent 2)},…

[0290] Because there are multiple agents, the agent trained together with the ASR module will be in each The training pairs are then switched.

[0291] Method 800 includes the following steps.

[0292] In step 810, training data pairs are received. Step 810 is, for example, the same as step 710.

[0293] In step 860, the token is exported. Step 860 corresponds to step 260. Note that the export of the token is also as per [the previous sentence, which is unclear due to missing context]. Figure 7 This is part of the training of the ASR system 500 as described. Therefore, the token derivation in step 860 can be related to, as per the description... Figure 7 The tokens described are derived in the same way. For example, one can utilize information about... Figure 7 The steps i. to ix. of the first or second variant described are used to derive the token.

[0294] In step 830, policy gradient training is performed. Step 830 corresponds to... Figure 7 Step 730. Step 830 includes the following steps.

[0295] In step 880, prediction correctness is obtained. Prediction correctness is obtained by comparing the derived token from step 860 with the corresponding training token from step 810. For example, if the derived token matches the training token, the prediction is correct; otherwise, the prediction is incorrect. Therefore, prediction correctness can have one of two values, such as correct or incorrect, true or false, or 1 or 0.

[0296] In step 870, the stop position K of the current period (second period) is obtained. E The stopping position K of the previous period (the first period) E-1 K E This is referred to as the second stopping position. K E-1 This is referred to as the first stopping position. A stopping position is determined for each decoder layer (l). Therefore, a set of positions using K is obtained in step 870. l E-1 The first stop position and a set of K are indicated. l E The second stop position is indicated. For convenience, the superscript "l" is not included below, but it should be understood that the stop position may be related to different layers. The stop position is determined for each decoding step (i). This makes it possible to compare the stop positions obtained in the first period and the second period for each decoder layer and for each decoder step, respectively.

[0297] In step 890, the reward is determined. A reward is determined for each decoder layer (l). A reward is determined for the agent within each decoder layer. A reward is determined for each decoder step (i). The reward depends on the prediction accuracy and the change in stopping position for that particular decoder step. The reward is referred to as R. The reward is the reward of the policy gradient algorithm. The reward is feedback used to measure the success or failure of the agent's action. The reasons behind using the reward and policy gradient algorithm approach will be further provided below. The purpose of the reward is to update the weights θ of the agent model, thereby improving the stopping position, i.e., reducing the stopping position. At each encoder time step (j), any action taken by the agent is acknowledged and either accepted by assigning a non-negative reward or rejected by assigning a negative reward. Through many such examples, the agent will learn to take appropriate actions.

[0298] Some rejected actions may be subject to more severe penalties, so further non-negative rewards with smaller values ​​(i.e., negative values ​​with larger numerical values) may also be used.

[0299] The reward reflects the agent's impact on the environment. For example, this impact can be measured using ASR accuracy and computational cost. ASR accuracy can be measured using prediction correctness as described above. Computational cost can be measured using stopping time as described above.

[0300] The reward R is designed as a function of the changes in ASR accuracy and computational cost observed as training progresses. The overall idea is that in each new data iteration, the policy is rewarded if accuracy is maintained at the appropriate stopping position, or penalized if accuracy decreases, regardless of the change in computational cost.

[0301] To ensure that fair rewards are distributed to agents, the following assumptions are made:

[0302] (1) Typically, the ASR module continuously improves its accuracy on the training set as the data is iterated;

[0303] (2) The later the agent stops the decoding step, the higher the ASR accuracy is likely to be. This is based on the fact that offline ASR systems are always superior to their online counterparts;

[0304] (3) If the agent identifies the same or later stopping position in the current data iteration compared to the previous data iteration, any decrease in ASR accuracy is attributed to the ASR module and the agent is not penalized.

[0305] (4) However, if the agent determines an earlier stop position and this results in a decrease in ASR accuracy, the agent is responsible and will be penalized for its actions.

[0306] In addition, to ensure training efficiency and avoid involving uncertain gradients, non-zero rewards will only be assigned to time steps that are directly related to the final 'stop' action of the decoding step, while other time steps will be considered neutral (assigned zero rewards).

[0307] Since the weights of ASR 500 are updated simultaneously with the parameters θ of the agent, potential variations in ASR accuracy could be attributed to both the training of ASR 500 and the training of agent 500b. This assumption helps to distinguish between the effects of the agent and the effects of ASR as training progresses.

[0308] The reward determined in step 890 is used to update the agent's weights according to the following equation:

[0309]

[0310]

[0311] Equation (6) represents the gradient ascent method, where the parameter θ is obtained by maximizing J(θ). α is the learning rate. In the example, α can be 0.010, 0.001, or 0.005.

[0312] Equation (7) represents an approximation of the gradient of the objective function J(θ). In equation (7), SG(·) is a function that prevents gradients of ASR components from entering the computation. SG(.) prevents the gradients of the ASR network from being included in the gradients of the surrogate network during training. For example, the SG(.) function is configured such that a in equation 7... K Remove context vector c in gradient calculation K The gradient.

[0313] a k This refers to the action performed at the current encoder time step K. K R is the context vector for the current encoder time step K. K It is the reward determined for the encoder time step K.

[0314] For each decoding step of each training pair, agent actions with the same encoder time step having index K are considered during policy gradient training.

[0315] Equation (7) involves backpropagation. This refers to the situation where, given the context vector c k In the case of taking action a k Time-Proxy DNN(π) θ The gradient of c. k The gradient is involved in learning the training progress of the ASR network. The gradient is blocked from equation (7) by the SG(.) function. k The gradient. π represents the derivative with respect to θ. θ This is the policy applied by the agent. D is the total number of output tokens in the N training pairs (microbatch).

[0316] The encoder time step K receives a reward R. K The following is about the reward R obtained as described in Table 1. K .

[0317] Equation (7) is used as the loss function for training the agent model.

[0318] The derivative (output of Equation 7) consists of calculating the derivative for each decoding step (i) of each training pair separately, and averaging the derivatives of D tokens (decoding steps). This is for ease of training.

[0319] R K a K and c K Not averaged out.

[0320] One of the agents is updated after every N training pairs. The microbatch size (N) used for training the agents is the same as the microbatch size (N) used for training the ASR modules.

[0321] The methods for obtaining equations (6) and (7) are further explained below. The update steps will be further explained with respect to step 895.

[0322] Table 1 below provides examples of rewards. A reward can have one of three values. The first value is used to accept the agent action. For example, the first value is a non-negative value. The second value is used to moderately punish the agent action. For example, the second value is a small negative value. The third value is used to severely punish the agent action. For example, the third value is a large negative value. The first value is greater than the second value, and the second value is greater than the third value.

[0323] Table 1. Reward scheme for the policy gradient algorithm used in agent training, regarding changes in prediction accuracy and stopping position.

[0324]

[0325] In Table 1, the leftmost column represents the prediction accuracy from period one (E-1) to period two (E). For example, in the first row, the first column indicates that the prediction was correct in both periods one and two. In the third row, the first column indicates that the prediction was incorrect in period one but correct in period two. The second column indicates the second stopping position (K). E ) equals the first stopping position (K) E-1 The third column indicates the second stop position (K). E ) greater than the first stop position (K) E-1 The fourth column indicates the second stop position (K). E ) less than the first stop position (K) E-1 (Scene)

[0326] The rewards are shown below.

[0327] when:

[0328] When the second stop position is greater than the first stop position; or

[0329] When the second stopping position is equal to the first stopping position and the prediction accuracy has changed between the first and second periods; or

[0330] The reward has a first value when the second stop position is less than the first stop position and the prediction in the second period is correct.

[0331] In the example in Table 1, the first value has the value "0". However, it should be understood that other non-negative values ​​can also be used.

[0332] when:

[0333] When the second stopping position is equal to the first stopping position and the prediction accuracy does not change between the first and second periods; or

[0334] The reward has a second value when the second stop position is less than the first stop position and the predictions for both the first and second periods are incorrect.

[0335] In the example in Table 1, the second value has a value of "-1". However, it should be understood that other negative values ​​can also be used.

[0336] when:

[0337] The reward has a third value when the second stop position is less than the first stop position and the prediction accuracy is correct in the first period but incorrect in the second period.

[0338] In the example in Table 1, the third value has a value of "-10". However, it should be understood that other negative values ​​can also be used. The third value is less than the second value (i.e., more negative than the second value).

[0339] Table 1 defines the reward scheme employed by the policy gradient algorithm for all agent behaviors that may be encountered during system training. Each row of the table lists four variations in the prediction accuracy for the same decoding step between two consecutive epochs (E-1 and E), and each row corresponds to three possible changes (K) in the stopping position presented by the columns. E-1 For K E The table is explained as follows:

[0340] a. Prediction Remains Correct (√→√): For all changes in the stopping position, only actions that repeatedly force a stop at the last time step T are penalized with -1, because in-line decoding should generally not keep reaching the end of the speech. Conversely, any other K... E Actions ≠ T are all accepted because they may be generated by the optimal policy at some optimization point. Since actions are monotonically generated with respect to the encoder state, we can always assume that the 'stop' decision is made in itself at the earliest time step.

[0341] b. Prediction changes from correct to incorrect (√→×): Based on hypothesis 4, for the earlier stopping position (K) in the current period E <K E-1 The action of ) is penalized with a larger penalty of -10, encouraging the agent to stop later, while according to hypothesis 3, equal (K) E =K E-1 ) or later (K E >K E-1 The stop position is accepted.

[0342] c. Prediction changes from incorrect to correct (×→√): In this case, all new stopping positions are valid because the validity of the old stopping positions in the previous period was unknown due to the incorrect prediction. Furthermore, the situation will certainly fall into either of the two categories mentioned above in the next period, and it is never too late to take further action.

[0343] d. Prediction Remains Incorrect (×→×): Similar to change b, if the stopping action creates a stopping position earlier than before, then the stopping action is moderately penalized. Note that if K E =K E-1 If K = 1, then the agent should also be prevented from stopping, because K E It is not possible to go back further on the timeline.

[0344] Additionally, and optionally, for the first epoch, the initial prediction correctness and stopping position for all decoding steps are set to '×' and T, and these two factors are tracked throughout the training process.

[0345] Return now Figure 8 In summary, in step 890, the reward R is determined for the current encoder time step K. K Although the reward is explained for a single decoder layer, it is important to understand that the reward is determined for each decoder layer of the self-attention decoder network.

[0346] In step 895, the reward for the agent in each transformer decoder layer is updated sequentially from bottom to top. The update for the agent in each decoder layer is performed according to Equations 6 and 7.

[0347] Additionally, and optionally, in update step 895, for each agent in the decoder layer, an update is performed after every N training pairs (where N is the batch size of the micro-batch method). Indicates the quantity of the micro-batch. When using... After a series of micro-batches of training agents, the agent being trained is modified. The purpose of using more than one micro-batch to train the agent is to prevent legacy issues from previous agents from being abruptly transferred to the next agent, giving each agent a better chance to develop a stable policy during this period.

[0348] Note that during system training, the stopping positions generated by the agent are processed independently of the stopping positions given by the accumulated threshold; the latter merely serves as a normalization mechanism. For inference, however, whichever occurs earlier will truncate the decoding process. Pseudocode for inference is given in Algorithm 1. For example, regarding... Figure 8 The training of the agent strategy described did not result in an accumulation threshold in the determination of reward (890), determination of correctness (880), or weight update (895).

[0349] Regarding the steps in policy gradient training 730 and 830, a further explanation of how to obtain equation (7) is provided below.

[0350] Agent 500b explores the earliest stopping position entirely on its own; no training data is provided for the stopping position. The agent relies on ASR's behavior. Deep reinforcement learning is used, enabling the agent to learn from its own experience during training.

[0351] In the framework of reinforcement learning, each decoding step in the decoder layer can be represented as a Markov Decision Process (MDP). Given an environment state S, the agent takes action A at each time step and immediately receives a reward R based on its impact on the environment. This generates a new state, and the process continues until a stopping criterion is met. The resulting sequence of the above elements is called the trajectory τ.

[0352] S1,A1,R1,S2,A2,R2,...,S K A K ,R K ,

[0353] Where S and A represent the temporary context vector c and the binary action a, respectively, and K is the number of computation steps before the agent decides to stop.

[0354] According to the reward hypothesis, the goal of a deep reinforcement learning (DRL) agent is to maximize the expected reward r received along the trajectory τ, as it follows the policy π. θ :

[0355]

[0356] As shown in equation (6), the parameter θ can be solved effectively by maximizing J(θ) through gradient ascent.

[0357] The policy gradient theorem indicates that the derivative of the expected reward can be calculated as the expectation of the product of the reward and the logarithm of the policy:

[0358]

[0359] Correspondingly, in the context of the proposed ASR system, for each decoding step (i), the derivative is:

[0360]

[0361] Here, SG(·) means preventing gradients of ASR components from entering the computation. The expectation term means that all possible stopping positions should be taken into account. To avoid the problem of high time complexity, the derivative is approximated by averaging the derivatives of all decoding steps within a microbatch of training utterances.

[0362]

[0363] Where D is the total number of output tokens in the microbatch. This approximation can be compared to the Markov chain Monte Carlo (MCMC) sampling technique. This provides multiple trajectories for gradient calculation. It also allows the agent to be jointly optimized with the ASR module.

[0364] By applying the above assumptions (1) to (4), the above expression can be expressed in the form shown in equation (7).

[0365] Examples of the DACS–ASR method

[0366] The following provides an explanation of a speech recognition method using an example. This method is called Decoder Adaptive Computation Step (DACS) ASR. This method can be similar to... Figure 5 It is implemented in the architecture. However, the DACS ASR architecture does not include the DRL-HS-DACS sublayer.

[0367] The DACS ASR method is summarized in the following pseudocode (Algorithm 2).

[0368] Algorithm 2 – DACS Inference for Converter ASR

[0369]

[0370] In Algorithm 2, after traversing the encoder time steps (line 8), the context vector c is calculated in line 15. h,l i First, make a stop decision (lines 11 and 12), then calculate the context vector.

[0371] Examples of the HS-DACS–ASR method

[0372] The following provides an explanation of a speech recognition method using another example. This method is called Head-Synchronous Decoder Adaptive Computation Step (HS-DACS) ASR. This method can be similar to... Figure 5 It is implemented in the architecture. However, the HS-DACS ASR architecture does not include the DRL-HS-DACS sublayer.

[0373] Algorithm 3 – HS-DACS Inference for Converter ASR

[0374] In Algorithm 3, after traversing the encoder time steps, the context vector c is calculated in line 15. h,l i First, make a stop decision (lines 10 and 11), then calculate the context vector.

[0375] Compared to the HS-DACS method (shown in Algorithm 3), which triggers ASR output solely by accumulating a threshold, methods and systems 200, 300A, 300B, and 500 improve stopping position by detecting acoustic information collected at each time step. This acoustic information corresponds to a temporary context vector.

[0376] For example, acoustic information can be collected by an agent. If the agent takes an active action, it will immediately emit an ASR output instead of waiting until a fixed threshold is reached through accumulation. This effectively alleviates the weak attention problem that occurs in many decoding steps. The agent model applied to the transformer decoder layer is trained using a policy gradient method, where the reward scheme for agent actions is based on variations in ASR accuracy and computational cost observed with respect to the training data, as discussed in this paper. Figure 7 and Figure 8 As stated above.

[0377] The results of experiments on the AIShell-1, Tedlium-2, and Librispeech datasets are described below, and it is shown that methods 200, 300A, 300B, and system 500 achieve reduced computational costs while maintaining similar ASR performance.

[0378] Experimental Results - DRL-HS-DACS

[0379] The following explains the research's use of information about... Figure 7 The second variant of the training method described and its use Figure 3B Experiments were conducted on the performance of a 500-bit speech recognition system trained using the 300B method. The method can be described as Automatic Speech Recognition (ASR) based on DRL-HS-DACS converters.

[0380] The performance of the DRL-HS-DACS ASR was validated on three datasets: AIShell-1 (Chinese task) and Tedlium-2 and Librispeech (English tasks). This performance is compared with other reference systems in the table below.

[0381] The input to all systems consists of 80-dimensional filter bank features plus 3-dimensional pitch-related parameters. Velocity perturbation was applied to the training data for AIShell-1 and Tedlium-2, while SpecAugment was applied to Tedlium-2 and Librispeech. The output labels for AIShell-1 consist of 4231 characters, while the output labels for Tedlium-2 and Librispeech are 1000 and 5000 BPE-tokenized word pieces, respectively.

[0382] The architecture of the online ASR system is as follows. The same front-end is shared across tasks, consisting of two CNN layers with 256 cores each, a size of 3×3 and a stride of 2×2, allowing for a 4x reduction in frame rate. The encoder stacks 12 standard self-attention layers and utilizes a chunk-by-chunk streaming strategy as described in this paper, where the left, middle, and right chunks are the same size, 64 frames. The encoder's attention dimension, number of heads, and FFN unit size are {256, 4, 2048} for AIShell-1 and Tedlium-2, and {512, 8, 2048} for Librispeech. The decoder consists of six DRL-HS-DACS-based self-attention layers with the same parameters as the encoder for each task. The hidden layer size of the proxy is set to be equal to the attention dimension.

[0383] Joint CTC / attention training is performed on all tasks with a weight of 0.3 to accelerate convergence. We follow the Noam weight decay scheme with 25,000 warm-up steps, training models on AIShell-1, Tedlium-2, and Librispeech using the total number of epochs {50,10}, {100,1}, and {120,5} and the initial learning rate, respectively. For each surrogate model, successive training steps... The bias term added to the surrogate output is set to 10 and given as -4. Maximum look-ahead step M is not applied for this training process.

[0384] During inference, for tasks in the aforementioned order, CTC scores are also merged using weights of {0.3, 0.3, 0.4}. An external language model is also trained to re-evaluate the n best hypotheses generated from beam search decoding; this external language model is a 650-unit, 2-layer Long Short-Term Memory (LSTM) network for AIShell1 and a 2048-unit, 4-layer LSTM for Tedlium-2 and Librispeech. The latency constraint M is set to 16.

[0385] Tables 2, 3, and 4 present the results of the proposed online ASR system on three datasets using the character / word error rate (CER / WER). For the token sequence, the CER / WER can be obtained in the form (S+D+I) / N, where S is the number of replacements, D is the number of deletions, I is the number of insertions, and N is the number of characters in the reference sequence.

[0386] To ensure a fair comparison, a reference system with a similar architecture and external language model to that used in our experimental setup was selected. Based on the results, it can be observed that the DRL-HS-DACS model achieves ASR performance very close to that of the HS-DACS model for all tasks. Specifically, for the test sets of AIShell-1 (6.7% CER) and Tedlium-2 (8.3% WER), and the clean test set of Librispeech (2.6% WER), the proposed system achieves even lower error rates, outperforming previous existing results of 6.8% CER, 8.4% WER, and 2.7% WER achieved by the HS-DACS system.

[0387] Table 5 compares the computational costs of the DRL-HS-DACS algorithm and HS-DACS using r-value measurements:

[0388]

[0389] The r-value was calculated during the adaptive computation steps. The ratio between the length of the input and the length of the utterance, T, both of which are within the N of the utterance. d The values ​​are summed across 1 decoder layer, H attention heads, and L decoding steps. A smaller r-value indicates less computational cost during decoding. The figures reported in Table 5 are the average r-values ​​for all utterances within the test set. Compared to the HS-DACS baseline, the DRL-HS-DACS-based system achieves a significant reduction in r-value, with relative gains of 40.0%, 32.8%, and 53.4% ​​for the AISHell-1, Tedlium-2 test sets, and Librispeech's test-clean / other set, respectively. Lower computational cost means that for most decoding steps, the surrogate-optimized stopping position occurs earlier than the stopping position generated using only a fixed threshold, facilitating faster ASR output.

[0390] Table 2. Character Error Rate (CERs%) for AIShell-1

[0391]

[0392] Table 3. Word Error Rate (WERs%) for Tedlium-2

[0393]

[0394] Table 4. Word Error Rate (WERs%) for Librispeech

[0395]

[0396] Table 5. R-values ​​for AIShell-1, Tedlium-2, and Librispeech

[0397]

[0398] Experimental Results - Impact of DRL-HS-DACS-Accumulation Threshold on Training

[0399] The following explains the research and related matters. Figure 7 The first variant of the training method described is compared to using about Figure 7 Experiments were conducted to demonstrate the performance of a speech recognition system 500 trained using a second variant of the training method described.

[0400] To explore the effect of the accumulated threshold H on normalizing multiple attention heads, we consider the case where the agent works alone (i.e., as with respect to...). Figure 7 A control experiment was conducted on AIShell-1 (as described in the first variant of the training method). ASR inference performance was also investigated. Table 6 provides a comparison of ASR performance and computational cost between DRL-HS-DACS systems with and without thresholds (Method 300B) and without thresholds (Method 300A).

[0401] Compared to systems with thresholding (Method 300B, second training variant), systems trained and inferred without thresholding (Method 300A, first training variant) achieved lower r-values. A reduction in CER was observed on both the development and test sets, with relative losses of 10.3% and 22.4%, respectively. When thresholding is absent (as in Method 300A and the first training variant), unrestricted stopping probabilities can easily be distributed throughout the encoder state, thus corrupting the context vector in the initial stages of training. As a result, the agent tends to stop decoding early to avoid propagating invalid acoustic information to subsequent computations. This leads to a reduced r-value at the cost of decreased ASR accuracy. Figure 9 The graph shows the ASR loss of the above systems on the development set during training. The system with only the surrogate (Method 300A, first variant) initially converges quickly but soon overfits the training data. The loss is consistently higher than that of the system trained with a threshold (Method 300B, second training variant), which explains the decrease in ASR accuracy.

[0402] Inference using cumulative attention algorithm

[0403] Figure 10 This is a flowchart of a speech recognition method according to another embodiment. In step 1210, a speech audio frame is received. Step 1210 corresponds, for example, to... Figure 2 Step 210.

[0404] In step 1230, the speech audio frame is encoded. Step 1230 corresponds, for example, to... Figure 2 Step 230.

[0405] In step 1240, the context vector c is determined (step 1240-a), the stopping probability is derived (1240-b), and it is determined whether the obtained stopping probability satisfies the predetermined condition (1250).

[0406] In step 1240-a, a context vector is determined based on the encoding of the speech audio frame in step 1230. The determination of the context vector is explained in more detail below. Briefly, the context vector is based on the acoustic information of the encoded frame. It may also carry historical acoustic information (e.g., from previous speech audio frames). The context vector is also referred to as a temporary context vector. As described later, a temporary context vector can be obtained for each time step (frame) in an autoregressive manner (i.e., based on its past values).

[0407] In step 1240-b, the stopping probabilities are derived from the context vector. For example, the stopping probabilities are derived using a trainable model. This model can be referred to as a stopping selector. The stopping selector comprises a trained neural network. The stopping selector includes parameters Φ as trainable parameters. Parameter Φ can also be referred to as the weights of the stopping selector. For example, the neural network is implemented as a single-layer or multi-layer DNN. The stopping selector takes the context vector as input. The stopping selector then maps the context vector to stopping probabilities.

[0408] The stopping probability represents the likelihood that the decoding step will stop at a specific time wavelength.

[0409] The trained neural network for the stop selector can be implemented as a single-layer / multi-layer deep neural network (DNN). For example, a DNN has an output dimension of 1. In this example, the DNN consists of two fully connected layers, where the first layer has input and output dimensions of {D, D}, and the second layer has input and output dimensions of {D, 1}, where D is the dimension of the context vector. Alternatively, the stop selector can consist of a neural network comprising two recurrent neural networks (RNNs) with the same dimensions as the DNN.

[0410] In step 1250, it is determined whether the obtained stopping probability meets a predetermined condition. If the condition is met, the method proceeds to step 1260. When the condition is met, a stopping decision is made. The stopping decision refers to stopping step 1240.

[0411] As described below, step 1240 can be performed at the decoder end of the transformer ASR. For example, step 1240 can be performed by the decoder layer of the decoder neural network. Step 1240 can be referred to as the decoding step.

[0412] In step 1260, the token is exported. Step 1260 corresponds, for example, to... Figure 2 Step 260. If the condition is not met, the method returns to step 1240, where a temporary context vector and stopping probability are determined. Then step 1240 is repeated for subsequent speech audio frames.

[0413] Optionally, satisfying the predetermined condition includes determining whether the stopping probability is greater than or equal to a predetermined value. For example, if the stopping probability is greater than or equal to "0.5", the condition is considered satisfied.

[0414] In step 1260, a token is derived. In the speech recognition method, when speech audio is recognized as text, the token can represent a character, phoneme, morpheme or other morphological unit, word portion or word.

[0415] In step 1270, a function is executed based on the exported token. The executed function includes at least one of command execution and / or text output.

[0416] Additionally and optionally, as described with respect to the following figures, step 1240 may include computing more context vectors and computing combined context vectors. Step 1240 may also include using one or more linear layers to process the context vectors or combined context vectors. Additionally or optionally, steps 1240 and 1260 may also include utilizing a classification layer to process the context vectors, combined context vectors, or the output of the one or more linear layers. The output of the classification layer may be a plurality of probabilities or scores, such as a vector of probabilities or scores, where each probability or score indicates the likelihood that a given token is correct. The classification layer may be a cross-entropy layer, a softmax layer, or a variant or equivalent thereof. The derived token may be the token with the highest corresponding probability or score, or may be selected probabilistically based on the probability or score, for example, a token with a higher probability or score is more likely to be selected. The probabilities or scores may be used with a joint decoder, which also utilizes a language model based on the probability or score of each token output from the previously output tokens. Such a joint decoder may compute joint probabilities or scores based on the language model probabilities or scores and the classification layer probabilities or scores. Tokens can be derived as the tokens with the highest combination probability or score, and can be selected probabilistically based on combination probability or score; for example, tokens with higher combination probability or score are more likely to be selected.

[0417] Inference using the cumulative attention algorithm – cross-attention layer

[0418] Figure 11 This is a schematic diagram of method 1300 according to an embodiment. Method 1300 can be used to implement, for example, regarding... Figure 10 Step 1240 is explained.

[0419] Method 1300 relates to a decoder layer in a decoder neural network. The decoder layer may be referred to as a self-attention decoder layer. The decoder neural network may be referred to as a self-attention decoder. Decoder layers and decoder neural networks will be further described below. A decoder neural network may include one or more decoder layers (l). Decoder layers may be stacked. Stacking layers means that the output of one layer is fed as input to subsequent layers.

[0420] Optionally, method 1300 involves the last decoder layer in a stacked decoder layer. The remaining decoder layers in this stacked decoder layer are equipped with self-attention modules and perform language modeling. This stacked decoder layer will... Figure 12A To clarify, the term "last decoder layer" refers to a decoder layer whose output is not passed to other decoder layers. For example, instead, the output is passed to an output layer. The output layer may include linear and / or classification layers. Applying Method 1300 only to the last layer improves ASR accuracy and reduces latency.

[0421] Alternatively, method 1300 can also be applied to other layers of the decoder neural network.

[0422] Method 1300 includes an initialization step 1330. In step 1330, the context vector is initialized (c h i,0 The context vector is represented as c. h i,j Here, h represents the index of the attention head. There can be H attention heads. "i" represents the decoder step, and "j" represents the encoder time step (and corresponds to the audio frame being considered). The above statements will be explained in more detail below.

[0423] The terms "context vector," "attention head," "decoder step," and "encoder time step" are related to the converter ASR. The converter ASR is streamlined to include combining attention weights across individual heads (when more than one head is included) and triggering the ASR to output for all heads at the same time step. This is known as Head-Synchronous Decoder-Side Adaptive Computation Step (HS-DACS).

[0424] Method 1300 can be referred to as Cumulative Attention (CA). CA refers to the triggering of the ASR output based on the accumulation of acoustic information at each encoding time step (j). When more than one attention head is included, the attention heads in the same decoder layer are synchronized to have a uniform stop position. Synchronizing the attention heads alleviates the problems caused by the different behaviors of the individual heads.

[0425] Method 1300 can be applied to the final decoder layer of a decoder neural network.

[0426] In step 1330, the context vector is initialized. For example, the context vector is initialized to 0. h i,0 =0.

[0427] For each audio frame (j) until the condition is met, perform the following steps.

[0428] In step 1340, the attention weights (a) are determined. h i,j To obtain the attention weights, the attention energy (e) is first calculated. h i,j For example, taking the decoding step i of the last decoder layer, for a certain head h, the attention energy e at time step j. h i,j Calculated as:

[0429]

[0430] In Equation 8, q and k are the decoder state and encoder state, respectively. Attention energy e h i,j It is then passed to the sigmoid unit to generate monotonic attention weights:

[0431]

[0432] The sigmoid unit scales the energy to the range (0,1) without accessing the entire input sequence. The result of Equation 9 represents the correlation between the encoder state and the current decoding step (i).

[0433] In step 1350, the temporary context vector (c) is calculated. h i,j For example, the following uses an autoregressive approach to generate a temporary context vector at each time step (j):

[0434]

[0435] In equation (10), c h i,j-1 v represents the temporary context vector from the previous encoder step size j-1. h i,j This represents the encoder state at time step j. When j = 1, term c h i,j-1 Deprecated (e.g., set to 0). The temporary context vector carries all processed acoustic information accumulated at the current time step.

[0436] Note that the temporary context vector is computed on the fly and carries all acoustic information from the past. The context vector is computed first, and then used to derive the stopping probability.

[0437] For each attention head h, proceed to steps 1340 and 1350.

[0438] After considering all self-attention heads h, proceed to step 1360.

[0439] In step 1360, the temporary context vectors obtained for each attention head h are concatenated (c i,j For example, it can be in c i,j =Concat(c 1 i,j ,…,c H i,j The context vector of the link is obtained in the form of ).

[0440] It should be understood that in an arrangement with only one attention head (h=1), step 1360 is optional. The context vector from step 1350 can then be used instead of the linked vector.

[0441] In step 1370, the stopping probability (p) is derived. i,j ). The context vector of the connection (c i,j The ) is immediately sent to it to derive the stopping probability (p) i , j The stop selector is ). Note that the temporary context vector is computed up to time step j.

[0442] The stopping probability (p) is obtained as follows i , j ):

[0443] p i，j =Sigmoid(HaltSelect(c i，j Equation (11) is given by: )+r+∈)

[0444] Stopping probability (p) i , j ) represents the cumulative acoustic features (c) of all attention heads so far. i , j In the case of ), the probability of stopping the i-th decoding step at time step j. The stopping probability of equation (11) can be called the third probability. The third probability is different from the second probability described with respect to equation (2). The third probability (equation (11)) has a similar physical meaning and function to the first probability of equation (4).

[0445] In equation (11), parameter r represents a bias term that can be initialized to a predetermined value, and ∈ is applied only during training to encourage p. i , j The cautious additive Gaussian noise. During inference, ∈ is not included. For example, r can have any negative value. For example, r is -4. "HaltSelect()" is applied by the stop selector described in this document. i , j It is the context vector of the connection as described above.

[0446] In step 1380, check the stopping probability p. i , j Does it meet the fourth condition? For example, the condition is: "stopping probability p..." i , j ≥0.5? If this condition is met, method 1300 terminates and triggers ASR output (e.g., by using the context vector c). i , j Output to the output layer to predict ASR output). If the condition is not met, repeat method 1300 for subsequent time steps (i.e., subsequent values ​​of j).

[0447] During CA inference, p is monotonically calculated at each time step starting from j=1. i,j And the decoding step i will be in, for example, p i , j The earliest point j with a value ≥0.5 was stopped.

[0448] Note that the stopping decision is based on the entire encoding history because the stopping probability is obtained from the temporary context vector.

[0449] about Figure 10 and Figure 11 The speech recognition method described can be represented by the following pseudocode.

[0450] Algorithm 4. Cumulative Attention Inference for Transformer ASR

[0451]

[0452] In the pseudocode (Algorithm 4), <sos>It is a token indicating the beginning of a sentence. <eos>This is a token indicating the end of a sentence. The other variables and parameters shown in Algorithm 4 correspond to those described regarding Method 1300. In line 3, initialization step 1330 is performed. In lines 5-8, for each attention head (for h = 1 to H), attention weights are determined (step 1340), and a temporary context vector is computed (step 1350). In line 9, the temporary context vector is concatenated (step 1360). In line 10, for the encoder time step (j), the stopping probability is derived (step 1370). Line 10 of the algorithm corresponds to Equation 11. In line 11, the stopping probability is checked to see if the condition is met (step 1380). In line 12, "break" indicates the stopping action where the iteration through the encoder time step (line 4) is stopped. In line 15, the context vector c... i Set as the temporary context vector for the connections computed so far. In line 16, t i This represents the time step at which the algorithm stops for a specific decoder step (i). In other words, t i Provides an indication of when the stopping decision was made (i.e., the stopping position). In line 16(t) i =max(t) i-1 ,j)),t i Set as t i-1 The larger of the current value or j (which is the stopping position of the current decoder layer). In line 17, for each decoder step (i), the token y is derived. i The token is the context vector c from the last decoder layer. i Derived. In line 18, increment the decoder step variable (i). Referring to system 1500, OutputLayer(.) in line 17 can correspond to linear and classification layers 528, joint online decoder 530, and / or language model 540.

[0453] In Algorithm 4, based on the stopping probability p i,j If the value is ≥0.5, the decoding step can be stopped. This can be referred to as the fourth condition or condition 4.

[0454] Architecture of cumulative attention algorithm

[0455] The following section explains the ASR system 1500. Refer to [link / reference] afterward. Figure 12B Explain the training of ASR system 1500.

[0456] Figure 12A This diagram illustrates a system for performing speech recognition according to an example embodiment. System 1500 can be implemented using one or more computing devices, such as one or more computer-executable instructions on the hardware described below. System 1500 may also be referred to as an ASR system, an ASR module, or a speech recognition system or speech recognition module.

[0457] System 1500 can use one or more computing devices, such as those related to computing devices. Figure 15 This is implemented using one or more computer-executable instructions on the hardware 900 described.

[0458] Besides the decoder neural network, the ASR system 1500 is related to... Figure 5 The ASR system 500 is similar to the converter system described.

[0459] The ASR system 1500 includes a self-attention encoder 1510. The self-attention encoder 1510 corresponds to... Figure 5 The self-attention encoder 510.

[0460] The ASR system 1500 includes a self-attention decoder 1520. Except for the differences in self-attention decoder layers 1525 and 1526, self-attention decoder 1520 is similar to self-attention decoder 520. Embedding layer 1522 corresponds to... Figure 5 Layer 522. Position coding layer 1524 corresponds to Figure 5 Layer 524. Linear and classification layers 1528 correspond to... Figure 5 Layer 528. The language model 1540 and the joint online decoder 1530 also correspond to... Figure 5 Layers 540 and 530.

[0461] The self-attention decoder 1520 includes a stack of self-attention decoder layers. For example, the self-attention decoder 1520 includes a stack of L self-attention decoder layers. Each of the first L-1 self-attention decoder layers 1525 has the following configuration. The first L-1 layers 1525 correspond to the self-attention decoder layers except for the last self-attention decoder layer 1526. Each self-attention decoder layer 1525 includes two sub-layers: a self-attention sub-layer and a positional feedforward sub-layer. Each self-attention decoder layer 1525 has the same configuration and operates in the same manner as the self-attention encoder layer 516. The input to the self-attention decoder layer 1525 goes into the self-attention sub-layer and then into the feedforward network. Each self-attention decoder layer 1525 performs language modeling.

[0462] Each self-attention decoder layer 1525 may use residual connections and layer normalization after each sublayer. The first of one or more self-attention decoder layers 1525 receives the enhanced embedding as input, i.e., the output of the position encoder layer. Subsequent self-attention decoder layers receive the output of the previous self-attention decoder layer. In the implementation, one or more self-attention decoder layers 1525 may be 6 self-attention decoder layers, the dimension of the self-attention sublayer of each self-attention decoder layer may be 256, the position feedforward sublayer of each self-attention decoder layer 1525 may have 2048 units, and each self-attention decoder layer may include 4 attention heads.

[0463] The final self-attention decoder layer 1526 comprises a cascade of multi-head self-attention sublayers, multi-head cross-attention sublayers, and feedforward network (FFN) sublayers. Layer normalization is also applied to their outputs to facilitate model convergence. Step 1240 or method 1300 is performed in the multi-head cross-attention sublayer of the final decoder layer 1526. For simplicity, this sublayer will be referred to as the final decoder layer. The final decoder layer implements the stop selector 1500b. The stop selector 1500b is implemented... Figure 10 Method or Figure 11 Method 1300, and is as follows regarding Figure 12B The training mentioned above.

[0464] Training to accumulate attention

[0465] Figure 12B This is a flowchart of method 1700 for training speech recognition system 1500. Speech recognition system 1500 can be implemented in... Figure 10 and Figure 11 The method described in the diagram. System 1500 includes a stop selector 1500b. Stop selector 1500b corresponds to the stop selector described above with respect to 1240 and 1300. Stop selector 1500b is trained together with ASR module 1500 using the same loss function.

[0466] The ASR system 1500 and stop selector 1500b were trained together. Both systems were trained using the same training data. The parameters of system 1500 and stop selector 1500b were updated together. Both systems were trained using the same loss function.

[0467] For each training pair in the training set, the following steps are performed. Each training pair 1710 includes multiple speech audio frames and a training token sequence. The training sequence of tokens can also be described as a gold standard sequence or a reference sequence. The training sequence of tokens can be derived from human indications of the utterance content corresponding to the multiple speech audio frames. For example, a token sequence representing words, phrases, or sentences.

[0468] A token sequence is derived from multiple speech audio frames. For each frame, the token can be derived using steps 1230 to 1260 of method 1200 or steps 1330 to 1380 of method 1300. The above steps can be implemented by the speech recognition system 1500.

[0469] When the token sequence includes multiple tokens, more tokens in the token sequence can be obtained by applying the above steps to the remaining frames in the multiple voice audio frames.

[0470] Exporting the token sequence and updating the weights of the speech recognition system includes the following steps.

[0471] i. Encode each frame as described in step 1230.

[0472] ii. Feed the encoder state of the encoder layer to the final decoder layer 1526 of the decoder neural network 1520.

[0473] iii. For each encoder time step (j), as described with respect to steps 1330, 1340, 1350, 1360, and 1370, derive the stopping probability p. i,j .

[0474] iv. Then, the distribution of the stopping hypothesis is derived. The distribution of the stopping hypothesis (α) is obtained as follows. i,j ):

[0475]

[0476] -α i,j When considering all possible stopping positions of the current decoding step (i), p i,j The expected value. In this case, assume the stop selector can act based on its own stopping probability p. i,j Each time step (j) is selected as the stopping position. To include all stopping positions (stopping hypotheses), the expected value of the stopping probability at the current decoding step is calculated.

[0477] - In Equation 12, the stopping probability p i,j Each result is multiplied by the sum of the stopping probabilities at each of the previous time steps from 1 to j-1.

[0478] - Since the stop selector imposes hard decisions, making the coefficient parameters non-differentiable, we can marginalize all possible stop positions by considering the context vector c. i The expected value. Hard decision means indicating which time step is the stopping position without considering any other time steps based on probability. Marginalizing all possible stopping positions means considering all stopping positions.

[0479] The so-called non-differentiability of system parameters means that, in this case, since the stopping probability (the output of the stopping selector) is not used, the parameters of the stopping selector are separated from the rest of the ASR system. Therefore, the parameters cannot be updated along with the ASR module, resulting in the stopping selector parameters being non-differentiable with respect to the cross-entropy loss function.

[0480] v. The expected context vector is calculated as follows:

[0481]

[0482] - In Equation 13, c i,j It is the context vector of the connection, and is obtained as described above regarding steps 1330, 1340, 1350, and 1360, α i,j As stated above.

[0483] vi. Then, the expected context vector from equation (13) is fed into the feedforward network to obtain the predicted ASR output. For example, the ASR output is as shown above. Figure 12A Or the token derived from Algorithm 4.

[0484] vii. The predicted ASR output is then guided to the loss function for parameter weight updates. The weight updates follow the loss function, based on the difference between the derived token sequence and the training sequence of tokens. For example, the loss function could be the cross-entropy loss function. The weight updates are similar to those concerning... Figure 7 The weight update 720 is described. However, in this case, the weight update 1720 involves the weights of the entire ASR system 1500 (including the weights of the stop selector 1500b).

[0485] Experimental results of the CA algorithm

[0486] about Figure 13 , Figure 14A and Figure 14B This describes an experiment demonstrating the performance of the 1500 speech recognition system. The 1500 system can achieve... Figure 10 Or the method of 11, and is as about Figure 12B The training mentioned above.

[0487] The performance of the CA algorithm was validated on two datasets: the AIShell-1 Chinese task and the Librispeech English task. This performance is compared with other reference systems and is shown in the table below.

[0488] A system that implements the CA algorithm is also called a CA converter.

[0489] The datasets are prepared as follows. Velocity perturbation is applied to AIShell-1, while SpecAugment is applied to Librispeech. Acoustic features consist of 80-dimensional filter bank coefficients plus 3-dimensional pitch information. The vocabulary of the datasets includes 4231 Chinese characters for AIShell-1 and 5000 BPE-tokenized word blocks for Librispeech.

[0490] The architecture of the online ASR system is as follows. The same front-end is shared across tasks, consisting of two CNN layers with 256 cores each, 3×3 width and 2×2 stride, allowing for a 2x reduction in frame rate. The encoder stacks 12 standard self-attention layers and utilizes a chunk-by-chunk streaming strategy as described in this paper, where the size of the left, middle, and right chunks is {64, 64, 32} frames. The number of heads, attention dimensions, and FFN unit size for each encoder layer are {4, 256, 2048} for AIShell-1 and {8, 512, 2048} for Librispeech. The decoder consists of 6 layers with the same parameters as the encoder for each task. For systems implementing the CA algorithm, the final decoder layer includes a stop selector.

[0491] Joint CTC / attention training was performed on all tasks with a weight of 0.3 to accelerate convergence. The learning rate for both tasks followed the Noam weight decay scheme, with the initial learning rate, warm-up steps, and number of epochs set to {1.0, 25000, 50} for AIShell-1 and {5.0, 25000, 120} for Librispeech. An external language model (LM) trained on the text of the training set was incorporated to re-evaluate the beam search (beam width = 10) hypothesis decoded by the system. The LM was a 650-unit 2-layer Long Short-Term Memory (LSTM) network for AIShell-1 and a 2048-unit 4-layer LSTM for Librispeech.

[0492] Tables 6, 7, and 8 present the results of the proposed online ASR system for the two datasets using the character / word error rate (CER / WER).

[0493] To ensure a fair comparison, reference systems with similar architectures and external language models used in systems implementing the CA algorithm were selected. For example, both the MoChA-based and HS-DACS-based reference systems utilize only one cross-attention layer (D=1) for training, except for the HS-DACS for Librispeech, which has three cross-attention layers (D=3, since models with D=1 or 2 do not converge well). Here, D represents the number of decoder layers starting from the top layer (i.e., the last layer) with cross-attention sublayers. If D=1, then only the topmost layer has a cross-attention sublayer.

[0494] Table 6. Character Error Rate (CERs%) for AIShell-1

[0495]

[0496] Table 7. Word Error Rate (WERs%) for Librispeech

[0497]

[0498] As shown in Tables 6 and 7, the CA converter achieves improved accuracy. For AIShell-1, CA achieves a 2.8% relative gain compared to MoChA and HS-DACS models, and performs similarly to HS-DACS. In the case of Librispeech, CA outperforms MoChA under both clean and noisy conditions, with relative gains of 16.1% and 10.8%, respectively. Furthermore, CA still achieves a WER comparable to HS-DACS even with fewer cross-attention layers in the CA system.

[0499] The delay of the CA converter during inference was also measured and compared with reference MoChA and HS-DACS systems. The delay was determined in the form of corpus-level delay according to the following formula:

[0500]

[0501] Where N represents the total number of utterances in the dataset, |y k | represents the number of output tokens in each utterance, and the boundary of the right input block where the stop position is located. The actual boundary b of the output token k i The difference between Actual boundary b k i This can be achieved through forced alignment using an Hidden Markov Model (HMM). Since there may be ASR errors in the hypothesis sequence that could lead to incorrect delay calculations, the above equation only includes correctly decoded tokens. While this might lead to different denominators in the latency equations above, the comparison remains reasonable considering that all three systems achieve similar ASR accuracy. Furthermore, since all online attention mechanisms in our experiments are performed independently at each decoding step, the stopping position may not be monotonic. Therefore, when calculating latency, Synchronized to the furthest time step seen during the decoding process (see line 16 of the algorithm).

[0502] Table 8 presents the latency levels of the MoChA, HS-DACS, and CA-based systems evaluated on the AIShell-1 and Librispeech datasets. For a fair comparison with CA, the maximum lookahead order (M) was not applied during decoding for both MoChA and HS-DACS. For AIShell-1, the latency levels of both systems appear reasonable compared to the offline system, while for Librispeech, the latency level is close to that of the offline system, attributable to the redundant head failing to capture effective attention. To reduce latency, a maximum lookahead order of M=16 was imposed only for the Librispeech task during recognition. It can be observed that for both AIShell-1 (without M) and Librispeech (with M), CA (without M) achieves better latency levels than both MoChA and HS-DACS.

[0503] Table 8. Latency (frames) for AIShell-1 and Librispeech

[0504]

[0505] The poor latency performance given by MoChA and HS-DACS can be explained by observing the stopping decisions of different heads that generate ASR outputs, such as... Figure 13 and Figure 14A As shown in the image. Figure 13 Eight subgraphs are shown, each corresponding to the output of one of the eight attention heads of the MoChA system. The subgraphs are shown sequentially and correspond to attention heads 1 through 4 (row 1, from left to right) and attention heads 5 through 8 (row 2, from left to right). Each subgraph is a color map showing the output sentence (vertical axis) relative to the input (horizontal axis), with colors representing stopping decisions. Bright colors indicate that decoding has stopped, while dark colors indicate that no stopping decision was obtained (and thus decoding was not stopped). The input can be understood as the index of an audio frame or encoder index (j). The output corresponds to the token sequence {_HE,_MUST,_HAVE,_REALI,Z,ED,_I,_WAS,_A,_STRANGER,_AND,_WISHED,_TO,_TENDER,_HIS,_HOSPITALITY,_TO,_ME,_I,_ACCEPTED,_IT,_GRATEFULLY,_I,_CLASPED,_HIS,_HAND,_HE,_PRESSED,_MINE,} <eos>}

[0506] from Figure 13 As can be seen, the first three (2, 3, and 6) of MoChA are essentially unable to stop decoding, relying solely on truncation performed by the maximum lookahead order. For these heads, MoChA fails to stop decoding in most cases, causing it to fail to stop because the stopping position depends on the head that makes the final decision. Instead, MoChA relies on the maximum lookahead order to stop. More specifically, for head 2, the stopping decision (the bright area) is not monotonic. For heads 3 and 6, no stopping decision is obtained for any output token (instead, the maximum lookahead order is used to stop decoding).

[0507] Figure 14A The stopping decision of the HS-DACS system is illustrated. Figure 14A In the middle, the horizontal and vertical axes are parallel to each other. Figure 13 Same. Color indicates the stop decision (corresponding to p in line 7 of Algorithm 3). h,l i,j Bright colors indicate that decoding has stopped, while dark colors indicate that no stop decision was made. From Figure 14A It can be observed that some stopping positions deviate from the previous stopping position, while the stopping position in the next decoding step returns to normal. This can be seen, for example, in the decoding steps of '_HAVE' and the first '_I'. It can be perceived that stopping is accomplished through the maximum look-ahead order. The accumulation of stopping probabilities in the HS-DACS head does not exceed the joint threshold (number of heads, 8) in some decoding steps, causing the inference process to reach the end of the speech early. For example, the stopping decision (bright area) is not monotonic.

[0508] Figure 14B The stopping probability of the CA converter is shown. Figure 14B In the middle, the horizontal and vertical axes are parallel to each other. Figure 13 Same. Color indicates the stop decision (corresponding to p in lines 10 and 11 of Algorithm 4). h,l i,j A bright color indicates that decoding has stopped (i.e., the condition in line 11 of Algorithm 4 has been met), while a dark color indicates that no stopping decision has been obtained.

[0509] With MoChA ( Figure 13 ) and HS-DACS ( Figure 14A Unlike other converters, the stopping decision in a CA converter is monotonically and always made in a timely manner. Although a CA may have redundant heads, these heads can be backed up by other normally functioning heads because they are synchronized, and the stopping decision is based on overall voice information.

[0510] Computer hardware

[0511] Figure 15 This is a schematic diagram of the hardware that can be used to implement the methods and systems described in the embodiments herein. It should be noted that this is only an example and other arrangements may be used.

[0512] The hardware includes the computing section 900. In this particular example, the individual components of this section will be described together. However, it is important to realize that they are not necessarily located in the same place.

[0513] The components of the computing system 900 may include, but are not limited to, a processing unit 913 (such as a central processing unit, CPU), a system memory 901, and a system bus 911 that couples the various system components, including the system memory 901, to the processing unit 913. The system bus 911 can be any of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of various bus architectures. The computing section 900 also includes external memory 915 connected to the bus 911.

[0514] System memory 901 includes computer storage media in the form of volatile and / or non-volatile memory such as read-only memory. A basic input / output system (BIOS) 903, containing routines such as those that help transfer information between components within the computer during startup, is generally stored in system memory 901. Additionally, system memory contains operating system 905, application programs 907, and program data 909 used by CPU 913.

[0515] Additionally, interface 925 is connected to bus 911. This interface can be a network interface for the computer system to receive information from other devices. It can also be a user interface that allows a user to respond to certain commands, etc.

[0516] In this example, a video interface 917 is provided. The video interface 917 includes a graphics processing unit 919 connected to a graphics processing memory 921.

[0517] Due to its suitability for data-parallel operations such as neural network training, the graphics processing unit (GPU) 919 is particularly well-suited for training speech recognition systems. Therefore, in an embodiment, the processing for training the speech recognition system can be divided between the CPU 913 and the GPU 919.

[0518] It should be noted that in some embodiments, different hardware can be used to train the speech recognition system and perform speech recognition. For example, training of the speech recognition system can occur on one or more local desktop or workstation computers, or on a cloud computing system that may include one or more discrete desktop or workstation GPUs, one or more discrete desktop or workstation CPUs (e.g., processors with PC-oriented architectures), and a large amount of volatile system memory (e.g., more than 16 GB). However, for example, the execution of speech recognition can use mobile or embedded hardware that may include a mobile GPU as part of a system-on-a-chip (SoC) or without a GPU; one or more mobile or embedded CPUs (e.g., processors with mobile-oriented or microcontroller-oriented architectures) and a smaller amount of volatile memory (e.g., less than 1 GB). For example, the hardware for performing speech recognition could be a voice assistant system 120, such as a smart speaker, or a mobile phone that includes a virtual assistant. Compared to hardware used to perform tasks using an agent, the hardware used to train the speech recognition system can have significantly greater computing power, such as being able to perform more operations per second, and has more memory. It is possible to use hardware with fewer resources because, for example, performing speech recognition by inference using one or more neural networks is far less computationally intensive than, for example, training a speech recognition system by training one or more neural networks. Furthermore, techniques can be employed to reduce the computational resources required for speech recognition, such as for inference using one or more neural networks. Examples of such techniques include model distillation, and for neural networks, neural network compression techniques such as pruning and quantization.

[0519] Although some embodiments have been described, these embodiments are given by way of example only and are not intended to limit the scope of the invention. In fact, the novel apparatus and methods described herein can be embodied in various other forms; furthermore, various omissions, substitutions, and modifications can be made to the form of the apparatus, methods, and products described herein without departing from the spirit of the invention. The appended claims and their equivalents are intended to cover such forms or modifications that fall within the scope and spirit of the invention.< / eos> < / eos> < / sos> < / eos> < / sos>

Claims

1. A computer-implemented method for speech recognition, the method comprising: Receive voice audio frames; Encode the received frames; The context vector is determined from the encoding of the received frame; Derive the action from the context vector; In response to the action satisfying a predetermined condition, a token is derived from the context vector; as well as Based on the token execution function, wherein the function includes at least one of text output or command execution. The action is derived through a proxy, wherein the proxy includes a trained model, and wherein deriving the action includes: The context vector is input into the trained model; The first probability is determined from the trained model; Compare the first probability with a predetermined threshold; and The action is set based on the comparison. The first probability is calculated using the self-attention decoder layer of the decoder neural network. Determining the context vector includes: Determine a second probability, where the second probability corresponds to the attention weight of the self-attention decoder layer.

2. The method according to claim 1, wherein determining the first probability from the trained model comprises: The output of the trained model is added to a predetermined bias; as well as Apply the sigmoid function to the sum to obtain the first probability.

3. The method according to claim 2, wherein the agent comprises a deep neural network.

4. The method of claim 1, wherein determining the context vector includes using a context vector determined from a previous speech audio frame.

5. The method according to claim 1, wherein determining the context vector comprises: The second probability is multiplied by the encoder state, which is derived from the received speech audio frames.

6. The method according to claim 4 or 5, wherein the self-attention decoder layer is a multi-head self-attention decoder layer comprising a plurality of attention heads, and wherein the stopping probability is calculated using the attention heads among the plurality of attention heads.

7. The method of claim 6, wherein determining the context vector comprises determining a first context vector from each of the plurality of attention heads, and concatenating the determined first context vectors to obtain the context vector.

8. The method according to any one of claims 4, 5, and 7, comprising adding the determined second probability to an accumulator variable, and wherein... In response to the accumulator variable satisfying the second condition, Derive the token from the context vector; and The token is used to execute functions, wherein the functions include at least one of text output or command execution.

9. The method of claim 1, wherein the action corresponds to a third probability, wherein the action is derived by a stop selector configured to generate a third probability, wherein the third probability represents the probability that a token should be derived from the context vector.

10. A computer-implemented method for training a speech recognition system, the method comprising: For training data that includes voice audio frames and training tokens, The speech audio frames are encoded; The context vector is determined from the encoding of the frame; Actions are derived from the context vectors through an agent that includes a trainable model; In response to the action satisfying a predetermined condition, the predicted token is derived from the context vector; The correctness of a prediction is determined by comparing the predicted token with the training token, where the correctness of a prediction indicates that the prediction is correct when the predicted token matches the training token. The stop position is obtained based on the predicted token, wherein the stop position represents the time step in which the predetermined condition is met; The reward is determined based on the accuracy of the prediction and the stopping position obtained. as well as The agent's weight is updated based on the determined reward.

11. The method of claim 10, wherein determining the reward comprises: Obtain the first prediction accuracy and the first stopping position from the first training period; The second prediction accuracy and the second stopping position are obtained from the second period, where the first period precedes the second period; and Compare the accuracy of the first prediction with the accuracy of the second prediction, and compare the first stopping position with the second stopping position.

12. The method according to claim 11, wherein: The determined reward has a first value in the following cases: The second stop position is greater than the first stop position; or The second stopping position is equal to the first stopping position, and the accuracy of the first prediction is different from the accuracy of the second prediction; or The second stop position is smaller than the first stop position, and the second prediction correctness indicates that the prediction is correct; The determined reward has a second value in the following circumstances: The second stopping position is equal to the first stopping position, and the first prediction accuracy is equal to the second prediction accuracy; or The second stopping position is smaller than the first stopping position, and the first prediction accuracy and the second prediction accuracy are equal but lower; as well as When the second stop position is less than the first stop position and the second prediction accuracy is less than the first prediction accuracy, the determined reward has a third value. The first value is greater than the second value, and the second value is greater than the third value.

13. The method according to claim 12, wherein the first value is zero, and the second and third values ​​are negative.

14. The method according to any one of claims 10 to 13, wherein determining the context vector comprises: Determine the second probability, where the second probability corresponds to the attention weight of the self-attention decoder layer; as well as The determined second probability is added to the accumulator variable, wherein satisfying the predetermined condition includes comparing the accumulator variable with a predetermined threshold.

15. The method according to claim 14, wherein the predetermined condition is satisfied when the accumulator variable is less than the predetermined threshold.

16. A carrier medium comprising computer-readable code configured to cause a computer to perform the method according to any one of the preceding claims.

17. A system for speech recognition, the system comprising a processor configured to: Receive voice audio frames; Encode the received frames; The context vector is determined from the encoding of the received frame; Derive the action from the context vector; In response to the action satisfying a predetermined condition, a token is derived from the context vector; as well as Based on the token execution function, wherein the function includes at least one of text output or command execution. The action is derived through a proxy, wherein the proxy includes a trained model, and wherein deriving the action includes: The context vector is input into the trained model; The first probability is determined from the trained model; Compare the first probability with a predetermined threshold; and The action is set based on the comparison. The first probability is calculated using the self-attention decoder layer of the decoder neural network. Determining the context vector includes: Determine a second probability, where the second probability corresponds to the attention weight of the self-attention decoder layer.

Images