Artificial intelligence system suitable for streaming applications

Abstract

The present disclosure provides an artificial intelligence (AI) system for sequence-to-sequence modeling with attention suitable for streaming applications. The AI system includes at least one processor and a memory having instructions stored thereon that, when executed by the processor, cause the AI system to process individual input frames in an input frame sequence through layers of a deep neural network (DNN) to generate an output sequence. At least some of the layers of the DNN include a dual self-attention module having a dual non-causal and causal architecture that attends to non-causal frames and causal frames. Further, the AI system renders the output sequence.

Description

Technical Field

[0001] This disclosure generally relates to artificial intelligence (AI), and more specifically, to an AI system for attention-based sequence-to-sequence modeling suitable for streaming applications. Background Technology

[0002] Today, self-attention mechanisms have become a core component in many neural network (NN) architectures used to solve sequence-to-sequence problems for various artificial intelligence (AI) applications. Examples of sequence-to-sequence problems include machine translation, speech processing, language modeling, automatic speech recognition (ASR), computer vision, and more. Self-attention mechanisms allow inputs to interact with each other (“self”) and figure out which they should pay more attention to (“attention”) to optimally compute the output for a given task. Self-attention mechanisms are advantageous for solving sequence-to-sequence problems when the entire input sequence is available for processing. However, waiting for the entire input sequence to be derived from time-dependent signals results in output latency for neural networks that process time series using self-attention modules, making them less suitable for online or streaming sequence-to-sequence processing that requires low latency.

[0003] For example, in speech processing applications, to interpret the utterance "she eats a green apple," a self-attention mechanism might allocate different levels of attention to different parts of the utterance. For instance, when interpreting the word "eats," the word "apple" might be more important than the word "green." Therefore, during the interpretation of "eats," the self-attention module could learn to allocate more attention to the word "apple" than to the word "green." However, to allocate more attention to the word "apple," the self-attention module should wait until the word "apple" becomes available for processing. Thus, in this example, to interpret the beginning of the sentence, the self-attention module needs to wait until the end of the sentence, resulting in a processing delay.

[0004] Restricted self-attention partially addresses this limitation of self-attention mechanisms. While self-attention processes the entire input sequence to interpret its individual inputs, restricted self-attention interprets the current input of the input sequence by looking at a limited field of view preceding the current input. Typically, the input sequence comprises a sequence of input frames; therefore, to interpret the current frame, restricted self-attention considers only a finite number of frames following the current frame. Frames following the current frame are referred to in this paper as lookahead frames or lookahead-only, and the number of lookahead frames is referred to in this paper as the lookahead size.

[0005] Restricted self-attention may degrade the performance of self-attention modules, but it reduces the latency of sequence-to-sequence processing. For example, when the lookahead size is two frames, restricted self-attention delays the interpretation of each current frame by only two frames, which is tolerable for streaming applications.

[0006] However, in practice, self-attention modules form complex architectures within deep neural networks, using multiple self-attention modules within stacked layers, each layer using the output of the preceding layer as input. While the architecture of deep neural networks improves the accuracy of sequence-to-sequence processing, this comes at the cost of increased processing latency. For example, when a restricted self-attention module introduces a two-frame processing latency and the deep neural network includes 12 consecutive self-attention-based layers to improve accuracy, the total latency becomes 24 frames, even though each self-attention layer uses only 2 look-ahead frames. Consequently, deep neural network designers need to balance processing accuracy with latency. They also need to balance the look-ahead size of each self-attention layer against the total latency of the DNN to provide satisfactory sequence-to-sequence processing performance without a large overall latency. Summary of the Invention

[0007] Technical issues

[0008] Some embodiments aim to provide an artificial intelligence (AI) system for sequence-to-sequence processing with self-attention, suitable for streaming applications. Additionally or alternatively, some embodiments aim to provide a dual self-attention module that provides performance guarantees without the large processing latency of sequence-to-sequence processing. Additionally or alternatively, some embodiments aim to provide a dual self-attention module that can be used to form deep neural networks of various architectures without accumulating processing latency across individual layers.

[0009] Solution to the problem

[0010] Some implementations are based on the understanding that self-attention, and restricted self-attention, has a non-causal architecture. The terms causal and non-causal are terms from the field of signal processing, and are used herein in their ordinary sense. For example, in signal processing, a causal filter is a linear time-invariant causal system, where the word "causal" indicates that the filter output depends only on past and present inputs. A filter whose output also depends on future inputs is non-causal. Therefore, as used herein, the non-causal architecture of a self-attention module uses past, present, and future inputs arriving after the current input frame to interpret the current input frame.

[0011] Similarly, restricted self-attention modules can be implemented using a causal architecture. This architecture does not use future information and therefore does not introduce any latency. As a result, any combination of causal self-attention does not lead to any additional latency. However, sequence-to-sequence processing using causal self-attention performs worse than non-causal self-attention. Therefore, some implementations aim to combine causal and non-causal architectures for dataflow applications to take advantage of the strengths of both.

[0012] Some implementations are based on the understanding that non-causal architectures use two main principles. The first principle is the way input frames are focused, and the second principle is the nature of the input frames. In non-causal architectures, both principles are non-causal. Specifically, non-causal self-attention focuses on future look-ahead frames, and therefore the way it focuses is non-causal. Furthermore, the second principle dictates that look-ahead frames should be generated considering future data according to non-causal principles. Therefore, the way look-ahead frames are generated can also be non-causal. As a result, according to these two non-causal principles, a non-causal architecture with a look-ahead size of two frames needs to focus on two future look-ahead frames, which can then be generated using two more future look-ahead frames, and so on, following the depth of the neural network using the self-attention module.

[0013] Some implementations are based on the understanding that the principle of a non-causal architecture can be modified by providing a dual-causal and non-causal (DCN) architecture. In a DCN architecture, the way input frames are considered remains non-causal; however, the input frames themselves can be causal or non-causal. Alternatively, in a DCN architecture, the way input frames are considered can be causal or non-causal, and the input frames can be causal or non-causal. As described herein, a causal input or causal frame is an input or frame generated without considering one or more future inputs, while a non-causal input or non-causal frame is an input or frame generated considering one or more future inputs. As a result, causal inputs within a non-causal architecture of a deep neural network (DNN) can be used to reduce overall processing latency with only a partial reduction in accuracy.

[0014] In some implementations, the lookahead frames for non-causal self-attention processing are causal frames, while the other frames of interest correspond to non-causal frames. When the lookahead frames are causal, no additional lookahead frames are needed in the preceding dual self-attention modules to generate them. It is worth noting that having non-causal past frames does not introduce any additional latency if the lookahead size of a non-causal past frame is no longer than its distance from the current frame. Therefore, implementations can use as many non-causal frames as possible to limit performance degradation, while using only causal lookahead frames to limit processing latency.

[0015] Some implementations are based on the understanding that causal frames (e.g., causal look-ahead frames) need to be generated in a manner consistent with non-causal frames, thereby allowing the DCN architecture to use both causal and non-causal frames together. To this end, some implementations employ dual self-attention, maintaining the causal and non-causal architectures operating in parallel with each other. The causal architecture processes its inputs causally, thereby generating a causal output without look-ahead. The non-causal architecture processes its inputs non-causally, with a look-ahead of a predetermined size relative to the input frame, to output a non-causal output. In some implementations, these look-ahead frames are causal frames copied from the causal architecture.

[0016] Therefore, a dual self-attention module with a DCN architecture can output both causal and non-causal outputs. Both causal and non-causal outputs can propagate through layers of a DNN that include dual self-attention. In this way, a non-causal architecture with dual self-attention can access both causal frames generated by a causal architecture with dual self-attention through previous layers of the DNN and non-causal frames generated by a non-causal architecture with dual self-attention through previous layers of the DNN. It is worth noting that the overall output of the DNN can include one or a combination of causal and non-causal outputs. However, since non-causal outputs are generally more accurate than causal outputs, in one implementation, the output sequence includes only non-causal outputs.

[0017] Some implementations are based on the understanding that, internally, dual self-attention transforms the input frame sequence into a sequence of keywords, values, and query frames. The relationships between queries and keywords, and between keywords and values, are differentiable. In this way, as the DNN learns, the attention mechanism can learn to reshape the relationship between the search input and the inputs that provide context.

[0018] To this end, each query frame in the query sequence searches the keyword sequence to compute the relation of each keyword frame relative to the query frame. Each keyword frame is associated with a value frame that encodes features about each input frame. The computed relation of each keyword frame relative to the query frame is used to assign weighting factors to each value frame for calculating a weighted average of the value frame sequence and the output used for the query search. For example, if each input frame in the input frame sequence corresponds to a word in a word sequence (i.e., a sentence), the estimated relation of each keyword frame relative to the query frame will represent the relationship between the word associated with the query frame and all other words in the sentence.

[0019] Therefore, in order to compute noncausal outputs, the dual self-attention noncausal architecture needs to replicate causal lookahead keyword and value frames, thereby allowing noncausal query frames to focus on noncausal past and current keyword frames as well as causal lookahead keyword frames to generate weights for past and current noncausal value frames and causal lookahead value frames to generate noncausal outputs.

[0020] Some implementations are based on another understanding: the quality of the dual self-attention module can be further improved by minimizing the use of causal frames whenever possible. To this end, in some implementations, the causal architecture uses past and current noncausal frames copied from the noncausal architecture, which does not require any information beyond the current query frame position, while the noncausal architecture uses causal look-ahead frames copied from the causal architecture.

[0021] Some implementations are based on the understanding that a DNN with layers having dual self-attention modules can serve as the primary interpreter for sequence-to-sequence problems and / or form part of a more complex neural network architecture. For example, in one implementation, the DNN is trained with a connectionist temporal classification (CTC) objective to output an interpretation of audio input. For example, in automatic speech recognition (ASR) applications, the DNN is trained to output a transcription of spoken utterances. Alternatively or additionally, the DNN can be trained as an encoder to output encoder states that can later be transformed into the desired form of the output. In one example, this later transformation can be accomplished by means of an additional projection layer trained together with the DNN. In another example, the DNN forms a dual self-attention encoder in an encoder / decoder architecture, where the decoder is also a deep neural network.

[0022] For example, in one implementation, the encoder neural network and the additional projection layer are trained with a CTC objective function. This implementation is advantageous because CTC-based neural networks can also be applied to streaming applications. In another example, the encoder is followed by an attention-based decoder neural network. This implementation is less convenient for streaming applications but provides more accurate interpretation results than CTC-based neural networks.

[0023] In another embodiment, the output of the dual self-attention encoder is processed by multiple different decoders that work together to achieve better results. For example, the output of the dual self-attention encoder is processed by both CTC-based and attention-based decoders to improve overall decoding. Alternatively or additionally, in one embodiment, the CTC-based output triggers the execution of the attention-based decoder to adapt it to streaming applications. To this end, some embodiments provide a speech recognition system that includes a dual self-attention encoder combined with an attention-based decoder.

[0024] Therefore, one embodiment discloses an artificial intelligence (AI) system suitable for data stream applications. The AI ​​system includes at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the AI ​​system to process individual input frames in an input frame sequence through layers of a deep neural network (DNN) to generate an output sequence, wherein at least some layers include dual self-attention modules with a bicausal and noncausal architecture that focuses on both noncausal and causal frames; and render the output sequence.

[0025] Therefore, another embodiment discloses an audio processing method. This audio processing method includes the following steps: processing individual input frames in an input frame sequence through layers of a deep neural network (DNN) to generate an output sequence, wherein at least some layers include a dual self-attention module with a bicausal and non-causal architecture that focuses on both non-causal and causal frames; and rendering the output sequence.

[0026] Therefore, another embodiment discloses a non-transitory computer-readable storage medium embodied thereon having a program executable by a processor to perform a method comprising the steps of: processing individual input frames in an input frame sequence through layers of a deep neural network (DNN) to generate an output sequence, wherein at least some layers include a dual self-attention module having a dual causal and non-causal architecture that focuses on both non-causal and causal frames; and rendering the output sequence.

[0027] The currently disclosed embodiments will be further described with reference to the accompanying drawings. The drawings shown are not necessarily to scale, but rather focus on illustrating the principles of the currently disclosed embodiments. Attached Figure Description

[0028] [ Figure 1 ] Figure 1 This is a block diagram illustrating a network environment for implementing an artificial intelligence (AI) system according to some embodiments of the present disclosure.

[0029] [ Figure 2A ] Figure 2A This is based on some embodiments of the present disclosure. Figure 1 A block diagram of an AI system is shown as an example.

[0030] [ Figure 2B ] Figure 2B A graphical representation of a deep neural network according to some embodiments of the present disclosure is shown.

[0031] [ Figure 3 ] Figure 3 A schematic diagram of a dual-causal and non-causal architecture according to some embodiments of the present disclosure is shown.

[0032] [ Figure 4 ] Figure 4 This diagram illustrates the weighting of individual value frames in a value frame sequence according to some embodiments of the present disclosure.

[0033] [ Figure 5 ] Figure 5 A schematic diagram is shown of a speech recognition system (ASR) configured for end-to-end speech recognition according to some implementations.

[0034] [ Figure 6A ] Figure 6A A schematic diagram of an alignment decoder according to some embodiments is shown.

[0035] [ Figure 6B ] Figure 6B An example of partitioning an encoder state sequence according to some implementations is shown.

[0036] [ Figure 6C ] Figure 6C An example of partitioning an encoder state sequence according to some implementations is shown.

[0037] [ Figure 7 ] Figure 7 An example of an attention-based decoder according to some implementations is shown.

[0038] [ Figure 8 ] Figure 8 A schematic diagram of a flow system based on some embodiments of an AI system according to this disclosure is shown.

[0039] [ Figure 9 ] Figure 9 A general block diagram of an AI system according to some example embodiments of the present disclosure is shown. Detailed Implementation

[0040] In the following description, numerous specific details are set forth for illustrative purposes in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these specific details. In other instances, apparatuses and methods are shown only as block diagrams to avoid obscuring this disclosure.

[0041] As used in this specification and claims, the terms "for example" and "such as," as well as the verbs "comprising," "having," "including," and other verb forms thereof, when used in conjunction with a list of one or more components or other items, shall each be interpreted as open-ended, meaning that the list should not be considered as excluding other additional components or items. The term "based on" means at least partially based on. Furthermore, it will be understood that the wording and terminology used herein are for descriptive purposes and should not be considered limiting. Any headings used within this description are for convenience only and have no legal or limiting effect.

[0042] Figure 1 This is a block diagram illustrating a network environment 100 for implementing an artificial intelligence (AI) system 102 according to some embodiments of the present disclosure. The network environment 100 is depicted as including a user 106 associated with a user device 108. In an exemplary scenario, the user 106 provides input to the user device 108, such as audio input 110. The user device 108 may receive the audio input 110 as an acoustic signal or as speech. The user device 108 may include applications hosted by a server 104, such as automatic speech recognition (ASR). The audio input 110 may be provided to the server 104 via a network 112. The server 104 may be configured to process the audio input 110 to perform various ASR-related tasks for data streaming applications.

[0043] Furthermore, network 112 may include suitable logic, circuitry, and interfaces configured to provide multiple network ports and multiple communication channels for sending and receiving data. Each network port may correspond to a virtual address (or physical machine address) for sending and receiving communication data. For example, the virtual address may be Internet Protocol version 4 (IPv4) (or an IPv6 address), and the physical address may be a Media Access Control (MAC) address. Network 112 may be associated with an application layer to implement communication protocols based on one or more communication requests from at least one of one or more communication devices. Communication data may be sent or received via the communication protocols. Examples of such wired and wireless communication protocols may include (but are not limited to) Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, Infrared (IR), IEEE 802.11, 802.16, cellular communication protocols, and / or Bluetooth (BT) communication protocols.

[0044] Examples of network 112 may include (but are not limited to) wireless channels, wired channels, and combinations thereof. Wireless or wired channels may be associated with network standards, which may be defined by one of the following: Local Area Network (LAN), Personal Area Network (PAN), Wireless Local Area Network (WLAN), Wireless Local Area Network (WAN), Wireless Wide Area Network (WWAN), Long Term Evolution (LTE) network, Common Old-Style Telephone Service (POTS), and Metropolitan Area Network (MAN). Additionally, wired channels may be selected based on bandwidth standards. For example, fiber optic channels may be used for high-bandwidth communication. Furthermore, coaxial cable-based or Ethernet-based communication channels may be used for medium-bandwidth communication.

[0045] In some implementations, the audio input 110 may be very long. In such cases, the computational complexity of the server 104 may be very high. Therefore, the server 104 may not be able to process the audio input 110 in a timely manner, which may result in delayed output. In addition, since processing a long audio input 110 may take time, this leads to a delayed response to user input. To address this, some implementations disclose an AI system 102 that can generate accurate output with less latency.

[0046] Figure 2A This is based on some embodiments of the present disclosure. Figure 1 A block diagram of an AI system 102 is illustrated below. The AI ​​system 102 is suitable for data-streaming applications. The AI ​​system 102 includes a processor 202, a memory 204, and an input / output (I / O) interface 210. In an example embodiment, the I / O interface 210 is configured to receive an input sequence, which may correspond to an audio input (e.g., audio input 110) having a time dimension. Furthermore, the processor 202 is configured to execute instructions stored in the memory 204. Executing the stored instructions causes the AI ​​system 102 to accept a sequence of input frames. The sequence of input frames represents an ordered sequence of features describing information about an input signal (e.g., audio input 110).

[0047] Some implementations are based on the understanding that self-attention, and restricted self-attention, has a non-causal architecture. The terms causal and non-causal are terms from the field of signal processing, and are used herein in their ordinary sense. For example, in signal processing, a causal filter is a linear time-invariant causal system, where the term "causal" indicates that the filter output depends only on past and present inputs. A filter whose output also depends on future inputs is non-causal. Therefore, as used herein, the non-causal architecture of a self-attention module uses past, present, and future inputs arriving after the current input frame to interpret the current input frame.

[0048] Similarly, restricted self-attention modules can be implemented using a causal architecture. This architecture does not use future information and therefore does not introduce any latency. As a result, any combination of causal self-attention modules does not lead to any additional latency. However, sequence-to-sequence processing using causal self-attention performs worse than non-causal self-attention. Therefore, some implementations aim to combine causal and non-causal architectures for dataflow applications to take advantage of the strengths of both.

[0049] Some implementations are based on the understanding that noncausal architectures use two main principles. The first principle is the way input frames are focused, and the second principle is the nature of the input frames. In noncausal architectures, both principles are noncausal. Specifically, noncausal self-attention focuses on future look-ahead frames, so the way it focuses is noncausal. Furthermore, the second principle dictates that look-ahead frames should be generated considering future data according to noncausal principles. Therefore, the way look-ahead frames are generated is also noncausal. As a result, a noncausal architecture with a look-ahead size of two frames needs to focus on two future look-ahead frames, which can then be generated using two more future look-ahead frames, and so on, following the depth of the neural network using the self-attention module.

[0050] Some implementations are based on the understanding that the principle of a noncausal architecture can be modified by providing a dual-causal and noncausal (DCN) architecture. In a DCN architecture, the way input frames are considered remains noncausal; however, the input frames themselves can be causal or noncausal. Alternatively, in a DCN architecture, the way input frames are considered can be causal or noncausal, and the input frames can be either causal or noncausal. As described herein, a causal input or causal frame is an input or frame generated without considering one or more future inputs, while a noncausal input or noncausal frame is an input or frame generated considering one or more future inputs. As a result, causal inputs within the noncausal architecture of a deep neural network can be used to reduce overall processing latency, with only a partial reduction in accuracy.

[0051] Therefore, in this embodiment, memory 204 includes a deep neural network (DNN) 206. The DNN 206 may include layers, at least some of which include dual self-attention modules 208 with a DCN architecture.

[0052] Figure 2B A schematic representation of a DNN 206 according to some embodiments of the present disclosure is shown. The DNN 206 includes layers 206a-206e. Some layers 206a-206e include dual self-attention modules 208. For example, the DNN 206 includes an input layer 206a, an output layer 206e, and dual self-attention layers 206b, 206c, and 206d. Self-attention layers 206b, 206c, and 206d include a DCN architecture. A processor 202 is configured to process individual input frames in an input frame sequence through the layers of the DNN 206 to generate an output sequence.

[0053] Figure 3A schematic diagram of a DCN architecture 300 for generating output sequences according to some embodiments of the present disclosure is shown. Processor 202 copies the input feature sequence to derive a causal input frame sequence 302 and a non-causal input frame sequence 304. Furthermore, the DCN architecture 300 transforms the causal and non-causal input frames into causal and non-causal keywords, values, and query frames. The DCN architecture 300 includes both causal and non-causal architectures. The DCN architecture 300 is configured to perform dual self-attention, maintaining the causal and non-causal architectures operating in parallel. According to embodiments, dual self-attention includes causal self-attention processing 306 and non-causal self-attention processing 308. Causal self-attention processing 306 and non-causal self-attention processing 308 are executed in parallel.

[0054] The causal input frame sequence 302 is fed as input to the causal self-attention processing 306. The causal self-attention processing 306 includes processing each causal input frame of the causal input frame sequence 302 in a causal manner (i.e., without look-ahead frames) through a causal architecture to generate a causal output frame sequence 310.

[0055] Furthermore, the non-causal input frame sequence 304 is fed as input to the non-causal self-attention processing 308. The non-causal self-attention processing 308 includes look-ahead processing of each non-causal input frame of the non-causal input frame sequence 304 in a non-causal manner relative to the input frame representing the current time by a predetermined size using a non-causal architecture, to generate a non-causal output frame sequence 312. In the non-causal self-attention processing 308, the look-ahead frames are causal frames, while the other frames of interest correspond to non-causal frames. For example, to interpret the current input frame 314, look-ahead frame 318, which is a causal frame, and other frames 316, which are non-causal frames, are considered. When the look-ahead frames are causal, no additional look-ahead frames are needed in the preceding dual self-attention modules to generate them. It is worth noting that having non-causal past frames does not introduce any additional latency. Therefore, implementations can use as many non-causal frames as possible to limit performance degradation, while using only causal look-ahead frames to limit processing latency.

[0056] Therefore, the DCN architecture 300 of the dual self-attention module 206 outputs a causal output frame sequence 310 and a non-causal output frame sequence 312. Furthermore, the causal output frame sequence 310 and the non-causal output frame sequence 312 can be propagated through layers of the DNN 206, which includes dual self-attention. In this way, the non-causal architecture with dual self-attention can access both the causal frames generated by the causal architecture with dual self-attention through previous layers of the DNN 206 and the non-causal frames generated by the non-causal architecture with dual self-attention through previous layers of the DNN 206.

[0057] According to the implementation, the causal output frame sequence 310 and the non-causal output frame sequence 312 correspond to the output sequence. However, since non-causal output frames are generally more accurate than causal output frames, in one implementation, the output sequence includes only the non-causal output frame sequence 312.

[0058] Some implementations are based on the understanding that, internally, dual self-attention transforms the input frame sequence into a sequence of keywords, values, and query frames. The relationships between queries and keywords, and between keywords and values, are differentiable. In this way, as the DNN206 learns, the attention mechanism can learn to reshape the relationship between the search input and the input that provides context.

[0059] To this end, each query frame in the query sequence searches the keyword sequence to compute the relation of each keyword frame relative to the query frame. Each keyword frame is associated with a value frame that encodes features about each input frame. The computed relation of each keyword frame relative to the query frame is used to assign weighting factors to each value frame for calculating a weighted average of the value frame sequence and the output used for the query search. For example, if each input frame in the input frame sequence corresponds to a word in a word sequence (i.e., a sentence), the estimated relation of each keyword frame relative to the query frame can represent the relationship between the word associated with the query frame and all other words in the sentence.

[0060] Figure 4 A schematic diagram illustrating the weighting of individual value frames in a sequence of value frames according to some embodiments of the present disclosure is shown. Input causal frames 400 are fed into a dual-self-attention causal architecture of the current layer of the DNN 206. According to embodiments, the input causal frames 400 may correspond to a causal output determined by a dual-self-attention causal architecture of a previous layer of the DNN 206. The dual-self-attention causal architecture of the current layer of the DNN 206 transforms the previous causal output into a sequence of causal keyword frames, causal value frames, and causal query frames to generate the current causal output 414 by weighting the causal value frames by making the causal query frames focus on the causal keyword frames.

[0061] The input noncausal frame 416 is fed into the dual-self-attention noncausal architecture of the current layer of the DNN 206. According to an implementation, the input noncausal frame 416 may correspond to the noncausal output determined by the dual-self-attention noncausal architecture of the previous layer of the DNN 206. The dual-self-attention noncausal architecture of the current layer of the DNN 206 transforms the previous noncausal output into a sequence of noncausal keyword frames, noncausal value frames, and noncausal query frames to generate the current noncausal output 430. Specifically, in an implementation, the current noncausal output 430 is generated by causing the noncausal query frame 422 to focus on the noncausal keyword frame 418 combined with the causal look-ahead keyword frame 420 copied from the causal architecture, to weight the noncausal value frame 424 combined with the causal look-ahead value frame 426 copied from the causal architecture. The weights estimated for the noncausal value frame 424 combined with the causal look-ahead value frame 426 are represented by 428.

[0062] Some implementations are based on another understanding: the quality of the dual self-attention module 208 can be further improved by minimizing the use of causal frames whenever possible. To this end, in some implementations, the causal architecture uses past non-causal frames copied from the non-causal architecture, which does not require any information beyond the current query frame position, while the non-causal architecture uses causal look-ahead frames copied from the causal architecture.

[0063] For example, the causal architecture generates the current causal output 414 by having the causal query frame 406 focus on the causal key frame 404 combined with the non-causal key frame 402 copied from the non-causal architecture, to weight the causal value frame 410 combined with the non-causal value frame 408 copied from the non-causal architecture. The weight estimated for the causal value frame 410 combined with the non-causal value frame 408 is represented by 412. Alternatively, the non-causal architecture generates the current non-causal output 430 by having the non-causal query frame 422 focus on the past and current non-causal key frames 418 combined with the causal look-ahead key frame 420 copied from the causal architecture, to weight the non-causal past and current value frames 424 combined with the causal look-ahead value frame 426 copied from the causal architecture.

[0064] Some implementations are based on the understanding that a DNN 206 with dual self-attention module layers can be used as the primary interpreter for sequence-to-sequence problems and / or can form part of a more complex neural network architecture.

[0065] For example, in one implementation, DNN 206 is trained with a connectionist temporal classification (CTC) objective to output an interpretation of audio input. For example, for ASR applications, DNN 206 is trained to output a transcription of spoken utterances. Alternatively or additionally, DNN 206 may be trained as an encoder to output encoder states that can later be transformed into the desired form of the output. In one example, this later transformation can be accomplished by means of an additional projection layer trained together with DNN 206. In another example, DNN 206 forms a dual self-attention encoder in an encoder / decoder architecture, where the decoder is also a deep neural network.

[0066] For example, in one implementation, the encoder neural network and the additional projection layer are trained with a CTC objective function. This implementation is advantageous because CTC-based neural networks can also be applied to streaming applications. In another example, the encoder is followed by an attention-based decoder neural network. This implementation is less readily applicable to streaming applications but provides more accurate interpretation results than CTC-based neural networks.

[0067] In another embodiment, the output of the dual self-attention encoder is processed by multiple different decoders that work together to achieve better results. For example, the output of the dual self-attention encoder is processed by both CTC-based and attention-based decoders to improve overall decoding. Alternatively or additionally, in one embodiment, the CTC-based output triggers the execution of the attention-based decoder to make the attention-based decoder suitable for streaming applications. To this end, some embodiments provide a speech recognition system that includes a dual self-attention encoder combined with an attention-based decoder.

[0068] Figure 5 A schematic diagram of a speech recognition system (ASR) 500 configured for end-to-end speech recognition according to some embodiments is shown. The speech recognition system 500 acquires an input acoustic sequence and processes it to generate a transcribed output sequence. Each transcribed output sequence is a transcription of a utterance or a portion of a utterance represented by a corresponding input acoustic signal. For example, the speech recognition system 500 may acquire an input acoustic signal 502 and generate a corresponding transcribed output 510 as a transcription of the utterance represented by the input acoustic signal 502.

[0069] The input acoustic signal 502 may include a sequence of multiple audio data frames, such as a continuous data stream, representing a digital representation of speech. The sequence of multiple audio data frames may correspond to a time-step sequence, for example, where each audio data frame is associated with a 25-millisecond audio stream data that has been shifted 10 milliseconds further in time from a previous audio data frame. Each audio data frame in the sequence may include feature values ​​characterizing the portion of the speech corresponding to that time step. For example, the sequence of multiple audio data frames may include a filter bank spectral feature vector.

[0070] Transcription output 510 may include a sequence of transcribed fragments of speech represented by input acoustic signal 502. Transcription output may include one or more characters. For example, transcription output may be characters or sequences of characters from a Unicode character set. For example, the character set may include the alphabets of English, Asian languages, Cyrillic, and Arabic. The character set may also include Arabic numerals, space characters, and punctuation marks. Alternatively or additionally, transcription output may include bits, words, and other linguistic structures.

[0071] The speech recognition system 500 includes an encoder 504 and an attention-based decoder 508. The encoder 504 may be a dual self-attention encoder. The encoder 504 processes the input acoustic signal 502 and generates an encoder state sequence 506 that provides an alternative (e.g., higher) representation of the input acoustic signal 502. The encoder state sequence 506 may include an alternative sequence of multiple audio data frames corresponding to a second set of time steps. The attention-based decoder 508 is trained to process the encoder state sequence 506 representing the alternative representation of the input acoustic signal 502 and generates a transcription output 510 from the encoder state sequence provided to the attention-based decoder 508.

[0072] Some implementations are based on the understanding that attention-based ASR systems may need to observe the entire speech utterance segmented by speech pauses in order to assign weights to each input frame in order to identify each transcribed output 510. Due to the lack of prior knowledge about which part of the input acoustic signal is relevant to identifying the next transcribed output and the need to assign weights to each encoder state, attention-based decoders typically need to process large input sequences. This processing allows for attention to be given to different parts of the utterance, but it also increases output latency, making it impractical for streaming / online speech recognition.

[0073] As used in this paper, the output latency of ASR is the difference between the time it takes to receive the acoustic frames of a speech utterance and the time it takes to recognize the received acoustic frames. For example, when an attention-based ASR system operates over the entire speech utterance, the recognition of words in the utterance is delayed until the last word of the utterance is received. This recognition delay leads to an increase in output latency.

[0074] Some implementations are based on the understanding that prior knowledge about the relevance of different parts of the input sequence to the recognition of the next transcribed output is exemplified by indications of the frame positions corresponding to the transcribed output to be recognized within the input sequence. In practice, if the transcribed output positions are known, by restricting the input sequence, an attention-based decoder can be forced to devote more attention to those positions and less or no attention to others. In this way, for each transcribed output, the attention-based network can focus its attention around its position within the input sequence. This guided attention reduces the need to process large input sequences, which in turn reduces output latency, making attention-based decoders practical for recognition in streaming / online modes.

[0075] To this end, ASR system 500 includes an alignment decoder 520, which is trained to determine the position 525 of the encoder state in sequence 506 that encodes transcribed outputs such as characters, bits, words, etc. CTC is an objective function and associated neural network output used to train recurrent neural networks (RNNs) such as Long Short-Term Memory (LSTM) networks to solve timing-variable sequence problems. CTC-based ASR systems are an alternative to attention-based ASR systems. CTC-based neural networks generate outputs for individual frames of the input sequence (i.e., input and output are synchronized), and a beam search algorithm is used to find the optimal output sequence before folding the neural network output into the output transcription. Attention-based ASR systems may outperform CTC-based ASR systems. However, some implementations are based on the understanding that attention-based ASR systems can overcome their aforementioned output delay drawbacks by using input and output frame alignment used in the intermediate operations of CTC-based ASR systems.

[0076] To utilize the alignment information 525 provided by the alignment decoder 520, the ASR system 500 includes a partitioning module 530 configured to partition the encoder state sequence 506 into a set of partitions 535. For example, the partitioning module 530 can segment the encoder state sequence for each position 525 of the identified encoder states, such that the number of partitions 535 is defined by (e.g., equal to) the number of identified encoder states 525 encoding the transcription output. In this way, the attention-based decoder 508 does not accept the entire sequence 506, but rather portions 535 of the sequence as input, and these portions may include new transcription outputs to form the transcription output sequence 510. In some implementations, the combination of the alignment decoder, the attention-based decoder, and the partitioning module is referred to as a triggered attention decoder. In practice, the triggered attention decoder can process utterance portions upon receipt, making the ASR system 500 useful for recognition in streaming / online modes.

[0077] Figure 6A A schematic diagram of an alignment decoder 520 according to some embodiments is shown. One goal of the alignment decoder 520 is to decode the encoder state sequence 506 generated by the encoder 504. To this end, the alignment decoder 520 is trained to decode the encoder state sequence 506 to generate a transcription output sequence 600. However, at least some embodiments do not use the transcription output 600 decoded by the alignment decoder 520. Instead, some embodiments use intermediate alignment information generated by the alignment decoder 520 to decode the encoder state sequence 506. In other words, some embodiments ignore the transcription output 600 decoded by the alignment decoder 520 and instead use the position 525 of the encoder state in sequence 506 to improve the performance of the attention-based decoder 508. The theoretical basis behind this approach is that the performance of the attention-based decoder 508 may be better than that of the alignment decoder 520. Therefore, the intermediate alignment information generated by the alignment decoder 520 is used to further improve the performance of the attention-based decoder 508.

[0078] However, in some embodiments, the transcribed output 600 decoded by the alignment decoder 520 is further combined with the transcribed output decoded by the attention-based decoder 508 to further improve recognition accuracy. In these embodiments, the alignment decoder 520 is utilized twice: firstly to assist in partitioning the encoder state sequence 506 for use by the attention-based decoder 508, and secondly to further improve the accuracy of the transcribed output decoded by the attention-based decoder 508.

[0079] Figure 6A This illustrates an example of how the alignment decoder 520 processes a portion of the utterance containing the word "dog". Boxes surrounding elements of the index sequence identify the locations of encoder states 525 that encode the transcriptional output in encoder state sequence 506. For example, encoder 504 encodes an input acoustic sequence with acoustic features such as log-Mel spectrum energy. Convert to Encoder state sequence : .

[0080] For example, in one implementation, the feature matrix is ​​processed by an encoder neural network. X Downsampling to a frame rate four times lower makes the feature matrix... When the frame rate reaches 100 Hz, the encoder output H It has a frame rate of 25 Hz. Let... This indicates the length calculated from encoder state H 506. The frame-by-frame sequence output by CTC. ,in A set representing different morphemes, which can be, for example, a single character or a fragment of a word. Represents a blank symbol. Let... Indicates length The sequence of character elements, where This allows the sequence to be collapsed into single occurrences and whitespace removed when repeating labels are folded into single occurrences. Reduce to .

[0081] In some implementations, the alignment decoder 520 performs probabilistic decoding on the encoder state sequence, wherein the probability is derived as follows:

[0082] in Represents the transition probability. Represents the acoustic model.

[0083] In some implementations, the alignment decoder 520 will be with The frame with the highest probability within each subsequence of frames corresponding to the same character in the sequence is identified as the recognized CTC output. For example, if for The first in Label The beginning and end of an occurrence are indicated by an index. and To write the sequence ,in And for all , This makes it possible for all other indexes, and The alignment decoder 520 performs the alignment from the sequence. Up to the same length of a subset including encoder state 525 that encodes the transcriptional output with the highest probability. sequence ,in This indicates zero or more repetitions, and each of them... In The corresponding frames with the highest probability of occurrence appear exactly once, as follows:

[0084] Alternatively, the alignment decoder 520 can be coupled with... The first or last frame within each subsequence of the same character in the frame is identified as the encoder state.

[0085] Figure 6Band Figure 6C An example of partitioning the encoder state sequence 506 according to some embodiments is shown. In various embodiments, partitioning is performed by a partitioning module 530 operatively connected to the alignment decoder 520, the attention-based decoder 508, and the encoder 504. The partitioning module 530 is configured to access alignment information 525 of the alignment decoder 520, partition the encoder state sequence generated by the encoder 504, and sequentially submit portions 535 of the encoder state sequence to the attention-based decoder 508.

[0086] For example, in Figure 6B In one implementation, each partition 602a includes encoder states from the beginning of the encoder state sequence up to the look-ahead encoder state determined by shifting the position of the identified encoder state forward by a fixed shift. For example, if the fixed shift value is 5 and the position of the identified state encoder is the 8th in the encoder state sequence, then partition 602a includes the first 13 encoder states. If the position of the subsequently identified encoder state is 11, then partition 602a includes the first 16 encoder states. In effect, each partition includes encoder states for the new transcribed output, thus increasing the partition length and allowing the attention-based decoder 508 to utilize its length.

[0087] exist Figure 6C In an alternative implementation, partition 602b corresponding to the location of the identified encoder state includes a predetermined number of encoder states centered on the location of the identified encoder state. For example, if the predetermined number of encoder states is 7 and the location of the identified encoder state is the 15th in the encoder state sequence, then partition 602b includes encoder states between the 12th and 18th in the encoder state sequence 506. In effect, while having partitions of fixed length, each partition includes encoder states for new transcribed outputs to reduce the computational burden on the attention-based decoder 508. Additionally or alternatively, partition 602b corresponding to the location of the identified encoder state includes a predetermined number of encoder states surrounding the location of the identified encoder state (e.g., offset from the center to provide off-center coverage).

[0088] Figure 7 An example attention-based decoder 508 according to some implementations is shown. The attention-based decoder 508 includes a context vector generator 704 and a decoder neural network 706. The context vector generator 704 receives the hidden decoder state 712 from the decoder neural network 706 at the previous time step, the attention weight distribution 710 from the context vector generator at the previous time step, and the alternative representation 506 (i.e., as shown above). Figure 5(The alternative representation of the described acoustic signal 502). The context vector generator 704 processes the previously hidden decoder state, the previously set attention weight distribution 710, and the alternative representation 506 of the decoder neural network 706 to compute the attention weight distribution over time frames of the alternative representation 506 and generate a time-step context vector 714 as output. The context vector generator 704 provides the time-step context vector 714 to the decoder neural network 706.

[0089] For different iterations, the attention-based decoder 508 receives different partitions 731, 733, and 735. For example, the set of partitions includes a first partition 731 and subsequent partitions 733 and 735. The attention-based decoder 508 processes the first partition 731 to generate a first transcribed output. After the attention-based neural network has completed processing the first partition, placing the attention-based network in its internal state, the attention-based decoder 508 processes subsequent partitions with the attention-based network without resetting the internal state of the attention-based network, to generate transcribed outputs for each subsequent partition.

[0090] In practice, the attention-based decoder 508 processes different partitions without resetting the internal state of the attention-based network, in order to utilize previously decoded information. Upon determining the end of a speech utterance, the attention-based decoder 508 is configured to reset its internal state.

[0091] The decoder neural network 706 receives the context vector 714 of the current time step, the transcription output 708 from the previous time step, and the hidden decoder state 712 as input. Before processing the context vector 714 and the transcription output 708 from the previous time step, the decoder neural network 706 initializes its internal hidden state using the previous hidden decoder state 712 to generate a set of transcription output scores 716 for the current time step as output. In some implementations, the decoder neural network 706 is a recurrent neural network (RNN) with a softmax output layer. Each transcription output score corresponds to a specific transcription output from the set of transcription outputs. For example, as referenced above. Figure 5 The transcription output set described herein can be characters or character sequences from a Unicode character set used for writing one or more natural languages, such as the alphabets of English, Asian languages, Cyrillic, and Arabic. The transcription output set may also include Arabic numerals, space characters, and punctuation marks. A score for a given transcription output indicates the probability that the corresponding transcription output is the current transcribed segment in the output sequence as a transcription of a discourse at that time step.

[0092] The speech recognition system processes the transcription output scores 716 at each time step to determine the transcription output sequence representing the utterance. For example, for each time step, the speech recognition system may select the transcription output with the highest score from the set of transcription output scores to determine the transcription output sequence.

[0093] Figure 8 A schematic diagram of a streaming system 800 based on an AI system 102 according to some embodiments of the present disclosure is shown. The AI ​​system 102 captures speech data (or audio data) from a user 802 speaking during a live event. Additionally, the AI ​​system 102 may also capture video data of the speaking user 802. The speech data may include speech utterances. The speech utterances comprise a sequence of multiple audio data frames. The AI ​​system 102 processes each audio data frame in the sequence of multiple audio data frames in real time through layers of a DNN 206 to generate an output sequence 804 for streaming and other live applications. The output sequence 804 may correspond to a transcription of the speech utterances.

[0094] For example, AI system 102 can be communicatively connected to user device 806. AI system 102 can send video data and voice speech data along with the transcription of the voice speech for use in live event streaming. In other words, during live event streaming, AI system 102 provides captions corresponding to the user's voice speech occurring during the live event.

[0095] In some implementations, AI system 102 or its functionality can be integrated into multiple different devices and computer systems, including, for example, general-purpose computing systems, server-client computing systems, mainframe computing systems, telephony computing systems, laptop computers, cellular phones, personal digital assistants (PDAs), tablet computers, and other mobile devices. AI system 102 can also be a component of other devices or systems that provide voice recognition capabilities, such as automated teller machines (ATMs), kiosks, home appliances (e.g., refrigerators, ovens), and vehicles (e.g., cars, buses, motorcycles) for various voice recognition applications.

[0096] In some cases, AI system 102 can also provide users with speech-to-text documentation. For example, a user can provide speech to a user device. The user device can communicate with AI system 102 to provide a transcription of the speech. AI system 102 can then provide a text document based on the transcription. This can help users (e.g., user 902) write text or maintain documents via voice input.

[0097] Figure 9 A block diagram of an AI system 900 according to some embodiments of the present disclosure is shown. The AI ​​system 900 corresponds to... Figure 1AI system 102. AI system 900 may have multiple interfaces for connecting AI system 900 to other systems and devices. For example, network interface controller (NIC) 916 is adapted to connect AI system 900 to network 922 via bus 914, network 922 connecting AI system 900 to sensors. Through network 922 (whether wirelessly or wired), AI system 900 receives input data 924 including a sequence of input frames. Additionally or alternatively, input data 924 including a sequence of input frames may be received via input interface 902.

[0098] AI system 900 includes a processor 904 configured to execute stored instructions and a memory 912 storing instructions executable by processor 904. Processor 904 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. Memory 912 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. Processor 904 is connected to one or more input and output devices via bus 914. Furthermore, AI system 900 includes a storage device 906 adapted to store different modules including instructions executable by processor 904. Storage device 906 may be implemented using a hard disk drive, optical disk drive, thumb drive, drive array, or any combination thereof.

[0099] Storage device 906 is configured to store deep neural network 908. Deep neural network 908 includes multiple layers. In one embodiment, some of the multiple layers include a dual self-attention module 910 with a dual causal and non-causal architecture that focuses on both non-causal and causal frames. Processor 904 is configured to process individual input frames in an input frame sequence through the layers of deep neural network 908 to generate an output sequence and render the output sequence.

[0100] According to one embodiment, the dual self-attention module 910 includes dual self-attention, outputting a causal output frame sequence derived without look-ahead and a non-causal output frame sequence derived with look-ahead relative to an input frame representing the current time by a predetermined size. Therefore, in one embodiment, the output sequence includes one or a combination of a causal output frame sequence and a non-causal output frame sequence. In an alternative embodiment, the output sequence includes only a non-causal output frame sequence.

[0101] Additionally, the AI ​​system 900 includes an output interface 918. The output interface 918 is configured to submit an output sequence to an output device.

[0102] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with a feasible description for implementing one or more exemplary embodiments. Various changes to the function and arrangement of the elements will be conceived without departing from the spirit and scope of the subject matter set forth in the appended claims.

[0103] Specific details are set forth in the following description to provide a thorough understanding of the embodiments. However, it will be understood by those skilled in the art that embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Furthermore, similar reference numerals and designations in the various figures indicate similar elements.

[0104] Furthermore, the various embodiments can be described as processes, depicted as flowcharts, data flow diagrams, structural diagrams, or block diagrams. Although flowcharts can describe operations as a sequential process, many operations can be performed in parallel or simultaneously. Additionally, the order of operations can be rearranged. A process may terminate upon completion of its operations, but may have additional steps not discussed or included in the figures. Moreover, not all operations in any specifically described process may appear in all embodiments. A process may correspond to a method, function, program, subroutine, subroutines, etc. When a process corresponds to a function, the termination of the function may correspond to the function returning to the calling function or the main function.

[0105] Furthermore, implementations of the disclosed subject matter can be carried out, at least partially, manually or automatically. They can be performed, or at least assisted by, using machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, for manual or automatic implementation. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the required tasks can be stored in a machine-readable medium. The processor can then execute the required tasks.

[0106] Furthermore, the various methods or processes outlined herein can be encoded as software that can be executed on one or more processors employing any of a variety of operating systems or platforms. Additionally, such software can be written using any of a variety of suitable programming languages ​​and / or programming or scripting tools, and can also be compiled into executable machine language code or intermediate code that executes on a framework or virtual machine. Typically, in various implementations, the functionality of program modules can be combined or distributed as needed.

[0107] Implementations of this disclosure can be specifically embodied as a method, examples of which have been provided. Actions performed as part of this method can be ordered in any suitable manner. Therefore, implementations can be constructed that perform actions in a different order than those shown, which may include performing some actions simultaneously, although they are shown as sequential actions in the illustrative embodiments.

[0108] Although this disclosure has been described with reference to specific preferred embodiments, it will be understood that various other adjustments and modifications may be made within the spirit and scope of this disclosure. Therefore, aspects of the appended claims cover all such variations and modifications that fall within the true spirit and scope of this disclosure.

Claims

1. An artificial intelligence (AI) system suitable for data flow applications, the AI ​​system comprising: At least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the AI ​​system to: The input frame sequence is processed by layers of a deep neural network (DNN) to generate an output sequence, wherein the input frame sequence is an ordered feature sequence describing information of the audio input, wherein the audio input is an acoustic signal or speech, and wherein at least some of the layers include dual self-attention modules with a bicausal and noncausal architecture that focuses on both noncausal and causal frames; and Render the output sequence; Here, the causal frame that is of interest in the current frame is the look-ahead frame, while the other frames of interest are the non-causal frames. The dual self-attention module includes dual self-attention, which outputs a causal output frame sequence derived without look-ahead and a non-causal output frame sequence derived with look-ahead relative to an input frame representing the current time by a predetermined size; and wherein the output sequence includes one or a combination of the causal output frame sequence and the non-causal output frame sequence. The causal architecture of the current layer of the DNN with dual self-attention transforms the previous causal output determined by the causal architecture of the previous layer of the DNN with dual self-attention into a sequence of causal keyword frames, causal value frames and causal query frames. The non-causal architecture of the current layer of the DNN transforms the previous non-causal output determined by the non-causal architecture of the previous layer of the DNN into a sequence of non-causal keyword frames, non-causal value frames and non-causal query frames, so as to generate the current non-causal output by focusing on the non-causal query frames. The causal architecture generates the current causal output by weighting the causal value frames combined with non-causal value frames copied from the non-causal architecture, with the causal query frames focusing on causal keyword frames. The non-causal architecture generates the current non-causal output by having the non-causal query frame focus on past and current non-causal keyword frames combined with causal look-ahead keyword frames copied from the causal architecture, and weighting non-causal past and current value frames combined with causal look-ahead value frames copied from the causal architecture.

2. The AI ​​system according to claim 1, wherein, The output sequence includes only the non-causal output frame sequence.

3. A speech recognition system comprising the AI ​​system according to claim 1, the speech recognition system comprising: An encoder that uses the DNN with said dual self-attention to encode data indicating speech input, such that the output sequence of the AI ​​system includes the encoded sequence of the speech input; as well as A decoder that decodes the recognition of the speech input from the output sequence.

4. The speech recognition system according to claim 3, further comprising: An alignment decoder is configured to identify codes in the encoding sequence that encodes the identification output, and in response to identifying each code that encodes the identification output, triggers the decoder to decode the identification output in the identified codes.

5. The speech recognition system according to claim 4, wherein, The recognition of the voice input is the transcription of the voice input.

6. An acoustic event detection system comprising the AI ​​system according to claim 1, the acoustic event detection system comprising: An encoder that uses the DNN with said dual self-attention to encode data indicating the acoustic signal, such that the output sequence of the AI ​​system includes the encoded sequence of the acoustic signal; as well as A decoder that decodes acoustic events from the output sequence.

7. An audio processing method, comprising the following steps: The input frame sequence is processed by layers of a deep neural network (DNN) to generate an output sequence, wherein the input frame sequence is an ordered feature sequence describing information of the audio input, wherein the audio input is an acoustic signal or speech, and wherein at least some of the layers include dual self-attention modules with a bicausal and noncausal architecture that pays attention to both noncausal and causal frames; and Render the output sequence; Here, the causal frame that is of interest in the current frame is the look-ahead frame, while the other frames of interest are the non-causal frames. The dual self-attention module includes dual self-attention, which outputs a causal output frame sequence derived without look-ahead and a non-causal output frame sequence derived with look-ahead relative to an input frame representing the current time by a predetermined size; and wherein the output sequence includes one or a combination of the causal output frame sequence and the non-causal output frame sequence. The causal architecture of the current layer of the DNN with dual self-attention transforms the previous causal output determined by the causal architecture of the previous layer of the DNN with dual self-attention into a sequence of causal keyword frames, causal value frames and causal query frames. Wherein, the non-causal architecture of the current layer of the DNN will transform the previous non-causal output determined by the non-causal architecture of the previous layer of the DNN into a sequence of non-causal keyword frames, non-causal value frames and non-causal query frames; The causal architecture generates the current causal output by weighting the causal value frames combined with non-causal value frames copied from the non-causal architecture, with the causal query frames focusing on causal keyword frames. The non-causal architecture generates the current non-causal output by having the non-causal query frame focus on past and current non-causal keyword frames combined with causal look-ahead keyword frames copied from the causal architecture, and weighting non-causal past and current value frames combined with causal look-ahead value frames copied from the causal architecture.

8. The audio processing method according to claim 7, wherein, The output sequence includes only the non-causal output frame sequence.

9. A computer-readable storage medium including instructions that, when executed by a computer, cause the computer to perform the method according to claim 7.

Images