An artificial intelligence system for capturing context through extended self-attention

The augmented self-attention mechanism addresses the computational complexity issue in transformer architectures by using a combination of constrained and expanded attention, achieving efficient and accurate processing of long input sequences in neural networks.

JP7580639B2Active Publication Date: 2024-11-11MITSUBISHI ELECTRIC CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2023574886
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-03-26
Filing Date
2021-11-30
Publication Date
2024-11-11
Estimated Expiration
2041-11-30

AI Technical Summary

Technical Problem

Attention-based architectures, particularly transformer architectures, face increased computational complexity with longer input sequences, leading to lower processing performance and efficiency in applications like automatic speech recognition due to quadratic complexity growth with input sequence length.

Method used

Implement an augmented self-attention mechanism that combines constrained self-attention with an expansion mechanism, allowing attention at full resolution for nearby frames and reduced resolution for distant frames, using techniques like subsampling and pooling to compress information, thereby reducing computational complexity to linear growth with input sequence length.

Benefits of technology

The augmented self-attention mechanism captures the entire context of input sequences efficiently, reducing computational cost and processing time while maintaining accuracy, thus enhancing the performance of neural networks in tasks like ASR and AMT.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007580639000015
    Figure 0007580639000015
  • Figure 0007580639000016
    Figure 0007580639000016
  • Figure 0007580639000017
    Figure 0007580639000017
Patent Text Reader

Abstract

An artificial intelligence (AI) system is disclosed. The AI ​​system includes a processor that processes a sequence of input frames using a neural network including an extended self-attention module trained to compute a sequence of outputs by converting each input frame into a corresponding key frame, a corresponding value frame, and a corresponding query frame resulting in a sequence of key frames, a sequence of value frames, and a sequence of query frames in the same order, and by performing attention computation for each query frame on a combination of a portion of the sequence of key frames and the sequence of value frames restricted based on a location of the query frame and an extended sequence of key frames and an extended sequence of value frames extracted by processing different frames of the sequence of key frames and the sequence of value frames with a predetermined extraction function. The processor also renders the sequence of outputs.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates generally to artificial intelligence (AI), and more specifically, to an AI system for capturing context through dilated self-attention. [Background technology]

[0002] Today, attention mechanisms have become a central component in many neural network (NN) architectures for different artificial intelligence (AI) applications, including machine translation, speech processing, language modeling, automatic speech recognition (ASR), computer vision, etc. Self-attention mechanisms are also widely used neural network components. Self-attention mechanisms allow inputs to interact with each other ("self") and find out to whom they should pay more attention ("attention") in order to compute the output in the best way for a given task. The output of a neural network component using such self-attention mechanisms is the collection of these interactions. Summary of the Invention [Problem to be solved by the invention]

[0003] Attention-based architectures (such as transformer architectures) have been successfully applied for various domains where attention is utilized across all model components. To further improve the results using deeper and wider architectures, many model parameters are increased. Attention-based architectures process inputs of different lengths (also called "input sequence length"). In general, the computational complexity of attention-based architectures depends on the input sequence length. Also, the computational complexity of self-attention mechanisms grows quadratically with increasing input sequence length. This can be problematic for applications including, but not limited to, automatic speech recognition (ASR), where the input sequence length of an utterance can be relatively long. The increase in computational complexity of neural networks results in poor processing performance, such as increased processing time, low processing speed, and increased storage space.

[0004] To solve the problem of computational complexity in neural networks, constrained self-attention mechanisms can be used. However, constrained self-attention mechanisms ignore distant information for queries associated with the current query frame. Thus, the output results of such mechanisms may be degraded.

[0005] Therefore, there is a need for a technical solution to overcome the aforementioned limitations, more specifically, to provide a high quality output while minimizing the computational cost (time and space requirements). [Means for solving the problem]

[0006] Some embodiments are based on the recognition of an attention mechanism, which is a method for retrieving information from an input sequence by using a query frame (i.e., a query vector). The input sequence in such a mechanism acts as a memory. Also, in an extended self-attention mechanism, a query frame computed from an input sequence is used to query information from itself. In an exemplary embodiment, the input sequence may correspond to a sequence of observation vectors (frames) extracted from a speech utterance that includes a sequence of speech sound events. The self-attention mechanism may convert such frames of the input sequence into a sequence of key frames, value frames, and query frames. In some embodiments, the neighboring frames of a query frame corresponding to a frame position in the input sequence may belong to a similar sound event as the sound event of the query frame, in which case detailed information may be required to recognize their logical relationship with one or more of the key frames, value frames, and query frames. Also, distant information (such as frames in the input sequence that are far from the query frame) may be relevant to recognize the context of the input sequence. Thus, neighboring frames may have dependencies, while distant frames are relevant to trace the context that may require less detailed information.

[0007] In some exemplary embodiments, such as in machine translation or language modeling, where each word in an input sequence is represented by an observation vector, words that are very close to each other in the input sequence may be more likely to have dependencies, while only a few distant words or word groups may be relevant to trace the semantic context and syntax of the sentence, which may require less detailed information.

[0008] In some other exemplary embodiments, in an automatic speech recognition (ASR) system, adjacent frames (or nearby frames) of a query frame may belong to the same phoneme, syllable, and word, in which case detailed information is needed to recognize their consistency. Meanwhile, distant information is related to recognizing the context of sounds and words in an utterance and matching speaker or recording characteristics, which typically requires less fine-grained information. In some embodiments, a transformer-based neural network may be used for an end-to-end ASR system. The transformer-based neural network may be trained simultaneously with a frame-level classification objective function. In one exemplary embodiment, the transformer-based neural network may be trained simultaneously with a connectionist temporal classification (CTC) objective. The transformer-based neural network may exploit both encoder / decoder attention and self-attention. The encoder / decoder attention may use a query vector based on the state of the decoder of the transformer-based neural network to control attention to a sequence of input values. The sequence of input values ​​is a sequence of the encoder neural network states. Both attention types of the transformer-based neural network can be based on a scaled dot-product attention mechanism. The CTC objective realized in the transformer-based neural network can achieve the combined advantages of both the label synchronization model and the time synchronization model while also enabling streaming recognition for encoder / decoder-based ASR systems.

[0009] Some embodiments are based on the recognition of the problem that in a constrained self-attention mechanism, distant information for a query associated with a current query frame may be excluded. A constrained self-attention mechanism may enable attention to adjacent or nearby frames of the current query frame that have high resolution. That is, in a constrained self-attention mechanism, the past and future context for the current query frame is limited based on a predefined number of look-back and look-ahead frames. However, distant information may be useful in providing accurate results.

[0010] Some embodiments are based on the recognition of a recursive process of far information to compute a summary frame up to the current query frame to determine the past (left) context for the query. In the recursive process, as the query moves forward, the summary frame is updated with new input frames. This process is performed until the last query frame is processed. This repeated updating of information results in an inaccurate determination of the far context as the recursive process proceeds, because the original information of the far frames in the summary frame is outdated. Also, the recursive process cannot be parallelized to accelerate the computation of the summary frame.

[0011] To avoid such staleness of far information, and to obtain equal access to past (left context) and future (right context) far information for the current query frame, an objective of some embodiments is to accurately summarize the far context without using a recursive approach. To that end, an objective of some embodiments is to provide an expansion mechanism in addition to constrained self-attention. The combination of the expansion mechanism and constrained self-attention is called extended self-attention. In the expansion mechanism, a sequence of value frames and a sequence of key frames, both derived from an input sequence, are extracted and stored in a value expansion sequence and a key expansion sequence. The expansion mechanism may use parallel computation to simultaneously compute the frames of the key expansion sequence and the frames of the value expansion sequence. The key expansion sequence and the value expansion sequence may be of lower frame rate compared to the sequence of key frames and the sequence of value frames.

[0012] Thus, the extended self-attention, which combines the limited self-attention with the extension mechanism, performs self-attention at full resolution for frames in the immediate vicinity of the query that are within the look-ahead and look-back range of the restriction window, and performs self-attention at reduced resolution for frames farther away that may be outside the restriction window. In some exemplary embodiments, the extension mechanism of the extended self-attention subsamples or summarizes the sequence of key frames and value frames of the input sequence. The summarized key frames and value frames may be used as the extension sequence. The extension sequence of key frames and value frames may correspond to a lower frame rate compared to the frame rate of the input sequence. The extension sequence of key frames and value frames may be added to the limited sequence of key frames and value frames generated by the limited self-attention mechanism. In this way, the entire context of the input sequence is captured in part at high (full) resolution and at lower resolution to provide an accurate self-attention output. The high resolution and lower resolution information may be in a compressed form, which may consume less memory and less computation time to process the input sequence for applications related to machine translation, language modeling, speech recognition, etc.

[0013] Some embodiments are based on the realization that a sequence of inputs, i.e., relevant information from an input sequence, can be extracted or compressed in chunks of frames based on different frame rate reduction methods, such as subsampling and pooling methods that extract or compress relevant information in chunks of frames. Examples of frame reduction methods may include, but are not limited to, average pooling, max pooling, attention-based pooling, etc.

[0014] In some embodiments, relevant information within chunks of frames is extracted or compressed by an attention-based pooling approach, which uses trained embedding vectors to derive one or more query vectors that are used to calculate a weighted average of the chunks.

[0015] In some embodiments, relevant information from the sequence of inputs may be extracted using block processing, subsampling, and compression techniques. In such approaches, distant frames relative to the current query frame are processed at a lower resolution, and adjacent frames relative to the current query frame are processed at a higher (full) resolution. The distant and nearby information may be combined to obtain a compressed form of the sequence of inputs. Different embodiments use different predefined extraction functions to extract information from all available / relevant keyframes and value frames. Different extraction functions use one or a combination of the above mentioned extraction techniques to combine the limited self-attention with other useful information captured by the extended self-attention.

[0016] Furthermore, some embodiments are based on the realization that the computational complexity of the self-attention mechanism grows quadratically with the input sequence length. Therefore, an objective of some embodiments is to mitigate the quadratic growth of the computational cost of self-attention with the input sequence length. According to one embodiment, the computational cost of the limited self-attention in the extended self-attention system grows only linearly with the input sequence length. The computational cost of paying attention to the extended sequence is reduced by 1 / M compared to the full sequence-based self-attention, where M represents the chunk size of the subsampling or pooling operation.

[0017] Thus, although the extended self-attention mechanism still captures the entire context of the input sequence at a different resolution, the overall complexity of the extended self-attention mechanism is significantly smaller compared to the full-sequence-based self-attention mechanism.

[0018] Accordingly, one embodiment discloses an artificial intelligence (AI) system that accounts for inputs in a sequence of inputs by examining their interdependencies with each other, the AI ​​system comprising at least one processor and a memory having instructions stored therein that, when executed by the at least one processor, cause the artificial intelligence (AI) system to perform a plurality of steps, the plurality of steps including: accepting a sequence of input frames; processing the sequence of input frames using a neural network including at least one extended self-attention module trained to compute a corresponding sequence of outputs from the sequence of input frames by converting each input frame of the sequence of input frames into a corresponding key frame, a corresponding value frame, and a corresponding query frame resulting in a sequence of key frames, a sequence of value frames, and a sequence of query frames in the same order, and by performing attention computation for each query frame in the sequence of query frames on a combination of a portion of the sequence of key frames and the sequence of value frames restricted based on a location of the query frame in the sequence of query frames, and an extended sequence of key frames and an extended sequence of value frames extracted by processing different frames of the sequence of key frames and the sequence of value frames using a predetermined extraction function; and rendering the sequence of outputs.

[0019] The presently disclosed embodiments are further described with reference to the accompanying drawings. The drawings shown are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a block diagram illustrating a network environment for implementing an artificial intelligence (AI) system in accordance with some embodiments of the present disclosure. [Diagram 2] FIG. 2 is a block diagram of the AI ​​system illustratively shown in FIG. 1 , according to some embodiments of the present disclosure. [Figure 3A] 1 is a diagrammatic representation of an attention mechanism of an extended self-attention module of an AI system according to an exemplary embodiment of the present disclosure. [Figure 3B] 1 is a diagrammatic representation of an attention mechanism of an extended self-attention module of an AI system according to another exemplary embodiment of the present disclosure. [Figure 4] FIG. 1 is a block diagram of an extended self-attention module of an AI system, according to some embodiments of the present disclosure. [Diagram 5] FIG. 1 is a block diagram illustrating a transformer-based neural network of an AI system according to some embodiments of the present disclosure. [Figure 6A] 1 is a graphical representation illustrating compression of a set of key / value frames of a sequence of input according to an exemplary embodiment of the present disclosure. [Figure 6B] 11 is a graphical representation illustrating compression of a set of key / value frames of a sequence of input according to another exemplary embodiment of the present disclosure. [Figure 6C] FIG. 1 illustrates an output sequence output by an AI system according to some embodiments of the present disclosure. [Figure 7A] FIG. 1 illustrates attention-based pooling, in accordance with some embodiments of the present disclosure. [Figure 7B] FIG. 1 illustrates an attention-based extension with post-processing, in accordance with some embodiments of the present disclosure. [Figure 7C] FIG. 1 illustrates multi-resolution extended self-attention via attention-based pooling, in accordance with some embodiments of the present disclosure. [Figure 8A] FIG. 1 is a block diagram of an AI system in an automatic speech recognition (ASR) system, according to some embodiments of the present disclosure. [Figure 8B]FIG. 1 is a block diagram of an AI system in an automatic machine translation (AMT) system, according to some embodiments of the present disclosure. [Figure 9] FIG. 1 illustrates an example scenario for the implementation of an AI system in accordance with some other embodiments of the present disclosure. [Figure 10] FIG. 1 illustrates an overall block diagram of an AI system according to some example embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without these specific details. In other instances, the devices and methods are shown only in block diagram form in order to avoid obscuring the present disclosure.

[0022] As used in this specification and claims, the terms "for example," "such as," "including," and the verbs "comprise," "have," "include," and other verb forms thereof, when used in conjunction with a list of one or more components or other items, should each be construed as open-ended. That is, the list should not be considered to exclude other additional components or items. The term "based on" means based at least in part on. It should also be understood that the language and terminology employed herein is for purposes of illustration and should not be considered limiting. Any headings used within this description are for convenience only and have no legal or limiting effect.

[0023] FIG. 1 is a block diagram illustrating a network environment 100 for implementing an artificial intelligence (AI) system 102 according to some embodiments of the present disclosure. The network environment 100 is depicted to include a user 106 associated with a user device 108. In the illustrated exemplary scenario, the user 106 provides an input, such as an input 110, to the user device 108. The user device 108 may receive the input 110 as an acoustic signal or a voice utterance. The user device 108 may include an application, such as an automatic speech recognition (ASR) or automatic machine translation (AMT) application, hosted by a server 104. The input 110 may be provided to the server 104 via a network 116. The server 104 may be configured to process the input 110 to perform different operations, such as operations related to ASR and AMT applications. In an exemplary embodiment, the user 106 may provide the input 110 as a voice input (also referred to as voice input 110) related to a technical problem that may be solved by a technical solution provider 112. The technical solution provider 112 may include, but is not limited to, a human agent, a virtual bot, and an interactive voice response (IVR) system. The server 104 receives voice input 110 from the user device 108 and forwards it to the technical solution provider 112 via the network 116.

[0024] Also, the network 116 may include suitable logic, circuits, and interfaces that may be configured to provide multiple network ports and multiple communication channels for transmitting and receiving data. Each network port may correspond to a virtual address (or physical machine address) for transmitting and receiving communication data. For example, the virtual address may be an Internet Protocol Version 4 (IPv4) (or IPv6 address) and the physical address may be a Media Access Control (MAC) address. The network 116 may be associated with an application layer for realization of a communication protocol based on one or more communication requests from at least one of the one or more communication devices. The communication data may be transmitted or received via the communication protocol. Examples of such wired and wireless communication protocols may include, but are not limited to, Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, Infrared (IR), IEEE 802.11, 802.16, cellular communication protocols, and / or Bluetooth (BT) communication protocols.

[0025] Examples of the network 116 may include, but are not limited to, wireless channels, wired channels, and combinations of wireless and wired channels. The wireless or wired channels may be associated with a network standard that may be defined by one of a Local Area Network (LAN), a Personal Area Network (PAN), a Wireless Local Area Network (WLAN), a Wireless Area Network (WAN), a Wireless Wide Area Network (WWAN), a Long Term Evolution (LTE) network, plain old telephone service (POTS), and a Metropolitan Area Network (MAN). In addition, the wired channel may be selected based on a bandwidth criterion. For example, a fiber optic channel may be used for high bandwidth communications. Also, a coaxial cable-based or Ethernet-based communications channel may be used for medium bandwidth communications.

[0026] In some embodiments, the speech input 110 may be very long. In such cases, the computational complexity of the server 104 may be high. Thus, the server 104 may not process the speech input 110 accurately and / or in a timely manner, which may result in the generation of inaccurate output. Also, processing a very long speech input 110 may take time, which may result in a delay in the response to the user input. Moreover, the server may also suffer from a backlog as the server takes more time in processing a very long speech input 110.

[0027] To that end, a high quality output (at low computational cost) for an audio input 110 can be generated using an AI system 102, which is now described with reference to FIG.

[0028] 2 is a block diagram of the AI ​​system 102 illustratively shown in FIG. 1, according to some embodiments of the present disclosure. The AI ​​system 102 accounts for inputs in a sequence of inputs by examining the interdependencies of the inputs with respect to one another. The AI ​​system 102 includes a processor 202, a memory 204, and an input / output (I / O) interface 210. The memory 204 has a neural network 206 that includes an extended self-attention module 208. In some embodiments, the neural network 206 may include multiple layers of multiple extended self-attention modules.

[0029] In an exemplary embodiment, the I / O interface 210 is configured to receive a sequence of inputs, which may correspond to an audio input having a time dimension (such as the audio input 110). The processor 202 is also configured to execute instructions stored in the memory 204. Execution of the stored instructions causes the AI ​​system 102 to accept a sequence of input frames, which represent an ordered sequence of features that describe information in the input signal (such as the audio input 110). The sequence of input frames is also processed using a neural network 206 that includes an augmented self-attention module 208 that is trained to compute a corresponding sequence of outputs from the sequence of input frames.

[0030] Some embodiments are based on the recognition that an input signal may include a sequence of input frames, and the input frames are transformed into a sequence of keys, values, and queries. Each query frame of the sequence of queries is searched on the sequence of keys to calculate the relationship of each key frame to the query frame. Each key frame is related to a value frame that encodes features related to each input frame. The estimated relationship of each key frame to the query frame is used to assign a weighting factor to each value frame to calculate a weighted average of the sequence of value frames and the output for the query search. For example, if each input frame of the sequence of input frames corresponds to a sequence of words, i.e., a word in a sentence, then the estimated relationship of each key frame to a query frame will represent the relationship of the word associated with the query frame to all other words in the sentence.

[0031] Some embodiments are also based on the realization that the relationship of queries to keys and the relationship of keys to queries are distinguishable: the attention mechanism can learn to reshape the relationships between search words and words that provide context as the network learns.

[0032] Thus, in some embodiments, the processor 202 processes the sequence of input frames through the neural network 206 by converting each input frame in the sequence of input frames into a corresponding key frame, a corresponding value frame, and a corresponding query frame, resulting in a sequence of key frames, a sequence of value frames, and a sequence of query frames in the same order. In some embodiments, the location of a query frame in the sequence of query frames corresponds to its location in the sequence of key frames and value frames.

[0033] The processor 202 also performs an attention computation for each query frame in the sequence of query frames on a combination of the portion of the sequence of key frames and the sequence of value frames and the extended sequence of key frames and the extended sequence of value frames via the neural network 206. The portion of the sequence of key frames and the sequence of value frames is determined based on the location or position of the query frame in the sequence of query frames.

[0034] For this purpose, the extended self-attention module 208 is trained to compute a sequence of outputs from a sequence of input frames based on the learned transformation and a sequence of attention computations that allow mapping a sequence of value frames to outputs by using a current query frame and a sequence of key frames. In some exemplary embodiments, the extended self-attention module 208 provides an attention mechanism for retrieving information from a sequence of inputs by using a current query frame if the current query frame corresponds to a query vector.

[0035] The extended self-attention module 208 is further adapted to compare different query frames of the sequence of query frames with different representations of the sequence of key frames. The comparison of different query frames with different representations of key frames generates different weight distributions for the different representations of the sequence of value frames. The different weight distributions are used to calculate weighted averages of the different representations of the value frames, which form the sequence of outputs of the output. The different sequence representations of the key frames and value frames are formed by combining subsequences of key frames and value frames with compressed or subsampled sequences of key frames and value frames. In some embodiments, the subsequences of key frames and value frames may be selected based on the location of the current query frame and the compressed or subsampled sequence of key frames and value frames.

[0036] In some embodiments, neighboring frames of a current query frame of a sequence of inputs may be utilized to provide information related to the current query frame (hereinafter, "related information"). The related information from the neighboring frames may contain elements similar to those of the current query frame or stronger relationships with the current query frame. In such cases, more information may be required to recognize the logical relationship between the elements of the related information and the elements of the current query frame. Thus, some embodiments are based on the recognition that frames neighboring the current query frame are more likely to have dependencies.

[0037] Also, frames in a sequence of inputs that are far from the current query frame may provide distant information that may be relevant to recognizing the context of the sequence of inputs. Accordingly, some embodiments are based on the recognition that distant neighboring frames may be relevant to tracing the context related to explaining the sequence of inputs.

[0038] For example, for speech recognition, adjacent frames of the current query frame may correspond to the same phoneme, syllable, or word. Distant information may be related to recognizing the context of sounds and words in an utterance and matching speaker or recording characteristics, which typically requires less fine-grained information. For machine translation, words adjacent to the current query frame are more likely to have dependencies, while only a few distant words or word groups may be related to tracing the semantic context and syntax of a sentence, which may require less detailed information.

[0039] To determine the context and syntax of the input sequence, an extended sequence of key frames and an extended sequence of value frames are generated by the AI ​​system 102. For that purpose, the processor 202 performs a non-recursive sequence compression of the sequence of key frames and value frames. In particular, the processor 202 utilizes an extended self-attention module 208 to simultaneously process the entire time dimension of the sequence of input frames, including the sequence of key frames and value frames. In such a case, the output of each frame is independent of the other frames. For this reason, the original information carried by each frame is processed to determine the exact context and syntax of the input sequence.

[0040] Also, in some implementations, non-recursive sequence compression of the sequence of key frames and value frames is achieved by applying an extraction technique, e.g., a compression technique, in parallel to all of the frames of the sequence of key frames and value frames. Thus, the computational complexity of the self-attention process is reduced, and the self-attention process is expanded to sequence processing with different attention resolutions. Thus, the expanded mechanism can efficiently summarize different features of each frame of the sequence of key frames and value frames, providing a reduction in the computational complexity of the neural network 206. A neural network 206 trained with such an expansion mechanism provides a low computational complexity for generating a sequence of outputs. Thus, the processing speed of the processor 202 utilizing such a neural network is increased, which results in a fast response time of the AI ​​system 102. Thus, the AI ​​system 102 disclosed in the present disclosure facilitates output in a faster manner. Also, the processor 202 renders the sequence of outputs via the I / O interface 210.

[0041] In one embodiment in which the neural network 206 includes multiple neural network layers with extended self-attention modules, the extension mechanism is performed independently for each extended self-attention module at each layer of the neural network.

[0042] In another embodiment, the extended self-attention module 208 applies multiple extension mechanisms in succession to generate multiple extended sequences for keys and values ​​on the process pipeline. In particular, the processor 202 generates a first extended sequence for key frames and value frames and a second extended sequence for key frames and value frames in succession on the process pipeline. In such a case, the first extended sequence of key frames and value frames rendered by the first extension mechanism with a first chunk size forms an input to a second extension mechanism with a second chunk size for rendering a second extended sequence of key frames from the first extended sequence of key frames and rendering a second extended sequence of value frames from the first extended sequence of value frames. In this way, extended sequences with different frame rates, i.e. different resolutions, can be obtained.

[0043] 3A is a diagrammatic representation 300 depicting the principle of the attention mechanism used by some exemplary embodiments. For example, the extended self-attention module 208 may use the principle of the attention mechanism to retrieve information from a sequence of inputs based on a current query frame, such as the query 302. In one exemplary embodiment, in the attention mechanism of the extended self-attention module 208, the input of the source sequence 308 is first converted into key frames and value frames. The key frames and value frames may include key frames 304A, 304B, 304C, and 304D (also referred to as keys 304A-304D) with corresponding value frames 306A, 306B, 306C, and 306D (also referred to as values ​​306A-306D). The source sequence 308 may correspond to a feature sequence derived from the speech input 110.

[0044] In one example embodiment, the extended self-attention module 208 determines the similarity between the query 302 and each of the keys 304A-304D. The similarity is used to calculate an attention score for each of the values ​​306A-306D.

[0045] In some exemplary embodiments, the attention scores may be normalized based on a softmax function to calculate the attention weight distribution. To that end, the augmented self-attention module 208 of the neural network 206 utilizes a softmax function such that the unnormalized scores of the augmented self-attention module 208 are mapped to a probability distribution over a sequence of value frames. The softmax function is a function that turns a vector of K real values ​​into a vector of K real values ​​that sum to one. While the input values ​​can be positive, negative, zero, or values ​​greater than one, the softmax transforms the input values ​​into values ​​between 0 and 1 such that the input values ​​can be interpreted as probabilities that sum to one. Thus, the input of the softmax function may be the dot product score of the query 302 and the keys 304A-304D utilized to determine the attention score. Each of the corresponding values ​​306A, 306B, 306C, and 306D is weighted according to the normalized attention score. That is, each of the values ​​306A, 306B, 306C, and 306D is multiplied by a normalized attention score. Also, the weighted values ​​306A-306D are summed. The extended self-attention module 208 determines an output vector, such as attention value 310, based on the sum of the weighted values ​​306A-306D. The attention scores for each of the corresponding values ​​306A, 306B, 306C, and 306D are further described with reference to FIG. 3B.

[0046] 3B is a diagrammatic representation 312 of an attention mechanism used by the extended self-attention module 208 of the AI ​​system 102 according to another exemplary embodiment of the present disclosure. In the illustrated exemplary scenario, a query 302 is selected by the AI ​​system 102 from a sequence of inputs 314. The sequence of inputs 314 comprises input frames. In an exemplary embodiment, each input frame may correspond to a word in the sequence of inputs 314, e.g., words 314A (w1), 314B (w2), 314C (w3), and 314D (w4). The sequence of inputs 314 corresponds to the source sequence 308. The input word 314C (w3) is selected as the query 302. Additionally, a set of keys 316A, 316B, 316C, and 316D (also referred to as the sequence of keys 316A-316D) and a corresponding set of values ​​320A, 320B, 320C, and 320D (also referred to as the sequence of values ​​320A-320D) are generated by the AI ​​system 102 for a sequence of inputs 314. The sequence of keys 316A-316D and the sequence of values ​​320A-320D correspond to the sequence of keys 304A-304D and the sequence of values ​​306A-306D.

[0047] The extended self-attention module 208 determines attention scores for the sequence of values ​​320A-320D. In particular, attention score 318A is determined for value 320A, attention score 318B is determined for value 320B, attention score 318C is determined for value 320C, and attention score 318D is determined for value 320D. The self-attention mechanism provided by the extended self-attention module 208 is further described with reference to FIG.

[0048] FIG. 4 is a block diagram of the extended self-attention module 208 of the AI ​​system 102 according to an exemplary embodiment of the present disclosure. In one embodiment, the extended self-attention module 208 includes a self-attention layer configured to perform extended self-attention. Additionally or alternatively, in some embodiments, the neural network 206 may include multiple layers of extended self-attention modules, each layer corresponding to an extended self-attention module 208. For example, in one embodiment, the extended self-attention module 208 includes one extended self-attention layer 402 and one feed-forward neural network (FFN) module 404. Additionally or alternatively, in different embodiments, the self-attention module has different combinations of self-attention layers, residual layers, feedforward layers, and other layers that are task-specific. The extended self-attention sub-network 402 learns information relations in a pairwise manner. For example, the extended self-attention layer 402 learns logical relationships of input frames in the source sequence 308 for applications such as automatic speech recognition (ASR) applications, automatic machine translation (AMT) applications, etc. The extended self-attention layer 402 and the feedforward neural network module 404 are followed by "Add & Norm" layers 403 and 405, respectively. The "Add & Norm" layer 403 first adds the input of the extended self-attention layer 402 to its output using residual connections, and then applies layer normalization. Similarly, the "Add & Norm" layer 405 first adds the input of the feedforward neural network module 404 to its output, and then applies layer normalization as well.

[0049]

number

[0050] In an exemplary application, a sequence of inputs 314 corresponds to a sequence of word embeddings for the sentence "She eats green apples." Each word in the sequence of inputs 314, i.e., "She," "eats," "green," and "apples," is converted to an embedding vector using a lookup table that is trained along with other neural network modules. In the self-attention mechanism used by the exemplary implementation of the extended self-attention module 208, the vectors are multiplied by a matrix to create a query 302, a key (e.g., keys 316A-316D), and a value (e.g., values ​​320A-320D) for each of the word embeddings 314A-314D. An attention score (e.g., attention scores 318A-318D) is calculated by taking a dot product of the query 302 for each sequence of inputs 314 with the sequence of keys 316A-316D. For example, the attention score for the first word "She" is calculated by comparing the word "She" with all other words in the sentence using a dot product of the query corresponding to the word "She" with the keys of all words 316A-316D. After the attention scores are normalized to sum to 1 using a softmax function, the estimated attention weights would be "She": 0.05, "eats": 0.8, "green": 0.05, and "apples": 0.1. Here, we denote the weight for each word after the colon. These weights are applied to the sequence of values ​​320A-320D, which are then summed to form the output vector. Similarly, the attention scores for each of the remaining words in the input sequence, i.e., "eats", "green", and "apples", are calculated by comparing the corresponding query against all the keys of the sentence in the keys 316A-316D and summing the sequence of values ​​320A-320D with the corresponding weights. In this manner, the extended self-attention module 208 transforms a sequence of inputs 314 into a sequence of outputs by using the sequence of query frames, the sequence of key frames 316A-316D, and the sequence of value frames 320A-320D.The extended self-attention module 208 compares different query frames of a sequence of query frames with different representations of key frames and value frames to generate a sequence of outputs.

[0051] To combine limited self-attention with extended self-attention, in some embodiments, a different representation of the key frames and value frames is formed by using a subset of the key frames 316A-316D and the corresponding value frames 320A-320D. The subset of key frames and value frames may be selected based on the location of the current query frame 302. Additionally, an extension mechanism may be applied to the sequence of key frames and the sequence of value frames to compute an extension sequence for the key frames and value frames.

[0052] In some embodiments, the extended self-attention layer 402 corresponds to the multi-head attention used in the transformer-based neural network of the AI ​​system 102, which is described next with reference to FIG. 5.

[0053] FIG. 5 is a block diagram illustrating a transformer-based neural network 500 of the AI ​​system 102 according to some embodiments of the present disclosure. In some exemplary embodiments, the transformer-based neural network 500 may utilize an attention-based encoder / decoder neural network, e.g., the encoder 502 and the decoder 504. In such an attention-based encoder / decoder neural network, the decoder state may be used as a query (e.g., the query 302) to control attention to a sequence of encoder states of the encoder 502. The sequence of encoder states may correspond to a sequence of outputs of the encoder 502. The transformer-based neural network 500 may also utilize an extended self-attention module 208. In some embodiments, the transformer-based neural network 500 may comprise multiple extended self-attention modules. In some exemplary embodiments, the transformer-based neural network 500 utilizes an encoder / decoder-based attention for the encoder 502 and the decoder 504, etc., and an extended self-attention. Both attention calculations may be based on a scaled dot-product attention, where the attention is calculated based on equation (1) as described above in the description of FIG. 4. Also, the transformer-based neural network 500 may comprise multiple layers of extended self-attention-based neural network modules.

[0054]

number

[0055]

number

number

[0056]

number

[0057]

number

[0058]

number

[0059] The loss function L is used for training, and γ is the objective function p ctc andp att is a hyperparameter that controls the weighting between

[0060] Also, the multi-head attention and feedforward layers in the encoder 502 and decoder 504 are followed by an “add & norm” layer, which first adds the input of each layer to its output using residual connections, and then layer normalization is applied. For example, the multi-head attention 512 is a layer that is connected to the “add & norm” layer 51. 4 Through the feedforward layer 51 6 Similarly, multi-head Note 5 30 is the "add & norm" layer 53 2 Through the feedforward layer 53 4The feedforward layer 516 applies two linear transformations to the output of the "add & norm" layer 514, separated by an activation function, e.g., rectified linear unit (ReLU). The output of the feedforward layer 516 is sent through another "add & norm" layer, which also applies a residual connection to the output, followed by layer normalization. The encoder layers 512, 514, 516, and 518 are repeated E times (without sharing parameters), and the output of the last encoder layer 518 is passed to the multi-head attention layer 530 of the decoder 504, after which layer 518 provides an input to layer 512. As an additional input, the multi-head attention layer 530 receives the previous decoder output tokens 520, which are used to compute the decoder state by processing them through layers 522, 524, 526, and 528. Layer 522 converts previously output tokens 520 into embedding vectors, which are input to a multi-head attention layer 526 after positional encoding is added by layer 524. The output of layer 526 is further processed as described above using an "add & norm" layer. The output of multi-head attention layer 530 is provided to a feedforward layer 534 through an "add & norm" layer 532. The output of feedforward layer 534 is further processed by another "add & norm" layer 536. Decoder layers 526, 528, 530, 532, 534, and 536 are applied D times in this order (without sharing parameters), and after the first application, the output of layer 536 is input to layer 526. Finally, after applying the decoder layer D times, the output of layer 536 is forwarded to a linear layer 538, a fully connected neural network layer, which projects the output vector of the decoder 504 into a score for each of the output tokens. The output of the linear layer 538 is provided to a softmax layer 540 to convert the decoder scores into probabilities for each output token of the decoder 504.

[0061] In some embodiments, the transformer-based encoder neural network 50 0The self-attention module 511 in is replaced by the extended self-attention module 208. In such a setup, instead of self-attention, extended self-attention with multiple heads is performed at each of the E transformer-based encoder layers to perform self-attention with multiple resolutions and to save computational cost.

[0062] Some embodiments are based on the recognition that distant information for a current query frame (e.g., query 302) may be useful in providing accurate results. To that end, the extended self-attention module 208 provides a self-attention mechanism that enables attention to neighboring frames of the current query frame, including distant information that is relevant to capture the accurate context of the sequence of inputs 314. In some embodiments, the extended mechanism of the extended self-attention module 208 may summarize distant information, including relevant information. Some embodiments are based on the recognition that relevant information from a sequence of inputs may be extracted or compressed within chunks of frames, which are further described with reference to FIGS. 6A-6C.

[0063] FIG. 6A is a diagrammatic representation 600 representing a sequence of input frames 602 to self-attention according to an exemplary embodiment of the present disclosure. The sequence of input frames 602 may correspond to a sequence of inputs 314. In some exemplary embodiments, a query frame, such as a current query frame 604, is obtained from the sequence of inputs. In full sequence-based self-attention 606, attention connection of all neighboring frames of the current query frame 604 is enabled, as shown in FIG. 6A. However, connecting all neighboring frames with the current query frame 604 may increase computational complexity. To that end, limited self-attention 608 may be used to reduce computational complexity.

[0064] In the limited self-attention 608, nearby frames surrounding the current query frame 604 are used for self-attention. These neighboring frames may correspond to past and future context information for the current query frame 604. In some embodiments, the extended self-attention module 208 may be configured to execute a selection function that selects a subset of input frames (e.g., neighboring frames of the current query frame 604) from the sequence of inputs 602 based on the location of the current query frame 604 to form part of the representation of the sequence of inputs 602. The selection function accepts the location of the input as a parameter and returns the neighboring frames 610A and 610B in the sequence of inputs 602. The selection function may also accept values ​​for look-ahead size and look-back size to form a window size around the current query frame to select the neighboring frames 610A and 610B. In some embodiments, the window may be a time-limited window. The selection function may limit the use of neighboring frames of the full resolution sequence of inputs 602. In some embodiments, the selection function may correspond to a restriction window 610 for selecting a subset of the input frames. In an exemplary embodiment, the selected subset of the input frames may correspond to a fixed number of lookback frames 610A and lookahead frames 610B. The lookback frames 610A may contain the past (left) context and the lookahead frames 610B may contain the future (right) context of the sequence of inputs 602 for the query 604.

[0065] However, the constrained self-attention 608 excludes distant information relative to the current query frame 604. The exclusion of distant information in the constrained self-attention 608 may degrade results. To that end, the constrained self-attention 608 may be combined with an extension mechanism to provide extended self-attention, which is described next with reference to FIG. 6B.

[0066] 6B is a diagrammatic representation 612 illustrating compression of a sequence of inputs 602 according to another exemplary embodiment of the present disclosure. In some embodiments, the extended self-attention module 208 provides an extension mechanism 612 combined with limited self-attention 608.

[0067] In the extension mechanism 612, the sequence of input 602 may be summarized to form a compression of a sequence of key frames and a sequence of value frames (e.g., a sequence of keys 316A-316D and a sequence of values ​​320A-320D). For example, the sequence of key frames and the sequence of value frames in the sequence of input 602 may be divided, via at least one processor (e.g., processor 202), into a sequence of key chunks and a sequence of sequences of value chunks, such as chunks 616A, 616B, 616C, 616D, 616E, 616F, 616G, and 616H (also referred to as chunks 616A-616H). Each key chunk includes multiple key frames, and each value chunk includes multiple value frames. In some exemplary embodiments, the extended self-attention module 208 is configured to divide the sequence of key frames and the sequence of value frames at a predetermined frequency. Additionally, each of the chunks 616A-616H may be summarized into extension frames 618A, 618B, 618C, 618D, 618E, 618F, 618G, and 618H (also referred to as digests 618A-618H), which provide a compressed form (an extension sequence 620) corresponding to the set of keys 316A-316D and corresponding values ​​320A-320D.

number

[0068] Also, the at least one processor applies a predetermined function to each of the key chunks and each of the value chunks to compress the multiple key frames of the key chunks into a smaller, predetermined number of key frames of the same dimensionality for the extended sequence of key frames and compress the multiple value frames of the value chunks into a smaller, predetermined number of value frames of the same dimensionality for the extended sequence of value frames. In some embodiments, at least some of the key chunks and at least some of the value chunks are compressed simultaneously using parallel computation processes so that fast processing speeds of the processor are achieved.

[0069] Examples of the predetermined function include, but are not limited to, a sampling function, an average pooling function (also called average pooling), a max pooling function, attention-based pooling, and convolutional neural network (CNN)-based pooling.

[0070]

number

[0071]

number

[0072] In some embodiments, a CNN-based extension mechanism is applied, where CNN-based pooling applies a convolution with trained weights and a kernel size similar to the chunk size to the sequence of key frames and the sequence of value frames.

[0073] In some embodiments, a max-pooling based expansion mechanism is applied, where a max-pooling function selects a single keyframe with maximum energy from multiple keyframes in a key chunk and selects a corresponding frame from multiple value frames in a value chunk.

[0074] In some embodiments, a sampling function based extension mechanism is applied, where the sampling function selects a single frame from multiple key frames of a key chunk and a corresponding frame from multiple value frames of a value chunk. In the sub-sampling and max pooling functions, a single key frame from a chunk of key frames is selected and the corresponding value frame from the chunk of value frames and information contained by other frames are ignored.

[0075]

number

[0076] In a preferred embodiment, attention-based pooling (AP) may be applied to summarize the information content of each chunk of key and value frames, which is further described with respect to Figures 7A and 7B.

[0077] FIG. 6C illustrates an output sequence output by the AI ​​system 102 according to some embodiments of the present disclosure. FIG. 6C is described with reference to FIG. 6A and FIG. 6B. At least one processor (such as the processor 202) calculates a sequence of outputs 622 by combining a portion of the sequence of key frames and value frames within the limit window 610 with an extended sequence 620 (i.e., an extended sequence of key frames and an extended sequence of value frames). The extended sequence 620 is determined by a non-recursive sequence compression of the sequence of key frames and value frames to reduce the computational complexity of the self-attention process. The extended sequence 620 is a sequence of outputs 622. 0 corresponds to a distant frame that adds context to the query frame and enables any system, including the AI ​​system 102, to provide accurate output with less processing time. The at least one processor also renders the sequence of outputs 622 via an output interface (such as the I / O interface 210).

[0078] FIG. 7A illustrates attention-based pooling 700 in accordance with some embodiments of the present disclosure.

[0079] In attention-based pooling (AP) 700, one or more trained query vectors (such as trained query vector 706) are utilized to determine multiple weight distributions 704A, 704B, and 704C by paying attention to chunks of key frames or chunks of value frames from an input sequence 702. Thus, attention-based pooling assigns relevance to either key frames in the chunks of key frames or value frames in the chunks of value frames to derive weight distributions 704A, 704B, and 704C. Based on the multiple weight distributions 704A, 704B, and 704C, weighted averages of the chunks of key frames and chunks of value frames are calculated.

[0080]

number

[0081] In particular, a frame of an extended sequence (such as extended sequence 708) for a sequence of key frames is calculated as a weighted average of a chunk of key frames. Additionally or alternatively, a frame of an extended sequence (such as extended sequence 708) for a sequence of value frames is calculated as a weighted average of all value frames for a chunk of value frames.

[0082] In some embodiments, the attention-based enhancement is performed in conjunction with post-processing techniques.

[0083] FIG. 7B illustrates attention-based augmentation with post-processing 710 according to some embodiments of the present disclosure.

[0084] Some embodiments are based on the recognition that application of post-processing techniques can refine the output of a system (e.g., AI system 102). To that end, output frames of key chunks and value chunks (such as the plurality of output frames 714) are processed according to post-processing rules to generate one or more frames 714 for an extended sequence of key frames and an extended sequence of value frames.

[0085] To derive output frames 714 for the extended sequence of key frames and the extended sequence of value frames, the post-processing rules include one or a combination of: preserving the output frames 714 determined for the key chunks and the value chunks, combining the output frames 714 determined for the key chunks and the value chunks, and removing at least one of the output frames 714 determined for the key chunks and the value chunks.

[0086] The post-processing rules may be adjusted for at least two different types of key chunks and value chunks, including a first type of key chunks and value chunks whose distance to the query frame under the attention calculation is less than a threshold, and a second type of key chunks and value chunks whose distance to the query frame under the attention calculation is greater than or equal to the threshold.

[0087] In some other embodiments, the neural network (such as the transformer-based neural network 500) includes multiple extension mechanisms as part of an extended self-attention module to generate at least two extended sequences for key frames and value frames. In such a setup, at least one processor (such as the processor 202) stores frames of a first extended sequence for key frames and value frames that correspond to key chunks and value chunks that have a frame distance less than a predetermined threshold to the query frame under the attention calculation. In addition, the at least one processor also stores frames of a second extended sequence for key frames and value frames that correspond to key chunks and value chunks that have a frame distance greater than or equal to a predetermined threshold to the query frame under the attention calculation.

[0088] In some embodiments, the at least one processor stores the determined number of output frames 714 for the first type of key chunks and value chunks and removes at least one of the determined number of frames 714 for the second type of key chunks and value chunks.

[0089] In some alternative embodiments, the at least one processor stores a plurality of frames 714 for the first type of key chunks and value chunks and combines a plurality of frames of the second type of key chunks and value chunks using one or a combination of average pooling and merging a plurality of output frames 714 determined for the key chunks and value chunks by neural network processing. The neural network processing includes the use of two linear transforms having trained parameters separated by a nonlinear activation function and having a bottleneck structure such that the first linear transform projects the input to a smaller dimensionality and the second linear transform projects the output of the first linear transform to the dimensionality of the query frame, the key frame, and the value frame.

[0090]

number

[0091]

number

[0092] 7C illustrates multi-resolution extended self-attention 718 via attention-based pooling according to some embodiments of the present disclosure, which will be described with reference to FIGS. 7A and 7B.

[0093] Based on performing the attention-based pooling (as described above with reference to FIG. 7A and FIG. 7B), a number of frames 720 are output (also called output frames or augmented vectors). The output frames 720 are multi-resolution frames, where the resolution corresponding to each frame varies based on the location of the frame relative to the current query frame 702A within the time-bounded window 702B. The at least one processor is configured to analyze the output frames to generate the augmented sequence 720. For that purpose, the at least one processor is configured to determine a distance between each output frame and the current query frame 702A. If the distance is greater than a threshold, the corresponding output frame is interpreted as being far from the current query frame 702A. In such a case, the at least one processor is configured to discard the output frames. Thus, the output frames 720 located at a distance greater than the threshold distance are discarded. Meanwhile, the output frames 720 located at a distance less than the threshold distance are preserved. Thus, the augmented sequence 720 is generated. Thus, information from distant frames corresponding to the query frame and neighboring frames within the time-bounded window 702B is utilized to determine the context of the current query frame 702A with lower complexity. The processor is also configured to generate an output sequence based on a combination of the extended sequence 720 and the current query frame 702A with corresponding neighboring frames within the time-bounded window 702B.

[0094] An implementation of the AI ​​system 102 in ASR and AMT applications will now be described with reference to Figures 8A and 8B.

[0095] 8A is a block diagram 800 of an AI system 102 in an automatic speech recognition (ASR) system 802 according to some embodiments of the present disclosure. The ASR system 802 also includes an input interface 804 and an output interface 806. The input interface 804 is configured to receive an acoustic signal representing at least a portion of a voice utterance. The neural network of the AI ​​system 102 2 06 converts the acoustic signal into a sequence of inputs. The neural network 206 further converts the sequence of inputs into a sequence of outputs using an augmented self-attention module 208. In some embodiments, the sequence of inputs is converted into a sequence of outputs using multiple augmented self-attention modules, each using an augmentation mechanism that can summarize distant information, including related information. The converted sequence of inputs is converted into a transcription of speech utterances. The transcription of speech utterances is provided as output via the output interface 806.

[0096] FIG. 8B is a block diagram 808 of an AI system 802 in an automatic machine translation (AMT) system 810 according to some embodiments of the present disclosure. The AMT system 810 also includes an input interface 812 and an output interface 814. The input interface 812 is configured to receive an input signal representing a speech utterance in a first language. For example, the speech utterance may be a speech utterance in English. The neural network 206 of the AI ​​system 102 converts the input signal into a sequence of inputs. The neural network 206 converts the sequence of inputs into a sequence of outputs using an augmented self-attention module 208. The converted sequence of inputs is converted into an output signal representing a speech utterance in a second language. For example, the second language may correspond to German. The output signal representing the speech utterance in the second language is output to the output interface 814. 14 is provided as output via

[0097] Also, an AI system (such as AI system 102) including an extended self-attention module (such as extended self-attention module 208) may be used in streaming applications. In such a scenario, input frames are received continuously. The extended self-attention module is utilized to generate a self-attention output for each of the input frames with a limited delay. To that end, the extended sequence of keys and the extended sequence of values ​​are expanded whenever at least one new chunk of key frames and at least one new chunk of value frames are received.

[0098] Similarly, the disclosed AI system, including the extended self-attention module, can be implemented in different applications such as sound event detection, audio tagging system, acoustic source separation system, etc.

[0099] FIG. 9 illustrates an example scenario 900 for implementation of the AI ​​system 102 according to some other embodiments of the present disclosure. In the illustrated scenario, a user 902 may provide input using a user device 904. The user device 904 may include a smartphone, a tablet, a laptop, a smartwatch, a wearable device, a desktop, or any other electronic device. The user 902 may request a service from a digital assistant 906. The digital assistant 906 may include a virtual chatbot, an interactive voice response (IVR) system, or the like. The user 902 may provide an acoustic signal representing at least a portion of a voice utterance from the user device 904 to the ASR system 702 via the network 116. The ASR system 702 may provide a transcription of the voice utterance using the AI ​​system 102. The transcription may also be provided to the digital assistant 906. The digital assistant 906 may act based on the received transcription and provide a service to the user 902. For example, the service may correspond to hiring a vehicle 908. The vehicle 908 may include an autonomous vehicle, a manually driven vehicle, or a semi-autonomous vehicle. The vehicle 908 may be connected to the network 116. The map may include pick-up and drop-off locations for the user 902. The user 902 may also drive the vehicle 908 using the ASR system 702. In some cases, the ASR system 702, AMT 710, or a combination thereof implemented with the AI ​​system 102 may be used for operations related to the navigation system of the vehicle 908.

[0100] In some cases, the AI ​​system 102 may also provide speech-to-text documentation to the user 902. For example, the user 902 may provide a voice utterance to the user device 904. The user device 904 may communicate with the AI ​​system 102 to provide a transcription of the voice utterance. The AI ​​system 102 may provide a text document based on the transcription. This may assist a user (e.g., the user 902) in writing text or maintaining a document via voice input.

[0101] In some other cases, the user 902 may be traveling in a foreign region. The user 902 may not be able to communicate with people in the foreign region in the corresponding foreign language. The user 902 may hire a vehicle 908, and the driver of the vehicle 908 may not speak the native language of the user 902 or may not know a common language to communicate with the user 902. In such a case, the user 902 may provide input to the user device 904 for machine translation of the native language of the user 902 into the foreign language. The input may correspond to an input signal representing an audio utterance in the native language, e.g., English. The audio utterance in the native language may be provided from the user device 904 to the AMT 710 via the network 116. The AMT system 710 may use the AI ​​system 102 to translate the native language and provide the audio utterance in the foreign language in a fast and efficient manner.

[0102] FIG. 10 illustrates a block diagram of an AI system 1000 according to some embodiments of the present disclosure. The AI ​​system 1000 corresponds to the AI ​​system 102 of FIG. 1. The AI ​​system 1000 comprises an input interface 1002, a processor 1004, a memory 1006, a network interface controller (NIC) 1014, an output interface 1016, and a storage device 1020. The memory 1006 is configured to store a neural network 1008. The neural network 1008 includes an extended self-attention module 1010. In some exemplary embodiments, the neural network 1008 has a transformer, conformer, or similar architecture that includes the extended self-attention module 1010 as part of the encoder, decoder, or both.

[0103] The extended self-attention module 1010 is trained to transform a sequence of inputs into a corresponding sequence of outputs. The sequence of inputs is transformed by comparing each input with a different representation of the sequence of inputs. When the processor 1004 executes the instructions stored in the memory 1006, the extended self-attention module 1010 is caused to transform inputs from the sequence of inputs to form a representation of the sequence of inputs. The representation is formed by combining a first part of the representation that depends on the location of the input in the sequence of inputs with a second part of the representation that is independent of the location of the input. The first part is a compression of the sequence of inputs that changes to transform different inputs while the second part remains constant to transform all inputs from the sequence of inputs. Also, the inputs are transformed into corresponding outputs by comparing the inputs with the formed representation.

[0104] In some embodiments, the extended self-attention module 1010 is configured to execute a selection function that selects a subset of inputs from the sequence of inputs based on a location of the inputs to form a first portion of the representation of the sequence of inputs. The selection function accepts the location of the input as a parameter and returns a subset of inputs in the sequence of inputs centered on the location of the input. The selection function may also accept a value for the size of the subset of inputs as another parameter.

[0105] In some other embodiments, the extended self-attention module 1010 is configured to form the compression by summarizing the sequence of inputs using an average pooling-based approach. In a preferred embodiment, the summarization of the sequence of inputs may use attention-based pooling with or without a post-processing stage.

[0106] The input interface 1002 is configured to accept input data 1024. In some embodiments, the AI ​​system 1000 receives the input data 1024 over a network 1022 using the NIC 1014. In some cases, the input data 1024 may be online data received over the network 1022. In some other cases, the input data 1024 may be recorded data stored in the storage device 1020. In some embodiments, the storage device 1020 is configured to store a training data set for training the neural network 1008.

[0107] In some demonstrative embodiments, the input data 1024 may include an acoustic signal representing at least a portion of a speech utterance, an input signal representing a speech utterance in a first language, etc. The neural network 1008 may be configured to convert the acoustic signal into a sequence of inputs, convert the sequence of inputs into a sequence of outputs using an augmented self-attention module 1010, and convert the sequence of outputs into a phonetic transcription of the speech utterance. The output of the phonetic transcription may be provided to the output device 1018 via the output interface 1016. Similarly, the neural network 1008 may be configured to convert the input signal into a sequence of inputs, convert the sequence of inputs into a sequence of outputs using an augmented self-attention module, and convert the sequence of outputs into an output signal representing a speech utterance in a second language. The output signal may be provided to the output device 1018 via the output interface 1016.

[0108] Various embodiments of the present disclosure provide an AI system, such as the AI ​​system 1000, that provides extended self-attention. Extended self-attention improves the accuracy and modeling ability of limited self-attention. Extended self-attention also helps to reduce the computational complexity of self-attention for long input sequences. In this way, the computational cost and memory usage of the speech processing system (e.g., the ASR system 702 and the AMT system 710) may not increase quadratically, thereby increasing the efficiency of the system in a feasible manner.

[0109] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0110] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Additionally, the same reference numbers and names in the various drawings refer to the same elements.

[0111] Also, individual embodiments may be described as a process, which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe operations as a sequential process, many of the operations may occur in parallel or simultaneously. In addition, the order of operations may be rearranged. A process may terminate when its operations are completed, but may have additional steps not described or included in the drawings. Moreover, not all operations in any specifically described process may occur in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the end of the function may correspond to the function returning to the calling function or to the main function.

[0112] Furthermore, embodiments of the disclosed subject matter may be implemented, at least in part, manually or automatically. The manual or automatic implementation may be performed or at least assisted through the use of a machine, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored on a machine-readable medium. A processor may perform the necessary tasks.

[0113] The various methods or processes outlined herein may be coded as software executable on one or more processors employing any one of a variety of operating systems or platforms. In addition, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine code or intermediate code that runs on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0114] The embodiments of the present disclosure may be embodied as a method, examples of which are provided. The operations performed as part of the method may be ordered in any suitable manner. Thus, while an example embodiment shows sequential operations, embodiments may be constructed in which operations are performed in an order other than that shown, including performing some operations simultaneously. Additionally, the use of ordinal terms such as "first," "second," etc. in the claims to modify claim elements does not, in itself, imply any priority, precedence, or order of a claim element relative to another claim element, or any chronological order in which the operations of the method are performed, but is merely used as a label to distinguish a claim element having a certain name from another element having the same name (except for the use of the ordinal term).

[0115] Although the disclosure has been described with reference to certain preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the disclosure. It is therefore the object of the appended claims to cover all such variations and modifications as fall within the true spirit and scope of the disclosure.

Claims

1. 1. An artificial intelligence (AI) system that explains inputs in a sequence of inputs by examining the interdependencies of the inputs relative to one another, the AI ​​system comprising: At least one processor; and a memory having instructions stored thereon that, when executed by at least one of the processors, cause the artificial intelligence (AI) system to perform a plurality of steps, the plurality of steps including: accepting a sequence of input frames; processing the sequence of input frames with a neural network including at least one extended self-attention module trained to compute a corresponding sequence of outputs from the sequence of input frames by converting each input frame of the sequence of input frames into a corresponding key frame, a corresponding value frame, and a corresponding query frame resulting in a sequence of key frames, a sequence of value frames, and a sequence of query frames in the same order, and by performing an attention computation for each query frame in the sequence of query frames on a combination of a portion of the sequence of key frames and the sequence of value frames restricted based on a location of the query frame in the sequence of query frames and an extended sequence of key frames and an extended sequence of value frames extracted by processing different frames of the sequence of key frames and the sequence of value frames with a predefined extraction function; and rendering the sequence of outputs; the sequence of input frames represents an ordered sequence of features describing information of a speech input having a time dimension; To generate the extended sequence of key frames and the extended sequence of value frames by an extension mechanism, at least one of the processors comprises: configured to divide the sequence of key frames and the sequence of value frames into a sequence of key chunks and a sequence of value chunks, each key chunk including a plurality of key frames and each value chunk including a plurality of value frames, and wherein at least one of the processors is further configured to: An AI system configured to apply the predetermined extraction function to each of the key chunks and each of the value chunks to compress the multiple key frames of a key chunk into a smaller, predetermined number of key frames of the same dimensionality for the extended sequence of the key frames, and to compress the multiple value frames of a value chunk into the smaller, predetermined number of value frames of the same dimensionality for the extended sequence of the value frames.

2. 2. The AI ​​system of claim 1, wherein the processor is configured to compress at least some of the key chunks and at least some of the value chunks concurrently using parallel computational processes.

3. 2. The AI ​​system of claim 1, wherein the predetermined extraction function is one or a combination of a sampling function, an average pooling function, a max pooling function, attention-based pooling, and a convolutional neural network (CNN)-based pooling.

4. 2. The AI ​​system of claim 1, wherein the predetermined extraction function is an attention-based pooling that determines a weight distribution of the multiple keyframes in the key chunk and calculates a frame of the extended sequence for the sequence of keyframes as a weighted average of the multiple keyframes in the key chunk with weights selected according to the determined weight distribution.

5. 2. The AI ​​system of claim 1, wherein the predetermined extraction function is an attention-based pooling that determines a weight distribution of the multiple key frames in the key chunk and uses the same determined weight distribution to calculate a frame of the extended sequence for the sequence of value frames as a weighted average of the multiple value frames in the value chunk.

6. the predetermined extraction function is an attention-based pooling, which generates a plurality of weight distributions by paying attention to either the chunks of key frames or the chunks of value frames, and uses the plurality of weight distributions to calculate a plurality of output frames corresponding to weighted averages of the chunks of key frames and the chunks of value frames; the processor is further configured to perform post-processing on the plurality of output frames of the key chunks and the value chunks according to a post-processing rule to generate one or more frames for the extended sequence of key frames and the extended sequence of value frames; To derive frames for the extended sequence of key frames and the extended sequence of value frames, the post-processing rule comprises: storing the determined output frames for the key chunks and the value chunks; combining the determined output frames for the key chunks and the value chunks; and removing at least one of the plurality of output frames determined for the key chunk and the value chunk; 2. The AI ​​system of claim 1, wherein the processor is further configured to adjust the post-processing rules for at least two different types of key chunks and value chunks, including a first type of key chunks and value chunks whose distance to the query frame under the attention calculation is less than a threshold, and a second type of key chunks and value chunks whose distance to the query frame under the attention calculation is greater than or equal to a threshold.

7. The processor further comprises: storing the determined number of frames for the first type of key chunks and value chunks; configured to remove at least one of the plurality of frames of the second type key and value chunks; 7. The AI ​​system of claim 6, wherein the neural network processing includes the use of two linear transforms having trained parameters separated by a non-linear activation function and having a bottleneck structure such that a first linear transform projects an input to a smaller dimensionality and a second linear transform projects an output of the first linear transform to the dimensionality of a query frame, a key frame, and a value frame.

8. The processor further comprises: storing the plurality of frames for the first type of key chunks and value chunks; 7. The AI ​​system of claim 6, configured to combine the multiple frames of the second type of key chunks and value chunks using one or a combination of average pooling and merging the multiple output frames determined for the key chunks and value chunks by processing the neural network.

9. The extended self-attention module is used in a streaming application, wherein the sequence of input frames is continuously received, and a self-attention output is generated for each of the input frames with a bounded delay by extending the extended sequence of key frames and the extended sequence of value frames when at least one new chunk of key frames and at least one new chunk of value frames are generated; The AI ​​system of claim 1 , wherein a portion of the key frames from the sequence of key frames and a portion of the value frames from the sequence of value frames are selected based on a position of the query frame in the sequence of query frames.

10. 2. The AI ​​system of claim 1, wherein the extended self-attention module transforms the sequence of input frames into the sequence of query frames, the sequence of key frames, and the sequence of value frames such that the attention computation compares a query frame with a portion of a key frame from the sequence of key frames and an extended sequence transformed from the sequence of key frames to generate an output from a sequence of a portion of value frames from the sequence of value frames and the extended sequence of value frames.

11. The AI ​​system of claim 1 , wherein the neural network includes multiple layers of the extended self-attention module.

12. the processor is further configured to generate, in succession on a process pipeline, a first expanded sequence for the key frames and the value frames according to a first expansion mechanism and a second expanded sequence for the key frames and the value frames according to a second expansion mechanism; 2. The AI ​​system of claim 1, wherein the first extended sequence having a first chunk size corresponds to an input to a second extension mechanism having a second chunk size for rendering the second extended sequence of key frames from the first extended sequence of key frames and for rendering the second extended sequence of value frames from the first extended sequence of value frames.

13. The processor further comprises: preserving the frames of the first extended sequence for the key frames and value frames corresponding to key chunks and value chunks having a frame distance to the query frame less than a predetermined threshold under the attention computation; The AI ​​system of claim 11 , configured to preserve the frames of a second extended sequence for the key frames and value frames that correspond to key chunks and value chunks that have a frame distance to the query frame that is greater than or equal to a predetermined threshold under the attention calculation.

14. 2. The AI ​​system of claim 1, wherein the neural network has a transformer or conformer architecture that includes the extended self-attention module as part of an encoder, a decoder, or both.

15. 10. The AI ​​system of claim 1 forming at least a part of one or a combination of an automatic speech recognition (ASR) system, a sound event detection system, a speech tagging system, an acoustic source separation system, and a machine translation system.

Citation Information

Patent Citations

  • Generation device, learning device, generation method and program

    JP2020135457A

  • Multitask Learning as Question Answering

    JP2021507429A

  • Contextual grounding of natural language phrases in images

    US20210081728A1

  • Answer generating device, answer learning device, answer generating method, and answer generating program

    WO2020174826A1