Systems and methods for using residual transformer in natural language processing

Abstract

A method includes providing an embedded vector representing a flag in an input to a transducer comprising a plurality of transducer layers arranged in sequence, each transducer layer having a residual connection to each previous transducer layer. The method also includes, for each transformer layer, determining an input embedded vector for the first flag based on a combination of output embedded vectors from a previous transformer layer. The method further includes, for each converter layer, processing the input embedded vector for a first flag to generate an output embedded vector to be provided to each subsequent converter layer.

Description

Technical Field

[0001] The present disclosure relates generally to natural language processing systems and more particularly to systems and methods for using residual transformers in natural language processing. Background Art

[0002] Language models based on the Transformer architecture, such as BERT, provide good performance in many natural language processing (NLP) tasks. At their core, these models are built by stacking multiple transformer layers on top of each other in a sequential manner. The input passes through each transformer layer, where the output of one transformer layer is used as the input to the layer above it. It has been found that the ability of these transformer-based language models to solve tasks and adapt to new NLP tasks is directly proportional to the number of transformer layers stacked on top of each other. However, deeper language models with a large number of transformer layers have some disadvantages, such as slower training times and weaker loss signal propagation to the initial layers. Summary of the invention

[0003] The present disclosure provides a system and method for using a residual transformer in natural language processing.

[0004] In a first embodiment, a method includes providing an embedding vector representing a token in an input to a transformer including a plurality of transformer layers arranged in sequence, each transformer layer having a residual connection to each previous transformer layer. The method also includes, for each transformer layer, determining an input embedding vector for a first token based on a combination of output embedding vectors from previous transformer layers. The method also includes, for each transformer layer, processing the input embedding vector for the first token to generate an output embedding vector to be provided to each subsequent transformer layer.

[0005] In a second embodiment, an electronic device includes: at least one processing device configured to provide an embedding vector representing a token in an input to a transformer including a plurality of transformer layers arranged in sequence, each transformer layer having a residual connection to each previous transformer layer. The at least one processing device is also configured to: for each transformer layer, determine an input embedding vector for a first token based on a combination of output embedding vectors from previous transformer layers. The at least one processing device is also configured to: for each transformer layer, process the input embedding vector for the first token to generate an output embedding vector to be provided to each subsequent transformer layer.

[0006] In a third embodiment, a non-transitory machine-readable medium includes instructions that, when executed, cause at least one processor of an electronic device to provide an embedding vector representing a token in an input to a transformer comprising a plurality of transformer layers arranged in sequence, each transformer layer having a residual connection to each previous transformer layer. The medium also includes instructions that, when executed, cause the at least one processor to: for each transformer layer, determine an input embedding vector for a first token based on a combination of output embedding vectors from previous transformer layers. The medium also includes instructions that, when executed, cause the at least one processor to, for each transformer layer, process the input embedding vector for the first token to generate an output embedding vector to be provided to each subsequent transformer layer.

[0007] Other technical features may be apparent to those skilled in the art from the following drawings, descriptions and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals represent like parts:

[0009] Figure 1 illustrates an example network configuration including electronic devices according to the present disclosure;

[0010] Figure 2 An example residual transformer according to the present disclosure is shown;

[0011] Figure 3 showing details of an example transformer layer according to the present disclosure;

[0012] Figure 4 An example residual attention transformer according to the present disclosure is shown;

[0013] Figure 5 It is shown that according to the present disclosure, Figure 4 An example of vertical residual attention performed in the Residual Attention Transformer of .

[0014] Figure 6 It is shown that according to the present disclosure, Figure 2 An example of hyper-attention performed in the residual transformer of ;

[0015] Figure 7 An example of dense-attention according to the present disclosure is shown;

[0016] Figure 8 shows an example transformer layer for performing dense-attention according to the present disclosure; and

[0017] Fig. 9An example method for using a residual transformer in natural language processing according to the present disclosure is shown. DETAILED DESCRIPTION

[0018] Before proceeding to the following detailed description, it may be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The terms "send," "receive," and "communicate," and their derivatives, include direct and indirect communications. The terms "include," and "comprise," and their derivatives, are meant to include, but are not limited to, this. The term "or" is inclusive, meaning and / or. The phrase "associated with..." and its derivatives are meant to include, be included within, be interconnected with, include, be contained within, be connected to or connected with, be coupled to or coupled with, be communicable with, collaborate with, be intertwined, be juxtaposed, be close to, be bound to or bound with, have, have the nature of, have a relationship with, etc.

[0019] In addition, the various functions described below can be implemented or supported by one or more computer programs, each of which is formed by a computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data or a portion thereof that are suitable for implementation in a suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a compact disk (CD), a digital video disk (DVD) or any other type of memory. "Non-transitory" computer-readable media does not include wired, wireless, optical or other communication links that transmit temporary electrical signals or other signals. Non-transitory computer-readable media include media that can permanently store data and media that can store data and rewrite data later, such as rewritable optical disks or erasable memory devices.

[0020] As used herein, terms and phrases such as "having", "may have", "include" or "may include" a feature (such as a number, function, operation or component such as a part) indicate the presence of the feature and do not exclude the presence of other features. In addition, as used herein, the phrases "A or B", "at least one of A and / or B" or "one or more of A and / or B" may include all possible combinations of A and B. For example, "A or B", "at least one of A and B" and "at least one of A or B" may indicate all of the following: (1) including at least one A, (2) including at least one B or (3) including at least one A and at least one B. In addition, as used herein, the terms "first" and "second" may modify various components regardless of their importance and do not limit the components. These terms are only used to distinguish one component from another. For example, a first user device and a second user device may indicate user devices that are different from each other, regardless of the order or importance of the devices. Without departing from the scope of the present disclosure, a first component may be represented as a second component, and vice versa.

[0021] It should be understood that when an element (such as a first element) is referred to as being (operably or communicatively) “coupled” or “connected” to or “connected to” another element (such as a second element), it may be coupled or connected to / connected to the other element directly or via a third element. Conversely, it should be understood that when an element (such as a first element) is referred to as being “directly coupled” or “directly coupled to” or “directly connected to” another element (such as a second element), there are no other elements (such as a third element) between the element and the other element.

[0022] As used herein, the phrase "configured (or set) to" may be used interchangeably with the phrases "suitable for," "capable of," "designed to," "suitable for," "manufactured to," or "capable of," depending on the circumstances. The phrase "configured (or set) to" does not inherently mean "specially designed in hardware to." On the contrary, the phrase "configured to" may indicate that a device may perform an operation together with another device or component. For example, the phrase "a processor configured (or set) to perform A, B, and C" may represent a general-purpose processor (such as a CPU or application processor) that can perform operations by executing one or more software programs stored in a memory device, or a special-purpose processor (such as an embedded processor) for performing operations.

[0023] The terms and phrases used herein are only used to describe some embodiments of the present disclosure, rather than to limit the scope of other embodiments of the present disclosure. It should be understood that the singular forms "one", "an" and "the" include plural references, unless the context clearly states otherwise. All terms and phrases used herein (including technical and scientific terms and phrases) have the same meaning as the meaning generally understood by those of ordinary skill in the art to which the embodiments of the present disclosure belong. It will be further understood that terms and phrases (such as those defined in a common dictionary) should be interpreted as having a meaning consistent with its meaning in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense, unless explicitly defined herein. In some cases, the terms and phrases defined herein may be interpreted as excluding embodiments of the present disclosure.

[0024] Examples of "electronic devices" according to embodiments of the present disclosure may include at least one of the following: smart phones, tablet personal computers (PCs), mobile phones, video phones, e-book readers, desktop PCs, laptop computers, netbook computers, workstations, personal digital assistants (PDAs), portable multimedia players (PMPs), MP3 players, mobile medical devices, cameras, or wearable devices (such as smart glasses, head-mounted devices (HMDs), electronic clothing, electronic bracelets, electronic necklaces, electronic accessories, electronic tattoos, or smart watches). Other examples of electronic devices include smart home appliances. Examples of smart home appliances may include at least one of the following: televisions, digital video disk (DVD) players, audio players, refrigerators, air conditioners, vacuum cleaners, ovens, microwave ovens, washing machines, dryers, air purifiers, set-top boxes, home automation control panels, security control panels, TV boxes (such as Samsung HOMESYNC, APPLETV, or GOOGLE TV), smart speakers or speakers with integrated digital assistants (such as Samsung GALAXY HOME, APPLE HOMEPOD, or AMAZON ECHO), game consoles (such as XBOX, PLAYSTATION, or NINTENDO). Electronic dictionary, electronic key, portable camera or electronic photo frame. Other examples of electronic devices include various medical devices (such as various portable medical measurement devices (such as blood sugar measurement devices, heart rate measurement devices or body temperature measurement devices), magnetic resource angiography (MRA) devices, magnetic resource imaging (MRI) devices, computer tomography (CT) devices, imaging devices or ultrasound devices), navigation devices, global positioning system (GPS) receivers, event data recorders (EDR), flight data recorders (FDR), car infotainment devices, navigation electronic devices (such as navigation navigation devices or gyrocompasses), avionics, security devices, vehicle head units, industrial or domestic robots, automatic teller machines (ATMs) point of sale (POS) devices or Internet of Things (IoT) devices (such as light bulbs, various sensors, electric meters or gas meters, sprinklers, fire alarms, thermostats, street lights, toasters, fitness equipment, hot water tanks, heaters or boilers). Other examples of electronic devices include at least a part of a piece of furniture or a building / structure, an electronic board, an electronic signature receiving device, a projector or various measuring devices (such as devices for measuring water, electricity, gas or electromagnetic waves). Note that according to various embodiments of the present disclosure, the electronic device may be one or a combination of the devices listed above. According to some embodiments of the present disclosure, the electronic device may be a flexible electronic device. The electronic device disclosed herein is not limited to the devices listed above, and may include new electronic devices according to the development of technology.

[0025] In the following description, an electronic device is described with reference to the accompanying drawings according to various embodiments of the present disclosure. As used herein, the term "user" may refer to a person using an electronic device or another device (such as an artificial intelligence electronic device).

[0026] Functions related to artificial intelligence (hereinafter referred to as AI) according to the present disclosure are operated by a processor and a memory. The processor may be composed of one or more processors. In this case, one or more processors may be general-purpose processors such as CPUs, APs, digital signal processors (DSPs), etc., or graphics-specific processors such as GPUs, visual processing units (VPUs), or AI-specific processors such as NPUs. One or more processors may be controlled to process input data according to predefined operating rules or AI models stored in a memory. Alternatively, when one or more processors are AI-specific processors, the AI-specific processors may be designed to have a hardware structure dedicated to processing a specific AI model.

[0027] The predefined operating rules or AI models are characterized as being performed by learning. Performing by learning means training the basic AI model using multiple learning data through a learning algorithm, thereby creating a predefined set of operating rules or AI models to perform the desired features (or purposes). This learning can be performed in the device itself that executes the AI ​​according to the present disclosure, or can be performed by a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the above examples.

[0028] The AI ​​model may be composed of multiple neural network layers. Each of the multiple neural networks has multiple weight values, and the neural network calculation is performed by the calculation between the calculation result of the previous layer and the multiple weights. The multiple weight values ​​of the multiple neural network layers can be optimized by learning the results of the AI ​​model. For example, multiple weights can be updated to reduce or minimize the loss value or cost value obtained from the AI ​​model during the learning process. The AI ​​neural network may include a deep neural network (DNN) (e.g., a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recursive deep neural network (BRDNN), a deep Q network, etc.), but is not limited to the above examples.

[0029] Definitions for certain other words and phrases may be provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most, instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

[0030] Nothing in the description of this application should be read as implying that any particular element, step, or function is an essential element that must be included within the scope of the claims. The scope of the patented subject matter is limited solely by the claims. In addition, none of the claims are intended to invoke 35 U.S.C. §112(f) unless the exact words "means for..." are followed by a participle. The use of any other terms within the claims, including but not limited to "mechanism," "module," "device," "unit," "component," "element," "member," "device," "machine," "system," "processor," or "controller," should be understood by applicants to refer to structures known to those skilled in the relevant art and are not intended to invoke 35 U.S.C. §112(f).

[0031] The following discussion is described with reference to the accompanying drawings. Figures 1 to 9 However, it should be understood that the present disclosure is not limited to these embodiments, and all changes and / or equivalents or replacements thereof also belong to the scope of the present disclosure.

[0032] As discussed above, language models based on transformer architectures, such as BERT, provide good performance in many NLP tasks. At their core, these models are built by stacking multiple transformer layers on top of each other in a sequential manner. The input passes through each transformer layer, where the output of one transformer layer is used as the input to the layer above it.

[0033] It has been found that the ability of these transformer-based language models to solve tasks and adapt to new NLP tasks is directly proportional to the number of transformer layers stacked on top of each other. This has led to the development and adoption of deeper language models with a larger number of transformer layers stacked together (such as the GPT-3 model includes 96 transformer layers). However, deeper language models have some disadvantages, such as slower training times and weaker loss signal propagation to the initial layers. In addition, the lower layers of the language model capture phrase-level information, while the middle layers capture syntactic information. However, this information is lost or diluted in the upper layers of the language model. Therefore, it would be advantageous to make this rich phrase-level syntactic information available to the upper layers of the language model.

[0034] There have been some attempts to introduce residual connections within the Transformer architecture, but they have been limited in scope and results. For example, one such technique adds the pre-SoftMax attention scores from layer L-2 to the current layer L. While this technique provides a path for direct loss propagation, it does not add any rich grammatical, phrase-level information from the underlying layers. Furthermore, this technique is simply a naive addition of attention scores from previous layers to provide a direct path for loss propagation.

[0035] The present disclosure provides various techniques for using residual transformers in natural language processing. As described in more detail below, the disclosed systems and methods provide various high-level architectures that can help make the information extracted at the lower layers of the language model available to the upper layers, while also using residual connections to solve the above problems. The disclosed embodiments provide rich grammatical text features for the upper layers of the language model, which can be used for better task solving capabilities and better task specific optimizations. This can lead to improved language understanding and reduced confusion. This can also lead to faster convergence, less computation, and reduced training costs.

[0036] Note that while some of the embodiments discussed below are described in the context of use in a consumer electronic device (such as a smart phone), this is merely one example, and it will be appreciated that the principles of the present disclosure may be implemented in any number of other suitable contexts and using any suitable devices.

[0037] Figure 1 An example network configuration 100 including electronic devices according to the present disclosure is shown. Figure 1 The embodiment of the network configuration 100 shown is for illustration only. Other embodiments of the network configuration 100 may be used without departing from the scope of the present disclosure.

[0038] According to an embodiment of the present disclosure, an electronic device 101 is included in a network configuration 100. The electronic device 101 may include at least one of a bus 110, a processor 120, a memory 130, an input / output (I / O) interface 150, a display 160, a communication interface 170, or a sensor 180. In some embodiments, the electronic device 101 may exclude at least one of these components, or may add at least one other component. The bus 110 includes circuits for connecting components 120-180 to each other and for transmitting communications (such as control messages and / or data) between components.

[0039] The processor 120 includes one or more processing devices, such as one or more microprocessors, microcontrollers, digital signal processors (DSPs) application specific integrated circuits (ASICs) field programmable gate arrays (FPGAs). In some embodiments, the processor 120 includes one or more of a central processing unit (CPU), an application processor (AP), a communication processor (CP), or a graphics processor unit (GPU). The processor 120 is capable of performing control on at least one other component of the electronic device 101 and / or performing operations or data processing related to communication or other functions. As described in more detail below, the processor 120 can perform one or more operations for using a residual transformer in natural language processing.

[0040] The memory 130 may include volatile and / or non-volatile memory. For example, the memory 130 may store commands or data related to at least one other component of the electronic device 101. According to an embodiment of the present disclosure, the memory 130 may store software and / or programs 140. The programs 140 include, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and / or an application program (or "application") 147. At least a portion of the kernel 141, the middleware 143, or the API 145 may be represented as an operating system (OS).

[0041] The kernel 141 can control or manage system resources (such as bus 110, processor 120 or memory 130) for executing operations or functions implemented in other programs (such as middleware 143, API 145 or application 147). The kernel 141 provides an interface that allows the middleware 143, API 145 or application 147 to access the various components of the electronic device 101 to control or manage system resources. The application 147 can support one or more functions for using the residual transformer in natural language processing. These functions can be performed by a single application or multiple applications, each of which performs one or more of these functions. For example, the middleware 143 can be used as a relay to allow the API 145 or application 147 to communicate data with the kernel 141. Multiple applications 147 can be provided. The middleware 143 is capable of controlling work requests received from the application 147, such as by assigning a priority to at least one of the multiple applications 147 using the system resources (such as bus 110, processor 120 or memory 130) of the electronic device 101. The API 145 is an interface that allows the application 147 to control functions provided from the kernel 141 or the middleware 143. For example, the API 145 includes at least one interface or function (such as a command) for archive control, window control, image processing, or text control.

[0042] The I / O interface 150 serves as an interface that can, for example, transmit commands or data input from a user or other external devices to other components of the electronic device 101. The I / O interface 150 can also output commands or data received from other components of the electronic device 101 to the user or other external devices.

[0043] The display 160 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, an organic light emitting diode (QLED) display, a micro-electromechanical system (MEMS) display, or an electronic paper display. The display 160 may also be a depth perception display, such as a multi-focal display. The display 160 is capable of displaying, for example, various contents (such as text, images, videos, icons, or symbols) to the user. The display 160 may include a touch screen and may receive, for example, a touch, gesture, proximity, or hover input using an electronic pen or a body part of the user.

[0044] For example, the communication interface 170 can establish communication between the electronic device 101 and an external electronic device (such as the first electronic device 102, the second electronic device 104, or the server 106). For example, the communication interface 170 can be connected to the network 162 or 164 through wireless or wired communication to communicate with the external electronic device. The communication interface 170 can be a wired or wireless transceiver or any other component for sending and receiving signals.

[0045] Wireless communication can use, for example, long term evolution (LTE), advanced long term evolution (LTE-A), fifth generation wireless system (5G), millimeter wave or 60GHz wireless communication, wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro) or global system for mobile communications (GSM) as a cellular communication protocol. Wired connection can include, for example, at least one of universal serial bus (USB), high-definition multimedia interface (HDMI), recommended standard 232 (RS-232) or plain old telephone service (POTS). Network 162 or 164 includes at least one communication network, such as a computer network (such as a local area network (LAN) or a wide area network (WAN)), the Internet or a telephone network.

[0046] The electronic device 101 also includes one or more sensors 180, which can measure physical quantities or detect the activation state of the electronic device 101 and convert the measured or detected information into electrical signals. For example, one or more sensors 180 include one or more cameras or other imaging sensors for capturing images of scenes. The sensor 180 may also include one or more buttons for touch input, gesture sensors, gyroscopes or gyroscope sensors, air pressure sensors, magnetic sensors or magnetometers, acceleration sensors or accelerometers, grip sensors, proximity sensors, color sensors (such as red, green, and blue (RGB) sensors), biophysical sensors, temperature sensors, humidity sensors, illumination sensors, ultraviolet (UV) sensors, electromyogram (EMG) sensors, electroencephalogram (EEG) sensors, electrocardiogram (ECG) sensors, infrared (IR) sensors, ultrasonic sensors, iris sensors, or fingerprint sensors. The sensor 180 may also include an inertial measurement unit, which may include one or more accelerometers, gyroscopes, and other components. In addition, the sensor 180 may include a control circuit for controlling at least one sensor included here. Any of these sensors 180 may be located in the electronic device 101.

[0047] The first external electronic device 102 or the second external electronic device 104 may be a wearable device or a wearable device (such as an HMD) in which the electronic device can be installed. When the electronic device 101 is installed in the electronic device 102 (such as an HMD), the electronic device 101 may communicate with the electronic device 102 through the communication interface 170. The electronic device 101 may be directly connected to the electronic device 102 to communicate with the electronic device 102 without involving a separate network. The electronic device 101 may also be an augmented reality wearable device including one or more imaging sensors, such as glasses.

[0048] The first external electronic device 102 and the second external electronic device 104 and the server 106 can each be a device of the same or different type as the electronic device 101. According to certain embodiments of the present disclosure, the server 106 includes a group of one or more servers. In addition, according to certain embodiments of the present disclosure, all or some of the operations performed on the electronic device 101 can be performed on another one or more other electronic devices (such as the electronic devices 102 and 104 or the server 106). In addition, according to certain embodiments of the present disclosure, when the electronic device 101 should automatically or upon request perform a certain function or service, the electronic device 101 can request another device (such as the electronic devices 102 and 104 or the server 106) to perform at least some functions associated with it, rather than performing the function or service alone or additionally. Other electronic devices (such as the electronic devices 102 and 104 or the server 106) are capable of performing the requested function or additional function and transmitting the execution result to the electronic device 101. The electronic device 101 can provide the requested function or service by processing the received result as is or additionally. To this end, for example, cloud computing, distributed computing or client-server computing technology can be used. Although Figure 1 The electronic device 101 is shown to include a communication interface 170 to communicate with an external electronic device 104 or a server 106 via a network 162 or 164 , but according to some embodiments of the present disclosure, the electronic device 101 may operate independently without a separate communication function.

[0049] The server 106 may include components 110-180 (or a suitable subset thereof) that are the same or similar to the electronic device 101. The server 106 may support driving the electronic device 101 by performing at least one of the operations (or functions) implemented on the electronic device 101. For example, the server 106 may include a processing module or processor that may support the processor 120 implemented in the electronic device 101. As described in more detail below, the server 106 may perform one or more operations to support techniques for using a residual transformer in natural language processing.

[0050] although Figure 1 An example of a network configuration 100 including an electronic device 101 is shown, but the Figure 1 Various changes may be made. For example, network configuration 100 may include any number of each component in any suitable arrangement. In general, computing and communication systems have a wide variety of configurations, and Figure 1 The scope of the present disclosure is not limited to any particular configuration. Figure 1 One operating environment is shown in which the various features disclosed in this patent document may be used, but these features may be used in any other suitable system.

[0051] As described above, residual connections can be used in transformer architectures to smooth loss conditions, thereby reducing training time and computational costs. Residual connections can achieve these benefits by allowing features extracted at lower level layers to be accessed by higher level layers. The embodiments described below include a high-level transformer language model architecture that uses residual connections to extract phrase-level linguistic syntactic features at lower levels. Therefore, these rich features extracted at lower levels can be accessed by higher levels, while also providing faster training of the model. The high-level transformer language model architecture described herein includes the following:

[0052] Residual Transformer Architecture. This architecture consists of simple residual connections across transformer layers. In this architecture, each nth transformer layer receives the residual outputs from the first to n-1th transformer layers.

[0053] Vertical residual attention structure. Instead of adding residuals at each layer, this architecture uses attention layers to decide which features from lower layers are meaningful for the current layer. For each token, vertical residual attention focuses on the same token from the layer below it.

[0054] Vertical-Horizontal Residual Attention Structure. In this architecture, the residual attention focuses on all tokens from all layers below it.

[0055] Hyper-attention architecture. In this architecture, the self-attention mechanism in the transformer layer is modified to pay attention to all tokens at the current transformer layer and also pay attention to all tokens extracted from the layer below it.

[0056] Dense-attention architecture. This architecture modifies the current transformer layer, which allows information to flow from multiple heads to the next layer. At each layer, each attention head focuses on all the labeled outputs from multiple heads in the previous layer.

[0057] The high-level transformer language model architecture described herein may also include one or more layer embeddings or head embeddings. These embeddings enhance the output from each transformer layer with information indicating which layer or head the embedding belongs to. For example, the layer embedding from transformer layer 3 includes information indicating that the output is from transformer layer 3. Layer embeddings may be used in any of the architectures described herein. In some embodiments, layer embeddings may be encoded using an addition operation. For example, at layer n, the addition operation between the layer output and the layer embedding for that particular layer may be given by the following equation:

[0058] FinalOutputOfLayer n =OutputOfLayer n +LayerEmbeddings n

[0059] Throughout this patent document, when referring to outputs from a layer, it is assumed that the layer outputs can be encoded using a layer embedding, such as described by the above equation.

[0060] Residual Transformer Architecture

[0061] Figure 2 An example residual transformer 200 according to the present disclosure is shown. For ease of explanation, the residual transformer 200 is described as using the above Figure 1 The residual transformer 200 is implemented by one or more components of the network configuration 100 (such as the electronic device 101). However, this is just an example, and the residual transformer 200 can be implemented using any other suitable (one or more) devices and in any other suitable (one or more) systems.

[0062] like Figure 2 As shown in , the residual transformer 200 includes a plurality of (N) transformer layers 201a-201n arranged in sequence. Here, N is any suitable integer greater than 1. The residual transformer 200 includes connections 205 between adjacent transformer layers 201a-201n for providing information, such as embedding vectors (as described in more detail below). In addition, the residual transformer 200 includes residual connections 210 between non-adjacent transformer layers 201a-201n. Specifically, each of the transformer layers 201a-201n has a residual connection 210 to each non-adjacent transformer layer 201a-201n. For example, the transformer layer 201a, which is the first transformer layer in the sequence, has a residual connection 210 to each of the transformer layers 201c-201n. The transformer layer 201b, which is the second transformer layer in the sequence, has a residual connection 210 to each of the transformer layers 201d-201n.

[0063] The transformer layer 201a receives a plurality of embedding vectors 202 as inputs to the residual transformer 200. The embedding vectors 202 represent tokens corresponding to the transformer inputs. In some embodiments, the tokens may correspond to natural language utterances (such as "Where is my cat?") to be processed using the residual transformer 200. Of course, this is just an example, and the tokens may correspond to other types of information.

[0064] The transformer layer 201a processes the embedding vector 202 of each token to generate a plurality of output embedding vectors representing the token. Figure 2 As shown, the output embedding vector includes: an output embedding vector 215, which is provided as input to the next transformer layer 201b; and a residual embedding vector 220, which is provided as input to each of the higher non-adjacent transformer layers 201c-201n.

[0065] Subsequent transformer layers 201b-201n receive output embedding vectors from previous transformer layers 201a-201n, including output embedding vectors 215 from adjacent previous transformer layers 201a-201n, and in the case of transformer layers 201c-201n, including residual embedding vectors 220 from non-adjacent previous transformer layers 201a-201n.

[0066] Each transformer layer 201b-201n determines an input embedding vector for each received token based on a combination of the output embedding vector 215 received from the previous transformer layer 201a-201n and the residual embedding vector 220. Each transformer layer 201b-201n then processes the input embedding vector for each token to generate an output embedding vector to be provided to each subsequent transformer layer.

[0067] Figure 3 Further details of an example transformer layer 300 according to the present disclosure are shown. The transformer layer 300 may represent one of the transformer layers 201a to 201n in the residual transformer 200.

[0068] like Figure 3 As shown, the transformer layer 300 includes a multi-head attention block 302, an addition and normalization block 304, a feedforward network 306, and an addition and normalization block 308. The multi-head attention block 302 calculates the query, key, and value of each of the multiple heads in parallel. The query, key, and value are used to calculate the attention weights of the multiple heads. Then, a weighted average is performed to obtain a contextual representation of each token, which is then connected and projected using a linear layer. The addition and normalization blocks 304 and 308 perform layer addition and layer normalization. The feedforward network 306 is a fully connected layer that receives the output from the addition and normalization block 304.

[0069] In contrast to conventional transformer architectures in which each transformer layer (n) receives input only from one adjacent previous layer (n-1), transformer layer 300 (which may represent layer n) receives input from both the adjacent previous layer (n-1) and other non-adjacent previous layers (layers 1 to n-2). In other words, each transformer layer n receives output from transformer layer 1 to transformer layer n-1. Specifically, Figure 3 As shown, the transformer layer 300 receives the output embedding vector 215 from the adjacent previous layer n-1, which is provided as input to the multi-head attention block 302. The transformer layer 300 also receives the residual embedding vector 220 from the non-adjacent previous layer (layers 1 to n-2). Then, the addition and normalization block 304 adds and normalizes the outputs from layers 1 to n-1 (such as using layer normalization). This can be represented by the following formula:

[0070] Input to layer n

[0071] Including residual embedding vectors 220 from non-adjacent previous layers provides faster training and convergence times in the transformer layer 300, thereby enabling easier training of deeper networks and easier improvements in performing natural language tasks.

[0072] although Figure 2 and Figure 3 An example and related details of the residual transformer 200 are shown, but the Figure 2 and Figure 3 For example, although the residual transformer 200 is described using various examples of components and operations, other embodiments may include other components and / or other operations. In addition, although a specific sequence of operations is shown, Figure 2 and Figure 3 The various operations shown in can overlap, occur in parallel, occur in a different order, or occur any number of times (including zero). As a specific example, although Figure 3 The transformer layer 300 is shown receiving the residual embedding vector 220 in the addition and normalization block 304, but the transformer layer 300 may alternatively receive the residual embedding vector 220 at the input of the transformer layer 300 before the multi-head attention block 302.

[0073] Residual Attention Structure

[0074] Figure 4 An example residual attention transformer 400 according to the present disclosure is shown. For ease of explanation, the residual attention transformer 400 is described as using the above Figure 1 The residual attention transformer 400 may be implemented by one or more components of the network configuration 100 (e.g., the electronic device 101). However, this is merely an example, and the residual attention transformer 400 may be implemented using any other suitable (one or more) devices and in any other suitable (one or more) systems.

[0075] like Figure 4 As shown, the residual attention transformer 400 includes many components that are the same as or similar to the corresponding components of the residual transformer 200. For example, the residual attention transformer 400 includes a plurality of (N) transformer layers 201a-201n arranged in sequence, connections 205 between adjacent transformer layers 201a-201n for providing embedding vectors, and residual connections 210 between non-adjacent transformer layers 201a-201n.

[0076] The residual attention transformer 400 also includes a residual attention layer 402 for each of the higher transformer layers 201b-201n. In contrast to the residual transformer 200 in which the residual is added at each of the higher transformer layers 201b-201n (without any decision about which features to include), the residual attention layer 402 for a particular transformer layer 201b-201n decides which features from the lower transformer layers 201a-201n are meaningful to the corresponding transformer layer 201b-201n. The residual attention layer 402 of the nth transformer layer 201b-201n uses the outputs from the lower transformer layers 201a-201n (1~n-1) to calculate a weighted sum, which is then provided as input to the nth transformer layer 201b-201n. As described in more detail below, the weighted sum is based on the embedding vector 215 from the adjacent previous transformer layer (n-1) and the residual embedding vector 220 from the non-adjacent previous transformer layer (1~n-2), which is modified by a set of attention weights for the embedding vectors 215 and 220. Therefore, instead of simply adding residual connections across transformer layers, the residual attention transformer 400 includes dynamic decisions made through residual attention, allowing higher transformer layers 201b-201n to choose how much information to pick up from the transformer layers 201a-201n below it.

[0077] As with the residual transformer 200, the residual embedding vector 220 can be introduced to each transformer layer 201a-201n of the residual attention transformer 400 at the multi-head attention block 302 or at the input of the transformer layer 201a-201n (i.e., before the multi-head attention block 302).

[0078] The residual attention transformer 400 can be configured according to a vertical residual attention transformer architecture or a vertical-horizontal residual attention transformer architecture. The difference between these two architectures lies in the way the residual attention is performed, which will now be described.

[0079] Vertical Residual Attention

[0080] In the vertical residual attention transformer architecture, for each token at the current transformer layer 201a-201n, the corresponding residual attention layer 402 attends to the representations of the same token from all previous transformer layers 201a-201n below the current layer. For example, for token t of layer l, the residual attention layer 402 computes the vertical residual attention Attention l,t as follows:

[0081] Attention l,t (Q,K,V)=SoftMax(QK T )V

[0082] in,

[0083]

[0084] where w represents the learned attention weights for each of the K (key), Q (query), and V (value) projections; e 1:l-1,t denotes the embedding of token t from layer 1 to layer l-1; and e l-1,t represents the embedding from layer l-1 for token t.

[0085] Figure 5 An example 500 of vertical residual attention in the residual attention transformer 400 according to the present disclosure is shown. Figure 5 As shown, the vertical residual attention 502 of the label k 504 at layer L is calculated using the label k at each layer in the previous layers 1 to L-1. Similarly, the vertical residual attention 506 of the label m 508 at layer L is calculated using the label m at each layer in the previous layers 1 to L-1. Therefore, in the residual attention transformer 400, by using the vertical residual attention based on l,t The set of attention weights in calculates the weighted sum of the set of output embedding vectors 215 from the previous layer to determine the resulting embedding vector 215.

[0086] Vertical-Horizontal Residual Attention

[0087] The vertical-horizontal residual attention transformer architecture is similar to the vertical residual attention transformer architecture, but the difference from the vertical residual attention transformer architecture is that the vertical-horizontal residual attention transformer architecture does not only focus on the same tags from all layers below, but focuses on all tags from all layers below. That is, in the vertical-horizontal residual attention transformer architecture, for a current transformer layer 201a-201n, the corresponding residual attention layer 402 focuses on all tags from all previous transformer layers 201a-201n below the current layer.

[0088] For example, for layer l and tag t, the residual attention layer 402 calculates the vertical-horizontal residual attention as follows l,t :

[0089] Attention l,t (Q,K,V)=SoftMax(QK T )V

[0090] in,

[0091]

[0092] where w represents the learned attention weight for each of the K, Q, and V projections; e 1:l-1,[T] represents the embedding from layer 1 to layer l-1 for all tokens [0, 1, ...T]; and e l-1,t represents the embedding for layer l-1 and tag t. Therefore, in the residual attention transformer 400, by using the vertical-horizontal residual attention l,t All sets of attention weights in the transformer 400 calculate the weighted sum of all sets of output embedding vectors 215 from previous layers to determine the resulting embedding vector 215.

[0093] although Figure 4 and Figure 5 An example of a residual attention transformer 400 and related details is shown, but may be Figure 4 and Figure 5 Various changes may be made. For example, although the residual attention transformer is described using various examples of components and operations, other embodiments may include other components and / or other operations. Furthermore, although a specific sequence of operations is shown, Figure 4 and Figure 5 The various operations shown in may overlap, occur in parallel, occur in a different order, or occur any number of times (including zero).

[0094] Hyper-attention architecture

[0095] Modifications to the transformer 200 may be introduced to facilitate a hyper-attention architecture. Specifically, the self-attention operation performed by the multi-head attention block 302 of each transformer layer 201a-201n may be modified as follows.

[0096] In a regular transformer, multi-head self-attention is performed at a given transformer layer n using only the output of the previous adjacent transformer layer (n-1), as follows:

[0097] MultiHead(Q,K,V)=Concat(head 1 ,head 2 ,…,head h )

[0098] Among them, head i =Attention(Q i ,K i ,V i )

[0099]

[0100] In the above equation, X represents the output from the previous adjacent transformer layer (ie, layer n-1).

[0101] In the hyper-attention architecture disclosed herein, for each token in a given transformer layer n, the multi-head attention block 302 not only focuses on all tokens in X (i.e., the outputs from the previous adjacent transformer layer n-1), but also focuses on the output tokens of all layers below n-1 (i.e., n-2, n-3, etc.). Therefore, for transformer layer n, self-attention is performed for each of the multiple heads to determine the weighted sum of the output embedding vectors of all tokens from all previous transformer layers for the same head. The weighted sums of the multiple heads are then combined. This can be represented by the following equation:

[0102] HyperMultiHead(Q,K,V)=Concat(head 1 ,head 2 ,…,head h )W O

[0103] Among them, head i =Attention(Q i ,K i ,V i )

[0104]

[0105] Y = Concat(X n-1 ,X n-2 ,…,X 0 )

[0106] In the above equation, W O represents the projection / feedforward layer, and Xk represents the labeled output from the transformer layer K.

[0107] Figure 6 An example 600 of hyper-attention that may be performed in the converter 200 according to the present disclosure is shown. Figure 6 As shown, hyper-attention 602 of label k 604 at layer n is calculated using all labels 606 of transformer layer n-1 and all labels 608 at each of the previous layers 1 to n-2.

[0108] Hyper-attention architecture with learnable downward weights

[0109] In some embodiments, the hyper-attention architecture can be modified to allow the multi-head attention block 302 of each transformer layer 201a-201n to jointly reduce the weight of the attention attributed to non-adjacent previous transformer layers. Downward weighting reduces the influence of non-adjacent previous transformer layers so that the attention mechanism is not overtaken in importance by the residual connections from those layers. For example, for transformer layer l, the attention attributed to non-adjacent previous transformer layers 1 to l-2 can be reduced by downward weighting. The attention attributed to the adjacent previous transformer layer l-1 is not weighted downward.

[0110] To achieve downward weighting, a predetermined learnable bias parameter b can be subtracted from the attention score QKT corresponding to the residual connection as follows:

[0111]

[0112] in,

[0113]

[0114] where w represents the attention weight of each of the K, Q, and V projections, and e l,t denotes the embedding of token t at layer l, and b denotes the bias parameter.

[0115] Dense-attention architecture

[0116] The dense-attention architecture is an extension of the hyper-attention architecture described above. The dense-attention architecture allows for easy information flow across multiple heads. To this end, the dense-attention architecture modifies the super-multi-head technique defined in the hyper-attention architecture to accommodate queries over all token embeddings from all heads of previous layers. Therefore, for a given transformer layer, self-attention is performed on each of the multiple heads to determine a weighted sum of the output embedding vectors of all tokens from all previous transformer layers for each of the multiple heads. The weighted sums of the multiple heads are then combined. For example, the super-multi-head equation shown above can be modified for dense-attention as follows:

[0117] DenseMultiHead(Q,K,V)=Concat(head 1 W 1 ,head 2 W 2 ,…,head h W h )

[0118] Among them, head i =Attention(Q i ,K i ,V i )

[0119]

[0120] here, represents the labeled output of multiple heads from transformer layer n. Note that W defined in the super-multi-head equation O The projection / feed-forward layers have been removed from the dense multi-head equation. Figure 7 An example 700 of dense-attention according to the present disclosure is shown. Figure 7 As shown, dense attention 702 for a tag k 704 at layer n focuses on all previous tags from previous layers 706 and multiple heads 708.

[0121] To achieve intensive attention, Figure 3 The transformer layer 300 is modified to have multiple feed-forward networks. For example, Figure 8 An example transformer layer 800 for performing dense attention according to the present disclosure is shown. The transformer layer 800 may represent one of the transformer layers 201a-201n in the residual transformer 200.

[0122] like Figure 8 As shown, the transformer layer 800 includes a multi-head attention block 802, a plurality of feed-forward networks 804a-804n, and an addition and normalization block 806. The multi-head attention block 802 is similar to Figure 3 Here, multiple heads 0 to n of the multi-head attention block 802 are shown. Unlike the transformer layer 300, the transformer layer 800 does not include an addition and normalization block immediately following the multi-head attention block 802. In addition, instead of a single feed-forward network, the transformer layer 800 includes multiple feed-forward networks 804a-804n, one for each head. These differences between the transformer layer 800 and the transformer layer 300 facilitate the flow of information from multiple heads to the next transformer layer in the dense-attention architecture.

[0123] The dense-attention architecture can also use multi-head embeddings, which indicates from which head the token embedding is received. Similar to the layer embeddings, the multi-head embeddings are learned and can be added to the key at the transformer layer as follows:

[0124] Keys i =Keys i +HeadEmbeddings i

[0125] Here, Keys i For Head i Keys for all tokens in the self-attention mechanism, and HeadEmbeddings i For Head i Learned head embeddings.

[0126] although Figures 6 to 8 shows an example of using the residual transformer in natural language processing and related details, but can be used for Figures 6 to 8 For example, although described as involving a specific sequence of operations, Figures 6 to 8 The various operations of the described techniques may overlap, occur in parallel, occur in a different order, or occur any number of times (including zero). Figures 6 to 8 The specific operations shown in are examples only and may be performed using other techniques. Figures 6 to 8 Each operation shown in .

[0127] Note that Figures 2 to 8 shown in or about Figures 2 to 8 The described operations and functions may be implemented in any suitable manner in the electronic device 101, the server 106, or other devices. For example, in some embodiments, one or more software applications or other software instructions executed by the processor 120 of the electronic device 101, the server 106, or other devices may be used to implement or support the operation of the electronic device 101, the server 106, or other devices. Figures 2 to 8 shown in or about Figures 2 to 8 In other embodiments, dedicated hardware components may be used to implement or support the operations and functions described in Figures 2 to 8 shown in or about Figures 2 to 8 In general, any suitable hardware or any suitable combination of hardware and software / firmware instructions may be used to implement at least some of the operations and functions described in the Figures 2 to 8 shown in or about Figures 2 to 8 Describes the operation and functionality.

[0128] Fig. 9 An example method 900 for using a residual transformer in natural language processing according to the present disclosure is shown. For ease of explanation, Fig. 9 The method 900 shown in FIG. 1 is described as involving Figure 1 The electronic device 101 and Figures 2 to 8 However, Fig. 9 The method 900 shown in can be used with any other suitable device(s) and architecture(s).

[0129] like Fig. 9As shown, at operation 901, an embedding vector representing a token in an input is provided to a transformer comprising a plurality of transformer layers arranged in sequence. Each transformer layer has a residual connection to each previous transformer layer. This may include, for example, electronic device 101 providing an embedding vector 202 representing a token of a natural language utterance as input to transformer 200 or transformer 400.

[0130] At operation 903, for a given transformer layer i, an input embedding vector is determined for each token. Each input embedding vector is based on a combination of output embedding vectors from previous transformer layers. This can include, for example, electronic device 101 determining an input embedding vector for each token for one of transformer layers 201a-201n, such as transformer layer 201c. The input embedding vector is based on a combination of output embedding vectors 215 and 220 from previous transformer layers, such as transformer layer 201a and transformer layer 201b.

[0131] At operation 905, for transformer layer i, for the first token, the input embedding vector for each token is processed to generate an output embedding vector to be provided to each subsequent transformer layer. This may include, for example, the electronic device 101 processing the input embedding vector for each token to generate an output embedding vector 215 (which is provided to an adjacent subsequent transformer layer (such as transformer layer 201d)) and a residual embedding vector 220 (which is provided to a non-adjacent subsequent transformer layer (such as a layer above transformer layer 201d)).

[0132] At operation 907, it is determined whether there are additional layers in the transformer to be processed. This may include, for example, electronic device 101 determining whether i is less than N (the number of transformer layers in transformer 200). If there are additional layers in the transformer, method 900 moves to operation 909, where the transformer layer index i is incremented by 1, and method 900 returns to operation 903 to process the next transformer layer. Otherwise, method 900 moves to operation 911.

[0133] At operation 911 , an output is generated from the transformer. This may include, for example, the electronic device 101 generating a solution to a natural language task from the transformer 200 , where the solution is based on a natural language utterance input to the transformer 200 .

[0134] although Fig. 9 An example of a method 900 for using a residual transformer in natural language processing is shown, but may be used for Fig. 9 For example, although shown as a series of steps, Fig. 9 The various steps in may overlap, occur in parallel, occur in a different order, or occur any number of times.

[0135] According to various embodiments, the electronic device may include at least one imaging sensor and / or at least one processor.

[0136] At least one imaging sensor may capture multiple image frames of a scene.

[0137] At least one processor may be described as at least one processing device.

[0138] At least one processor may receive input data. The input data may be described as input information. The input data may correspond to "input" of "providing embedding vectors representing tokens in the input to a transformer comprising a plurality of transformer layers arranged in sequence".

[0139] At least one processor may process input data to obtain output data.

[0140] The input data may include at least one of text information, voice information, or image information.

[0141] For example, based on input data including text information, at least one processor may directly process the text information.

[0142] For example, based on input data including voice information, at least one processor may transform the voice information into text information corresponding to the voice information. At least one processor may process the voice information to obtain text information corresponding to the voice information.

[0143] For example, based on input data including image information, at least one processor may identify text information corresponding to the image information. The image information may include a frame of a scene. The frame may include information indicating text. The at least one processor may process the image information to obtain text information corresponding to the image information.

[0144] The input data may include at least one tag.

[0145] At least one processor may obtain at least one embedding vector indicating a token (or at least one token) in the input data based on the input data. The token may be described as a text unit, a text element, or a text fragment. The embedding vector may be described as "at least one embedding vector".

[0146] The at least one processor may provide at least one embedded vector representation in the input data to a transformer including a plurality of transformer layers arranged in sequence. In addition, repeated explanations will be omitted.

[0147] The at least one processor may obtain output at least one embedding vector by processing the at least one embedding vector. The at least one processor may obtain output data based on the output at least one embedding vector.

[0148] At least one processor can obtain output data expected by a user (or model). At least one processor can obtain output data by processing input data. The output data corresponds to the input data.

[0149] The at least one embedding vector may be described as an “embedding vector” or “embedding vectors”.

[0150] Note that various embodiments of the present disclosure can be applied in a variety of natural language processing use cases, including machine translation, text summarization, information retrieval, question answering systems, sentence and text representation, text classification, named entity recognition, part-of-speech extraction, slot filling, etc. The disclosed embodiments can improve performance in these and other use cases, and can also achieve faster training times, which leads to more efficient use of resources.

[0151] Although the present disclosure has been described with reference to various exemplary embodiments, various changes and modifications may be suggested to one skilled in the art. The present disclosure is intended to encompass such changes and modifications as fall within the scope of the appended claims.

Claims

1. A method comprising: providing an embedding vector representing a token in an input to a transformer comprising a plurality of transformer layers arranged in sequence, each transformer layer having a residual connection to each previous transformer layer; as well as For each transformer layer: For a first tag, determining an input embedding vector based on a combination of output embedding vectors from previous transformer layers; and The input embedding vector is processed for a first token to generate an output embedding vector to be provided to each subsequent transformer layer.

2. The method according to claim 1, wherein: Determining an input embedding vector for a first token includes: receiving a first set of output embedding vectors from a previous transformer layer, each output embedding vector in the first set being output by one of the previous transformer layers for the first token; determining a first set of attention weights for a first set of output embedding vectors; and The input embedding vector is determined by computing a weighted sum of the first set of output embedding vectors based on a first set of attention weights.

3. The method according to claim 2, wherein: Determining the input embedding vector for the first token further includes: For each additional token in the input that is different from the first token: receiving a second set of output embedding vectors from a previous transformer layer, each output embedding vector in the second set being output by one of the previous transformer layers for the other token; and determining a second set of attention weights for a second set of output embedding vectors; and The input embedding vector is determined by computing a weighted sum of the first set of output embedding vectors and the second set of output embedding vectors based on the first set of attention weights and the second set of attention weights.

4. The method according to claim 1, wherein: The transducer includes a plurality of heads; and Determining the input embedding vector for the first token further includes: For a given transformer layer: performing self-attention on each of the plurality of heads to determine a weighted sum of output embedding vectors of all tokens from all previous transformer layers for the same head; and The weighted sums of the plurality of heads are combined.

5. The method according to claim 4, wherein: The self-attention attributed to at least one of the previous transformer layers is weighted downward using a bias parameter to reduce the influence of the at least one previous transformer layer.

6. The method according to claim 1, wherein: The transducer includes a plurality of heads; and Determining the input embedding vector for the first token further includes: For a given transformer layer: performing self-attention on each of the plurality of heads to determine a weighted sum of output embedding vectors of all tokens from all previous transformer layers for the same head and each other head in the plurality of heads; and The weighted sums of the plurality of heads are combined.

7. The method according to claim 6, wherein: The converter includes a feed-forward network for each of the plurality of heads.

8. The method according to claim 1, wherein: Each output embedding vector includes information indicating which transformer layer the output embedding vector is output from.

9. An electronic device, comprising: At least one processing device configured to: providing an embedding vector representing a token in an input to a transformer comprising a plurality of transformer layers arranged in sequence, each transformer layer having a residual connection to each previous transformer layer; and For each transformer layer: For a first tag, determining an input embedding vector based on a combination of output embedding vectors from previous transformer layers; as well as The input embedding vector is processed for a first token to generate an output embedding vector to be provided to each subsequent transformer layer.

10. The electronic device according to claim 9, wherein: To determine the input embedding vector for the first token, the at least one processing device is configured to: receiving a first set of output embedding vectors from a previous transformer layer, each output embedding vector in the first set being output by one of the previous transformer layers for the first token; determining a first set of attention weights for a first set of output embedding vectors; as well as The input embedding vector is determined by computing a weighted sum of a first set of output embedding vectors based on a first set of attention weights.

11. The electronic device according to claim 10, wherein: To determine the input embedding vector for the first token, the at least one processing device is further configured to: For each additional token in the input that is different from the first token: receiving a second set of output embedding vectors from a previous transformer layer, each output embedding vector in the second set being output by one of the previous transformer layers for the other token; as well as determining a second set of attention weights for a second set of output embedding vectors; as well as The input embedding vector is determined by computing a weighted sum of the first set of output embedding vectors and the second set of output embedding vectors based on the first set of attention weights and the second set of attention weights.

12. The electronic device according to claim 9, wherein: The converter includes a plurality of heads; and To determine the input embedding vector for the first token, the at least one processing device is configured to: For a given transformer layer: performing self-attention on each of the plurality of heads to determine a weighted sum of output embedding vectors of all tokens from all previous transformer layers for the same head; and The weighted sums of the plurality of heads are combined.

13. The electronic device according to claim 12, wherein: The self-attention attributed to at least one of the previous transformer layers is weighted downward using a bias parameter to reduce the influence of the at least one previous transformer layer.

14. The electronic device according to claim 9, wherein: The converter includes a plurality of heads; and To determine the input embedding vector for the first token, the at least one processing device is configured to: For a given transformer layer: performing self-attention on each of the plurality of heads to determine, for the same head and each other head in the plurality of heads, a weighted sum of output embedding vectors of all tokens from all previous transformer layers; and The weighted sums of the plurality of heads are combined.

15. The electronic device according to claim 14, wherein: The converter includes a feed-forward network for each of the plurality of heads.

Images