Methods, systems, and computer program products for simplifying transducers for sequential recommendation

Abstract

Methods, systems, and computer program products can simplify a transducer machine learning model for sequential recommendation via a softmax-free gated attention mechanism, and / or can use a gating unit to further sparse attention, which operation can simplify attention distribution and reduce the negative effects of noisy items.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 347,166, filed May 31, 2022, the disclosure of which is hereby incorporated by reference in its entirety. Background Art 1. Technical Field

[0003] The present disclosure relates to Transformer machine learning models, and in some non-limiting embodiments or aspects, to methods, systems, and computer program products for simplifying Transformer machine learning models for sequential recommendations via a softmax-free gated attention mechanism. 2. Background technology

[0005] The Transformer is a deep learning model that uses a mechanism called self-attention to weight the importance of each part of the input data differently. Summary of the invention

[0006] Therefore, improved systems, devices, products, apparatuses and / or methods for simplifying a transformer for sequential recommendation are provided. For example, a non-limiting embodiment or aspect can address the weakness of the transformer when processing noisy sequences. As an example, a non-limiting embodiment or aspect can use a gated attention layer that enables the use of a weaker single-headed attention with minimal quality loss and / or a probabilistic sparsification framework that can further prune noise and achieve high-quality sparse attention.

[0007] According to some non-limiting embodiments or aspects, a method is provided, comprising: receiving an input sequence having a corresponding input at each input position in a plurality of input positions in an input order; processing the input sequence by an encoder neural network to generate a corresponding encoded representation of each of the inputs in the input sequence, the encoder neural network comprising a sequence of one or more encoder sub-networks, each encoder sub-network configured to receive a corresponding encoder sub-network input for each of the plurality of input positions, and generate a corresponding encoder sub-network output for each of the plurality of input positions, and each encoder sub-network comprising: an encoder gated attention layer, the encoder gated attention layer configured to receive a sub-network input for each of the plurality of input positions, and for each of the plurality of input positions in the input order; For each specific input position: applying a gated attention mechanism to the encoder sub-network inputs at multiple input positions to generate a corresponding output for the specific input position, wherein applying the gated attention mechanism includes: determining a shared representation based on the encoder sub-network inputs at the multiple input positions, determining a query for the specific input position based on the shared representation, determining keys for the multiple input positions based on the shared representation, determining values ​​for the multiple input positions based on the shared representation, and generating a corresponding output for the specific input position using the determined queries, keys, and values; processing the encoded representation through a decoder neural network to generate an output sequence having a corresponding output at each output position in a plurality of output positions in an output order; and providing an output sequence having a corresponding output at each output position in a plurality of output positions in an output order.

[0008] In some non-limiting embodiments or aspects, the shared representation is determined by applying trainable variables to encoder sub-network inputs at multiple input positions.

[0009] In some non-limiting embodiments or aspects, a query for a particular input location is determined by applying a query transform to a shared representation, wherein keys for multiple input locations are determined by applying a key transform to the shared representation, and wherein values ​​for multiple input locations are determined by applying a value transform to the shared representation.

[0010] In some non-limiting embodiments or aspects, the gated attention layer generates a corresponding output for a specific input position by: (i) determining a corresponding compatibility output for the specific input position by applying a compatibility function between a query for the specific input position and keys generated for multiple input positions, and (ii) applying the learned difference to the compatibility output for the specific input position to generate a corresponding attention value for the specific input position.

[0011] In some non-limiting embodiments or aspects, the gated attention layer generates a corresponding output for a particular input position by applying a rectified linear unit (ReLU) activation function to the corresponding attention value for the particular input position and values ​​for multiple input positions.

[0012] In some non-limiting embodiments or aspects, each encoder sub-network is trained according to an objective function that depends on the learned differences and the desired attention capabilities.

[0013] According to some non-limiting embodiments or aspects, a system is provided, comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to implement a sequence transduction neural network, the sequence transduction neural network for transducing an input sequence into an output sequence, the input sequence having a corresponding network input at each input position in a plurality of input positions in the input order, the output sequence having a corresponding network output at each output position in the plurality of output positions in the output order, the sequence transduction neural network comprising: an encoder neural network configured to receive the input sequence and generate a corresponding encoded representation of each of the inputs in the input sequence, the encoder neural network comprising a sequence of one or more encoder sub-networks, each encoder sub-network configured to receive a corresponding encoder sub-network input for each of the plurality of input positions and generate a corresponding encoder sub-network output for each of a plurality of input positions, and each encoder sub-network comprises: an encoder gated attention layer configured to receive a sub-network input for each of a plurality of input positions, and for each specific input position in the input order: applying a gated attention mechanism to the encoder sub-network inputs at the plurality of input positions to generate a corresponding output for the specific input position, wherein applying the gated attention mechanism comprises: determining a shared representation based on the encoder sub-network inputs at the plurality of input positions, determining a query for the specific input position based on the shared representation, determining a key for the plurality of input positions based on the shared representation, determining a value for the plurality of input positions based on the shared representation, and generating a corresponding output for the specific input position using the determined query, key, and value; and a decoder neural network configured to receive the encoded representation and generate an output sequence.

[0014] In some non-limiting embodiments or aspects, the shared representation is determined by applying trainable variables to encoder sub-network inputs at multiple input positions.

[0015] In some non-limiting embodiments or aspects, a query for a particular input location is determined by applying a query transform to a shared representation, wherein keys for multiple input locations are determined by applying a key transform to the shared representation, and wherein values ​​for multiple input locations are determined by applying a value transform to the shared representation.

[0016] In some non-limiting embodiments or aspects, the gated attention layer generates a corresponding output for a specific input position by: (i) determining a corresponding compatibility output for the specific input position by applying a compatibility function between a query for the specific input position and keys generated for multiple input positions, and (ii) applying the learned difference to the compatibility output for the specific input position to generate a corresponding attention value for the specific input position.

[0017] In some non-limiting embodiments or aspects, the gated attention layer generates a corresponding output for a particular input position by applying a rectified linear unit (ReLU) activation function to the corresponding attention value for the particular input position and values ​​for multiple input positions.

[0018] In some non-limiting embodiments or aspects, each encoder sub-network is trained according to an objective function that depends on the learned differences and the desired attention capabilities.

[0019] According to some non-limiting embodiments or aspects, there is provided a computer program product comprising a non-transitory computer-readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to: receive an input sequence having a corresponding input at each input position in a plurality of input positions in an input order; process the input sequence through an encoder neural network to generate a corresponding encoded representation of each of the inputs in the input sequence, the encoder neural network comprising a sequence of one or more encoder sub-networks, each encoder sub-network being configured to receive a corresponding encoder sub-network input for each input position in the plurality of input positions, and to generate a corresponding encoder sub-network output for each input position in the plurality of input positions, and each encoder sub-network comprising: an encoder gated attention layer configured to receive a corresponding encoder sub-network input for each input position in the plurality of input positions, and to generate a corresponding encoder sub-network output for each input position in the plurality of input positions; subnetwork inputs for each of a plurality of input positions, and for each particular input position in an input order: applying a gated attention mechanism to the encoder subnetwork inputs at the plurality of input positions to generate a corresponding output for the particular input position, wherein applying the gated attention mechanism comprises: determining a shared representation based on the encoder subnetwork inputs at the plurality of input positions, determining a query for the particular input position based on the shared representation, determining keys for the plurality of input positions based on the shared representation, determining values ​​for the plurality of input positions based on the shared representation, and generating a corresponding output for the particular input position using the determined queries, keys, and values; processing the encoded representations through a decoder neural network to generate an output sequence having a corresponding output at each of the plurality of output positions in an output order; and providing an output sequence having a corresponding output at each of the plurality of output positions in an output order.

[0020] In some non-limiting embodiments or aspects, the shared representation is determined by applying trainable variables to encoder sub-network inputs at multiple input positions.

[0021] In some non-limiting embodiments or aspects, a query for a particular input location is determined by applying a query transform to a shared representation, wherein keys for multiple input locations are determined by applying a key transform to the shared representation, and wherein values ​​for multiple input locations are determined by applying a value transform to the shared representation.

[0022] In some non-limiting embodiments or aspects, the gated attention layer generates a corresponding output for a specific input position by: (i) determining a corresponding compatibility output for the specific input position by applying a compatibility function between a query for the specific input position and keys generated for multiple input positions, and (ii) applying the learned difference to the compatibility output for the specific input position to generate a corresponding attention value for the specific input position.

[0023] In some non-limiting embodiments or aspects, the gated attention layer generates a corresponding output for a particular input position by applying a rectified linear unit (ReLU) activation function to the corresponding attention value for the particular input position and values ​​for multiple input positions.

[0024] In some non-limiting embodiments or aspects, each encoder sub-network is trained according to an objective function that depends on the learned differences and the desired attention capabilities.

[0025] Additional embodiments or aspects are set forth in the following numbered clauses:

[0026] Clause 1. A method comprising: receiving an input sequence having a corresponding input at each input position in a plurality of input positions in an input order; processing the input sequence through an encoder neural network to generate a corresponding encoded representation of each of the inputs in the input sequence, the encoder neural network comprising a sequence of one or more encoder sub-networks, each encoder sub-network configured to receive a corresponding encoder sub-network input for each input position in the plurality of input positions and generate a corresponding encoder sub-network output for each input position in the plurality of input positions, and each encoder sub-network comprising: an encoder gated attention layer, the encoder gated attention layer configured to receive the sub-network input for each input position in the plurality of input positions, and for each particular input position in the input order: The encoder sub-network input at a position is applied with a gated attention mechanism to generate a corresponding output for the specific input position, wherein applying the gated attention mechanism includes: determining a shared representation based on the encoder sub-network input at the multiple input positions, determining a query for the specific input position based on the shared representation, determining keys for the multiple input positions based on the shared representation, determining values ​​for the multiple input positions based on the shared representation, and using the determined queries, keys and values ​​to generate the corresponding output for the specific input position; processing the encoded representation through a decoder neural network to generate an output sequence having a corresponding output at each output position in a plurality of output positions in an output order; and providing the output sequence having the corresponding output at each output position in the plurality of output positions in the output order.

[0027] Clause 2. A method as described in Clause 1, wherein the shared representation is determined by applying trainable variables to the encoder sub-network input at the multiple input positions.

[0028] Clause 3. A method as described in clause 1 or 2, wherein the query for the specific input location is determined by applying a query transformation to the shared representation, wherein the key for the multiple input locations is determined by applying a key transformation to the shared representation, and wherein the values ​​for the multiple input positions are determined by applying a value transformation to the shared representation.

[0029] Clause 4. A method as described in any of clauses 1 to 3, wherein the gated attention layer generates the corresponding output for the specific input position by: (i) determining the corresponding compatibility output for the specific input position by applying a compatibility function between the query for the specific input position and the keys generated for the multiple input positions, and (ii) applying the learned difference to the compatibility output for the specific input position to generate a corresponding attention value for the specific input position.

[0030] Clause 5. A method as described in any of Clauses 1 to 4, wherein the gated attention layer generates the corresponding output for the specific input position by applying a rectified linear unit (ReLU) activation function to the corresponding attention value for the specific input position and the values ​​for the multiple input positions.

[0031] Clause 6. A method as described in any of clauses 1 to 5, wherein each encoder sub-network is trained according to an objective function that depends on the learned differences and the desired attention capacity.

[0032] Clause 7. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement a sequence transduction neural network for transducing an input sequence into an output sequence, the input sequence having a corresponding network input at each input position in a plurality of input positions in an input order, the output sequence having a corresponding network output at each output position in a plurality of output positions in an output order, the sequence transduction neural network comprising: an encoder neural network configured to receive the input sequence and generate a corresponding encoded representation of each of the inputs in the input sequence, the encoder neural network comprising a sequence of one or more encoder sub-networks, each encoder sub-network configured to receive a corresponding encoder sub-network input for each of the plurality of input positions and generate a corresponding encoder sub-network input for each of the plurality of input positions Corresponding encoder sub-network outputs, and each encoder sub-network includes: an encoder gated attention layer, the encoder gated attention layer being configured to receive the sub-network input for each input position in the multiple input positions, and for each specific input position in the input order: applying a gated attention mechanism to the encoder sub-network input at the multiple input positions to generate a corresponding output for the specific input position, wherein applying the gated attention mechanism includes: determining a shared representation based on the encoder sub-network input at the multiple input positions, determining a query for the specific input position based on the shared representation, determining a key for the multiple input positions based on the shared representation, determining a value for the multiple input positions based on the shared representation, and generating the corresponding output for the specific input position using the determined query, key and value; and a decoder neural network, the decoder neural network being configured to receive the encoded representation and generate the output sequence.

[0033] Clause 8. A system as described in Clause 7, wherein the shared representation is determined by applying trainable variables to the encoder sub-network input at the plurality of input positions.

[0034] Clause 9. A system as described in clause 7 or 8, wherein the query for the particular input position is determined by applying a query transform to the shared representation, wherein the keys for the multiple input positions are determined by applying a key transform to the shared representation, and wherein the values ​​for the multiple input positions are determined by applying a value transform to the shared representation.

[0035] Clause 10. A system as described in any of clauses 7 to 9, wherein the gated attention layer generates the corresponding output for the specific input position by: (i) determining the corresponding compatibility output for the specific input position by applying a compatibility function between the query for the specific input position and the keys generated for the multiple input positions, and (ii) applying the learned difference to the compatibility output for the specific input position to generate a corresponding attention value for the specific input position.

[0036] Clause 11. A system as described in any of clauses 7 to 10, wherein the gated attention layer generates the corresponding output for the specific input position by applying a rectified linear unit (ReLU) activation function to the corresponding attention value for the specific input position and the values ​​for the multiple input positions.

[0037] Clause 12. A system as described in any of clauses 7 to 11, wherein each encoder sub-network is trained according to an objective function that depends on the learned differences and the desired attention capacity.

[0038] Clause 13. A computer program product comprising a non-transitory computer-readable medium, the non-transitory computer-readable medium comprising program instructions, the program instructions, when executed by at least one processor, cause the at least one processor to: receive an input sequence having a corresponding input at each input position in a plurality of input positions in an input order; process the input sequence through an encoder neural network to generate a corresponding encoded representation of each of the inputs in the input sequence, the encoder neural network comprising a sequence of one or more encoder sub-networks, each encoder sub-network being configured to receive a corresponding encoder sub-network input for each input position in the plurality of input positions and generate a corresponding encoder sub-network output for each input position in the plurality of input positions, and each encoder sub-network comprising: an encoder gated attention layer, the encoder gated attention layer being configured to receive the sub-network input for each input position in the plurality of input positions network inputs, and for each specific input position in the input order: applying a gated attention mechanism to the encoder sub-network inputs at the multiple input positions to generate a corresponding output for the specific input position, wherein applying the gated attention mechanism comprises: determining a shared representation from the encoder sub-network inputs at the multiple input positions, determining a query for the specific input position from the shared representation, determining a key for the multiple input positions from the shared representation, determining a value for the multiple input positions from the shared representation, and generating the corresponding output for the specific input position using the determined query, key, and value; processing the encoded representation through a decoder neural network to generate an output sequence having a corresponding output at each output position in a plurality of output positions in an output order; and providing the output sequence having the corresponding output at each output position in the plurality of output positions in the output order.

[0039] Clause 14. A computer program product as described in Clause 13, wherein the shared representation is determined by applying trainable variables to the encoder sub-network input at the plurality of input positions.

[0040] Clause 15. A computer program product as described in clause 13 or 14, wherein the query for the specific input location is determined by applying a query transformation to the shared representation, wherein the keys for the multiple input locations are determined by applying a key transformation to the shared representation, and wherein the values ​​for the multiple input positions are determined by applying a value transformation to the shared representation.

[0041] Clause 16. A computer program product as described in any of clauses 13 to 15, wherein the gated attention layer generates the corresponding output for the specific input position by: (i) determining the corresponding compatibility output for the specific input position by applying a compatibility function between the query for the specific input position and the keys generated for the multiple input positions, and (ii) applying the learned difference to the compatibility output for the specific input position to generate a corresponding attention value for the specific input position.

[0042] Clause 17. A computer program product as described in any of clauses 13 to 16, wherein the gated attention layer generates the corresponding output for the specific input position by applying a rectified linear unit (ReLU) activation function to the corresponding attention value for the specific input position and the values ​​for the multiple input positions.

[0043] Clause 18. A computer program product as described in any of clauses 13 to 17, wherein each encoder sub-network is trained according to an objective function that depends on the learned difference and the desired attention capacity.

[0044] These and other features and characteristics of the present disclosure, as well as methods of operation and functions of combinations of related structural elements and parts and economies of manufacture, will become more apparent when considering the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals indicate corresponding parts in the various figures. However, it should be expressly understood that the various figures are for illustration and description purposes only and are not intended as definitions of limitations. Unless the context clearly dictates otherwise, when used in this specification and claims, the singular forms "a", "an", and "the" include plural referents. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Additional advantages and details are explained in more detail below with reference to exemplary embodiments shown in the schematic drawings, in which:

[0046] Figure 1 are diagrams of non-limiting embodiments or aspects of environments in which the systems, devices, products, apparatuses, and / or methods described herein may be implemented;

[0047] Figure 2 yes Figure 1 Illustrations of non-limiting embodiments or aspects of one or more devices and / or components of one or more systems;

[0048] Figure 3 is a flow chart of a non-limiting embodiment or aspect for simplifying a process for sequentially recommending a converter;

[0049] Figure 4A is an illustration of a model architecture of a non-limiting embodiment or aspect of a converter;

[0050] Figure 4B is an illustration of a model architecture of a non-limiting embodiment or aspect of a gated attention network (GAN);

[0051] Figure 5 It is a statistical table of data for the example experiment;

[0052] Figure 6 is a table of performance results for the example experiments; and

[0053] FIG. 7 is a diagram of a model architecture of an existing converter. DETAILED DESCRIPTION

[0054] It should be understood that, except where expressly specified to the contrary, the present disclosure may employ various alternative variations and step sequences. It should also be understood that the specific devices and processes shown in the accompanying drawings and described in the following specification are merely exemplary and non-limiting embodiments or aspects. Therefore, specific dimensions and other physical characteristics associated with the embodiments or aspects disclosed herein should not be considered limiting.

[0055] The aspects, parts, elements, structures, actions, steps, functions, instructions, etc. used herein should not be understood as being critical or necessary unless explicitly described as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items, and can be used interchangeably with "one or more" and "at least one". In addition, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, a combination of related items and unrelated items, etc.), and can be used interchangeably with "one or more" or "at least one". In the case of intending to have only one item, the term "one" or similar language is used. Moreover, as used herein, the term "having" and / or its analogs are intended to be open terms. In addition, unless otherwise explicitly stated, the phrase "based on" is intended to mean "based at least in part on".

[0056] As used herein, the term "communication" may refer to the reception, acceptance, transmission, transfer, provision, etc. of data (e.g., information, signals, messages, instructions, commands, etc.). A unit (e.g., a device, a system, a component of a device or system, a combination thereof, and / or the like) communicating with another unit means that the unit is able to directly or indirectly receive information from the other unit and / or send information to the other unit. This may refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection, and / or the like) that is wired and / or wireless in nature. In addition, although the information sent may be modified, processed, relayed, and / or routed between the first unit and the second unit, the two units may also communicate with each other. For example, even if the first unit passively receives information and does not actively send information to the second unit, the first unit may communicate with the second unit. As another example, if at least one intermediate unit processes the information received from the first unit and transmits the processed information to the second unit, the first unit may communicate with the second unit.

[0057] Obviously, the systems and / or methods described herein can be implemented in different forms of hardware, software, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0058] Some non-limiting embodiments or aspects are described herein in conjunction with threshold values. As used herein, satisfying a threshold value may refer to a value greater than a threshold value, more than a threshold value, above a threshold value, greater than or equal to a threshold value, less than a threshold value, less than a threshold value, below a threshold value, less than or equal to a threshold value, equal to a threshold value, etc.

[0059] As used herein, the term "transaction service provider" may refer to an entity that receives transaction authorization requests from merchants or other entities and, in some cases, provides payment assurance through an agreement between the transaction service provider and the issuer organization. For example, a transaction service provider may include, for example The term "transaction processing system" may refer to one or more computing devices operated by or on behalf of a transaction service provider, such as a transaction processing server executing one or more software applications. A transaction processing system may include one or more processors and, in some non-limiting embodiments, may be operated by or on behalf of a transaction service provider.

[0060] As used herein, the term "account identifier" may include one or more primary account numbers (PANs), tokens, or other identifiers associated with a customer account. The term "token" may refer to an identifier used as a replacement or substitute identifier for an original account identifier such as a PAN. An account identifier may be alphanumeric, or any combination of characters and / or symbols. A token may be associated with a PAN or other original account identifier in one or more data structures (e.g., one or more databases, etc.) so that the token can be used to conduct transactions without directly using the original account identifier. In some examples, an original account identifier such as a PAN may be associated with multiple tokens for different individuals or purposes.

[0061] As used herein, the term "issuer institution", "portable financial device issuer", "issuer" or "issuer bank" may refer to one or more entities that provide one or more accounts to a user (e.g., a customer, consumer, organization, etc.) to conduct a transaction (e.g., a payment transaction), such as initiating a credit card payment transaction and / or a debit card payment transaction. For example, an issuer institution may provide an account identifier such as a PAN to a user that uniquely identifies one or more accounts associated with the user. The account identifier may be embodied on a portable financial device such as a physical financial instrument (e.g., a payment card), and / or may be electronic and used for electronic payments. In some non-limiting embodiments or aspects, the issuer institution may be associated with a bank identification number (BIN) that uniquely identifies the issuer institution. As used herein, the term "issuer institution system" may refer to one or more computer systems operated by or on behalf of an issuer institution, such as a server computer that executes one or more software applications. For example, an issuer institution system may include one or more authorization servers for authorizing payment transactions.

[0062] As used herein, the term "merchant" may refer to an individual or entity that provides goods and / or services or access to goods and / or services to a user (e.g., a customer) based on a transaction (e.g., a payment transaction). As used herein, the term "merchant" or "merchant system" may also refer to one or more computer systems, computing devices, and / or software applications operated by or on behalf of a merchant, such as a server computer that executes one or more software applications. As used herein, a "point of sale (POS) system" may refer to one or more computers and / or peripherals used by a merchant to conduct payment transactions with a user, including one or more card readers, near field communication (NFC) receivers, radio frequency identification (RFID) receivers, and / or other contactless transceivers or receivers, contact-based receivers, payment terminals, computers, servers, input devices, and / or other similar devices that can be used to initiate payment transactions. A POS system may be part of a merchant system. A merchant system may also include a merchant plug-in for facilitating online Internet-based transactions through a merchant webpage or software application. A merchant plug-in may include software that runs on a merchant server or is hosted by a third party to facilitate such online transactions.

[0063] As used herein, the term "mobile device" may refer to one or more portable electronic devices configured to communicate with one or more networks. As an example, a mobile device may include a cellular phone (e.g., a smart phone or a standard cellular phone), a portable computer (e.g., a tablet computer, a laptop computer, etc.), a wearable device (e.g., a watch, glasses, lenses, clothing, etc.), a personal digital assistant (PDA), and / or other similar devices. As used herein, the terms "client device" and "user device" refer to any electronic device configured to communicate with one or more servers or remote devices and / or systems. Client devices or user devices may include mobile devices, network-enabled appliances (e.g., network-enabled TVs, refrigerators, thermostats, etc.), computers, POS systems, and / or any other device or system capable of communicating with a network.

[0064] As used herein, the term "computing device" may refer to one or more electronic devices configured to process data. In some examples, a computing device may include the necessary components to receive, process, and output data, such as a processor, a display, a memory, an input device, a network interface, and / or the like. A computing device may be a mobile device. For example, a mobile device may include a cellular phone (e.g., a smartphone or a standard cellular phone), a portable computer, a wearable device (e.g., a watch, glasses, lenses, clothing, and / or the like), a PDA, and / or other similar devices. A computing device may also be a desktop computer or other form of non-mobile computer.

[0065] As used herein, the term "payment device" may refer to a portable financial device, an electronic payment device, a payment card (e.g., a credit or debit card), a gift card, a smart card, smart media, a payroll card, a healthcare card, a wristband, a machine-readable medium containing account information, a key chain device or pendant, an RFID transponder, a retailer discount or membership card, a cellular phone, an electronic wallet mobile application, a PDA, a pager, a security card, a computer, an access card, a wireless terminal, a transponder, etc. In some non-limiting embodiments or aspects, a payment device may include volatile or non-volatile memory to store information (e.g., an account identifier, an account holder's name, etc.).

[0066] As used herein, the terms "server" and / or "processor" may refer to or include one or more computing devices that are operated by or facilitate communications and processing by multiple parties in a network environment such as the Internet, but it should be understood that communications may be facilitated through one or more public or private network environments, and that there may be various other arrangements. In addition, multiple computing devices (e.g., servers, POS devices, mobile devices, etc.) that communicate directly or indirectly in a network environment may constitute a "system." As used herein, references to "server" or "processor" may refer to previously described servers and / or processors, different servers and / or processors, and / or combinations of servers and / or processors that are stated to implement previous steps or functions. For example, as used in the specification and claims, a first server and / or first processor stated to perform a first step or function may refer to the same or different servers and / or processors stated to perform a second step or function.

[0067] As used herein, the term "acquirer" may refer to an entity that is licensed by a transaction service provider and / or approved by a transaction service provider to initiate a transaction using a portable financial device of a transaction service provider. An acquirer may also refer to one or more computer systems operated by or on behalf of an acquirer, such as a server computer (e.g., "acquirer server") that executes one or more software applications. An "acquirer" may be a merchant bank, or in some cases, a merchant system may be an acquirer. The transaction may include an original credit transaction (OCT) and an account funds transaction (AFT). A transaction service provider may authorize an acquirer to sign a merchant of a service provider to initiate a transaction using a portable financial device of a transaction service provider. An acquirer may sign a contract with a payment service provider to enable the service provider to sponsor a merchant. An acquirer may monitor the compliance of a payment service provider according to the regulations of a transaction service provider. An acquirer may perform due diligence on a payment service provider and ensure that appropriate due diligence is performed before signing a sponsored merchant. An acquirer may be responsible for all transaction service provider programs that they operate or sponsor. An acquirer can be held responsible for the actions of its payment facilitator and the merchants it or its payment facilitator sponsors.

[0068] As used herein, the term "payment gateway" may refer to an entity and / or a payment processing system operated by or on behalf of such an entity, the entity (e.g., merchant service provider, payment service provider, payment service provider, payment service provider with a contract with an acquirer, payment aggregator, etc.) providing payment services (e.g., transaction service provider payment services, payment processing services, etc.) to one or more merchants. The payment service may be associated with the use of a portable financial device managed by a transaction service provider. As used herein, the term "payment gateway system" may refer to one or more computer systems, computer devices, servers, server groups, etc. operated by or on behalf of a payment gateway.

[0069] As used herein, the term "authentication system" may refer to one or more computing devices that authenticate users and / or accounts, such as, but not limited to, transaction processing systems, merchant systems, issuer systems, payment gateways, third-party authentication services, etc.

[0070] As used herein, the terms "request," "response," "request message," and "response message" may refer to one or more messages, data packets, signals, and / or data structures used to transfer data between two or more components or units.

[0071] As used herein, the term "application programming interface" (API) may refer to computer code that allows communication between different systems or (hardware and / or software) system components. For example, an API may include function calls, functions, subroutines, communication protocols, fields, etc. that can be used and / or accessed by other systems or other (hardware and / or software) system components.

[0072] As used herein, the term "user interface" or "graphical user interface" refers to a generated display, such as one or more graphical user interfaces (GUIs), with which a user can interact directly or indirectly (e.g., via keyboard, mouse, touch screen, etc.).

[0073] Sequential recommendations aim to recommend the next item based on the user’s historical behavior (e.g., recommending the next item after the user purchases a smartphone). Headphones, etc.), which has received great attention in recommender systems. However, learning sequential user behavior can be challenging because users' choices of items generally depend on both long-term and short-term preferences. Early Markov chain models have been proposed to capture short-term item transitions by assuming that the user's next decision is derived from several previous behaviors, but ignoring long-term preferences. To alleviate this problem, many deep neural networks have achieved great success in modeling entire user sequences, including recurrent neural networks, convolutional neural networks, memory networks, and graph neural networks.

[0074] Recently, transformers have shown promising empirical results in various tasks, such as machine translation, time series prediction, and token-based object detection. A component of the transformer is a self-attention network, which can be quite efficient and is able to learn long-term dependencies by computing attention weights between each pair of objects in a sequence. Inspired by the success of transformers, several self-attention sequential recommendations have been proposed and achieve state-of-the-art performance. For example, SASRec, described in the paper titled “Self-attentive sequential recommendation” by Wang-Cheng Kang and Julian McAuley (2018) at ICDM'18, pp. 197-206 (arXiv:1808.09781), is a pioneering framework for learning the importance of items at different positions using a self-attention network, the disclosure of which is hereby incorporated by reference in its entirety. In Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and PengJiang (2019). BERT4Rec described in the paper entitled “BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer” in CIKM'19, 1441-1450 (arXiv:1904.06690) further models the correlation of items from both the left-to-right direction and the right-to-left direction, the disclosure of which is hereby incorporated by reference in its entirety. SSE-PT described in the paper entitled “SSE-PT: Sequential recommendation via personalized transformer” by Liwei Wu, Shuqing Li, Cho-Jui Hsieh, and James Sharpnack (2020) in RecSys, 328-337 (), is a personalized transformer model that provides better interpretability of engagement patterns by introducing user embeddings, the disclosure of which is hereby incorporated by reference in its entirety.

[0075] Despite the encouraging performance achieved, the robustness of sequential recommendations has been little studied in the field of recommender systems. For example, there may not be any strong assumptions about how the historical behavior of users should be generated in online shopping scenarios. In addition, real-world item sequences are usually noisy, containing both true positive interactions and false positive interactions. For example, part of the item may end up with a negative rating and / or being returned. Obviously, not every item in the sequence is consistent with the user's preferences, especially for implicit feedback. Unfortunately, existing self-attention networks are not Lipschitz continuous (for example, if changing the input of a function by a specific amount will not significantly change the output of the function, then the function can be Lipschitz continuous, etc.) and are susceptible to the quality of the input sequence. For example, it has been found that a large amount of BERT's attention is focused on less meaningful tokens (such as "[SEP]" and "."), which may cause misleading interpretations, and if the self-attention network is not well regularized for noisy sequences, suboptimal performance may result.

[0076] To address these problems, a straightforward strategy is to design sparse transformer architectures that sparse the connections in the attention layer, which has been actively studied in language modeling tasks. Several representative models are Star Transformer, Sparse Transformer, Longformer, and Big-Bird. These sparse attention patterns can alleviate the noise problem and avoid assigning credit to irrelevant content for queries of interest. However, these existing models rely heavily on pre-defined attention schemes that lack flexibility and adaptability in practice. Unlike end-to-end training methods, whether these sparse patterns can be well generalized to sequential recommendations remains unknown and remains an open research problem. Another work to address these issues focuses on dropout techniques for transformers, including LayerDrop, DropHead, and UniDrop. However, these random dropout methods are susceptible to bias, i.e., the fact that attention can be randomly dropped does not mean that the model allows these attentions to be dropped, which may lead to overly aggressive pruning problems.

[0077] Non-limiting embodiments or aspects of the present disclosure provide methods, systems, and computer program products for improving transformers for sequential recommendations. For example, a method according to a non-limiting embodiment or aspect may: receive an input sequence having a corresponding input at each input position in a plurality of input positions in an input order; process the input sequence through an encoder neural network to generate a corresponding encoded representation of each of the inputs in the input sequence, the encoder neural network comprising a sequence of one or more encoder sub-networks, each encoder sub-network being configured to receive a corresponding encoder sub-network input for each input position in a plurality of input positions, and generate a corresponding encoder sub-network output for each input position in a plurality of input positions, and each encoder sub-network comprising: an encoder gated attention layer, the encoder gated attention layer being configured to receive a sub-network input for each input position in a plurality of input positions, and for each input position in the input order Specific input position: applying a gated attention mechanism to the encoder sub-network inputs at multiple input positions to generate a corresponding output for the specific input position, wherein applying the gated attention mechanism includes: determining a shared representation based on the encoder sub-network inputs at the multiple input positions, determining a query for the specific input position based on the shared representation, determining keys for the multiple input positions based on the shared representation, determining values ​​for the multiple input positions based on the shared representation, and generating a corresponding output for the specific input position using the determined queries, keys, and values; processing the encoded representation through a decoder neural network to generate an output sequence having a corresponding output at each output position in a plurality of output positions in an output order; and providing an output sequence having a corresponding output at each output position in a plurality of output positions in an output order.

[0078] In this way, non-limiting embodiments or aspects of the present disclosure may provide denoising strategies (e.g., Rec-Denoiser, etc.) for better training self-attention sequential recommendations. For example, non-limiting embodiments or aspects recognize that not all attention is necessary, and simply pruning redundant attention can further improve performance. Non-limiting embodiments or aspects use differentiable masks to discard task-irrelevant attention in the self-attention layer, rather than randomly discarding attention, which can produce exact zero probabilities for noisy items. The sparsity introduced in the attention may have several benefits. It is possible to learn to discard attention that is irrelevant to the parameterized mask in a data-driven manner (e.g., reference Figure 4A and Figure 4B, the Rec-de-noiser or gated attention layer (GAN) 450 can prune the item sequence in the attention graph including phone, bag and headphones for pants and the item sequence including phone, bag, headphones and pants for laptop, etc.). For example, a non-limiting embodiment or aspect can explicitly seek the next item prediction based on a subset of more informative items. In addition, the Rec-de-noiser or gated attention layer can still make full use of the transformer because the Rec-de-noiser or gated attention layer does not need to change the overall transformer architecture, but only needs to change its attention distribution. As a result, Rec-Denoiser can be easily implemented and / or can be compatible with any transformer, thereby making the transformer less complex and improving the interpretability of the transformer.

[0079] However, the discreteness of the binary mask (e.g., 0 is discarded and 1 is retained) is intractable in back propagation. To solve this problem, non-limiting embodiments or aspects relax discrete variables by probabilistic reparameterization using continuous approximations, for example, as described in Chris J Maddison, Andriy Mnih, and Yee Whye The (2017) in ICLR entitled "The concrete distribution: A continuous relaxation of discrete random variables", the disclosure of which is hereby incorporated by reference in its entirety. Non-limiting embodiments or aspects may further provide an unbiased and low-variance gradient estimator to efficiently estimate the gradient of binary variables. Thus, the differentiable mask of non-limiting embodiments or aspects can be jointly trained with the original transformer in an end-to-end manner. In addition, as mentioned herein, the scaled dot product attention is not Lipschitz continuous and is therefore susceptible to input perturbations. Therefore, to further improve training stability, non-limiting embodiments or aspects may further apply Jacobian regularization as described in the following: "Robust learning with jacobian regularization" by Judy Hoffman, Daniel ARoberts, and Sho Yaida (2019) (arXivpreprint arXiv:1908.02729), "Improving dnn robustness to adversarial attacks using jacobian regularization" by Daniel Jakubovitz and Raja Giryes (2018) in ECCV, 514 to 529, the disclosures of which are hereby incorporated by reference in their entirety.

[0080] Reference now Figure 1 , Figure 1 1 is a diagram of an example environment 100 in which the devices, systems, methods, and / or products described herein may be implemented. Figure 1As shown in , environment 100 includes transaction processing network 101, user device 112 and / or communication network 116, and transaction processing network may include merchant system 102, payment gateway system 104, acquirer system 106, transaction service provider system 108, issuer system 110. Transaction processing network 101, merchant system 102, payment gateway system 104, acquirer system 106, transaction service provider system 108, issuer system 110 and / or user device 112 may be interconnected (e.g., establish connection for communication, etc.) via wired connection, wireless connection or a combination of wired connection and wireless connection.

[0081] The merchant system 102 may include one or more devices that are capable of receiving information and / or data from the payment gateway system 104, the acquirer system 106, the transaction service provider system 108, the issuer system 110, and / or the user device 112 (e.g., via a communication network 116, etc.) and / or transmitting information and / or data to the payment gateway system 104, the acquirer system 106, the transaction service provider system 108, the issuer system 110, and / or the user device 112 (e.g., via a communication network 116, etc.). The merchant system 102 may include a device that is capable of communicating with the user device 112 via a communication connection (e.g., an NFC communication connection, an RFID communication connection, The merchant system 102 may include a device that receives information and / or data from the user device 112 and / or transmits information and / or data to the user device 112 through the communication connection, communication connection, etc. For example, the merchant system 102 may include a computing device, such as a server, a server group, a client device, a client device group, and / or other similar devices. In some non-limiting embodiments or aspects, the merchant system 102 may be associated with a merchant described herein. In some non-limiting embodiments or aspects, the merchant system 102 may include one or more devices that can be used by a merchant to conduct payment transactions with a user, such as a computer, a computer system, and / or a peripheral device. For example, the merchant system 102 may include a POS device and / or a POS system.

[0082] The payment gateway system 104 may include one or more devices that are capable of receiving information and / or data from the merchant system 102, the acquirer system 106, the transaction service provider system 108, the issuer system 110, and / or the user device 112 (e.g., via a communication network 116, etc.), and / or transmitting information and / or data to the merchant system 102, the acquirer system 106, the transaction service provider system 108, the issuer system 110, and / or the user device 112 (e.g., via a communication network 116, etc.). For example, the payment gateway system 104 may include a computing device, such as a server, a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the payment gateway system 104 is associated with the payment gateway described herein.

[0083] The acquirer system 106 may include one or more devices capable of receiving information and / or data from the merchant system 102, the payment gateway system 104, the transaction service provider system 108, the issuer system 110, and / or the user device 112 (e.g., via a communication network 116, etc.) and / or transmitting information and / or data to the merchant system 102, the payment gateway system 104, the transaction service provider system 108, the issuer system 110, and / or the user device 112 (e.g., via a communication network 116, etc.). For example, the acquirer system 106 may include a computing device, such as a server, a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the acquirer system 106 may be associated with an acquirer as described herein.

[0084] The transaction service provider system 108 may include one or more devices that are capable of receiving information and / or data from the merchant system 102, the payment gateway system 104, the acquirer system 106, the issuer system 110, and / or the user device 112 (e.g., via a communication network 116, etc.), and / or transmitting information and / or data to the merchant system 102, the payment gateway system 104, the acquirer system 106, the issuer system 110, and / or the user device 112 (e.g., via a communication network 116, etc.). For example, the transaction service provider system 108 may include a computing device, such as a server (e.g., a transaction processing server, etc.), a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the transaction service provider system 108 may be associated with a transaction service provider described herein. In some non-limiting embodiments or aspects, the transaction service provider system 108 may include and / or access one or more internal and / or external databases that include transaction data.

[0085] The issuer system 110 may include one or more devices that are capable of receiving information and / or data from the merchant system 102, the payment gateway system 104, the acquirer system 106, the transaction service provider system 108, and / or the user device 112 (e.g., via a communication network 116, etc.), and / or transmitting information and / or data to the merchant system 102, the payment gateway system 104, the acquirer system 106, the transaction service provider system 108, and / or the user device 112 (e.g., via a communication network 116, etc.). For example, the issuer system 110 may include a computing device, such as a server, a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the issuer system 110 may be associated with an issuer organization described herein. For example, the issuer system 110 may be associated with an issuer organization that issues a payment account or instrument (e.g., a credit account, a debit account, a credit card, a debit card, etc.) to a user (e.g., a user associated with the user device 112, etc.).

[0086] In some non-limiting embodiments or aspects, the transaction processing network 101 includes multiple systems in a communication path for processing transactions. For example, the transaction processing network 101 may include a merchant system 102, a payment gateway system 104, an acquirer system 106, a transaction service provider system 108, and / or an issuer system 110 in a communication path (e.g., a communication path, a communication channel, a communication network, etc.) for processing electronic payment transactions. For example, the transaction processing network 101 may process (e.g., initiate, conduct, authorize, etc.) electronic payment transactions via the communication path between the merchant system 102, the payment gateway system 104, the acquirer system 106, the transaction service provider system 108, and / or the issuer system 110.

[0087] The user device 112 may include one or more devices that can receive information and / or data from the merchant system 102, the payment gateway system 104, the acquirer system 106, the transaction service provider system 108, and / or the issuer system 110 (e.g., via the communication network 116, etc.) and / or transmit information and / or data to the merchant system 102, the payment gateway system 104, the acquirer system 106, the transaction service provider system 108, and / or the issuer system 110 (e.g., via the communication network 116, etc.). For example, the user device 112 may include a client device, etc. In some non-limiting embodiments or aspects, the user device 112 can communicate with the merchant system 102, the payment gateway system 104, the acquirer system 106, the transaction service provider system 108, and / or the issuer system 110 via a short-range wireless communication connection (e.g., an NFC communication connection, an RFID communication connection, The user device 112 may receive information via a short-range wireless communication connection (e.g., from the merchant system 102, etc.), and / or transmit information via a short-range wireless communication connection (e.g., to the merchant system 102). In some non-limiting embodiments or aspects, the user device 112 may include an application associated with the user device 112, such as an application stored on the user device 112, a mobile application stored and / or executed on the user device 112 (e.g., a mobile device application, a native application for a mobile device, a mobile cloud application for a mobile device, an electronic wallet application, an issuer bank application, etc.). In some non-limiting embodiments or aspects, for one or more transactions in the payment network, the user device 112 may be associated with a sender account and / or a receiving account in the payment network.

[0088] The communication network 116 may include one or more wired and / or wireless networks. For example, the communication network 116 may include a cellular network (e.g., a long term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN), a private network, an ad hoc network, an intranet, the Internet, a fiber-optic-based network, a cloud computing network, etc., and / or a combination of these or other types of networks.

[0089] supply Figure 1 The number and arrangement of devices and systems shown are examples. Additional devices and / or systems, fewer devices and / or systems, different devices and / or systems, and / or devices and / or systems may be present. Figure 1 The devices and / or systems shown are arranged differently. In addition, the invention can be implemented in a single device and / or system. Figure 1 two or more devices and / or systems shown in, or Figure 1 The single device and / or system shown in the environment 100 may be implemented as multiple distributed devices and / or systems. In addition or alternatively, a group of devices and / or systems (e.g., one or more devices or systems) of the environment 100 may perform one or more functions described as being performed by another group of devices and / or systems of the environment 100.

[0090] Reference now Figure 2 , Figure 2is a diagram of example components of device 200. Device 200 may correspond to one or more devices of merchant system 102, one or more devices of payment gateway system 104, one or more devices of acquirer system 106, one or more devices of transaction service provider system 108, one or more devices of issuer system 110, and / or user device 112 (e.g., one or more devices of a system of user device 112, etc.). In some non-limiting embodiments or aspects, one or more devices of merchant system 102, one or more devices of payment gateway system 104, one or more devices of acquirer system 106, one or more devices of transaction service provider system 108, one or more devices of issuer system 110, and / or user device 112 (e.g., one or more devices of a system of user device 112, etc.) may include at least one device 200 and / or at least one component of device 200. Figure 2 As shown, device 200 may include a bus 202 , a processor 204 , a memory 206 , a storage component 208 , an input component 210 , an output component 212 , and a communication interface 214 .

[0091] The bus 202 may include components that permit communication between components of the device 200. In some non-limiting embodiments or aspects, the processor 204 may be implemented in hardware, software, or a combination of hardware and software. For example, the processor 204 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component that can be programmed to perform a function (e.g., a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc.). The memory 206 may include a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by the processor 204.

[0092] Storage component 208 may store information and / or software associated with the operation and use of device 200. For example, storage component 208 may include a hard disk (e.g., magnetic disk, optical disk, magneto-optical disk, solid-state disk, etc.), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a tape, and / or another type of computer-readable medium, and a corresponding drive.

[0093] Input components 210 may include components that permit device 200 to receive information, for example, via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, a microphone, etc.). Additionally or alternatively, input components 210 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.). Output components 212 may include components that provide output information from device 200 (e.g., a display, a speaker, one or more light emitting diodes (LEDs), etc.).

[0094] The communication interface 214 may include a transceiver-type component (e.g., a transceiver, a separate receiver and transmitter, etc.) that enables the device 200 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. The communication interface 214 may permit the device 200 to receive information from another device and / or provide information to another device. For example, the communication interface 214 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, Interface, cellular network interface, etc.

[0095] The device 200 may perform one or more processes described herein. The device 200 may perform these processes based on a processor 204 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), etc.) executing software instructions stored by a computer-readable medium such as a memory 206 and / or a storage component 208. A computer-readable medium (e.g., a non-transitory computer-readable medium) is defined herein as a non-transitory memory device. A non-transitory memory device includes a memory space located within a single physical storage device or a memory space spread across multiple physical storage devices.

[0096] The software instructions may be read into the memory 206 and / or storage component 208 from another computer-readable medium or from another device via the communication interface 214. When executed, the software instructions stored in the memory 206 and / or storage component 208 may cause the processor 204 to perform one or more processes described herein. Additionally or alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, the embodiments or aspects described herein are not limited to any specific combination of hardware circuitry and software.

[0097] The memory 206 and / or storage component 208 may include a data storage device or one or more data structures (e.g., a database, etc.). The device 200 can receive information from the data storage device or one or more data structures in the memory 206 and / or storage component 208, store information in the data storage device or one or more data structures, transmit information to the data storage device or one or more data structures, or search for information stored therein.

[0098] supply Figure 2 The number and arrangement of components shown in FIG. 2 are provided as examples. In some non-limiting embodiments or aspects, the device 200 may include a Figure 2 Components shown in the figure may include additional components, fewer components, different components, or differently arranged components than those shown in the figure. Additionally or alternatively, a group of components (e.g., one or more components) of the device 200 may perform one or more functions described as being performed by another group of components of the device 200.

[0099] Reference now Figure 3 , Figure 3 300 for maintaining model state. In some non-limiting embodiments or aspects, one or more steps in process 300 may be performed (e.g., in whole, in part, etc.) by transaction service provider system 108 (e.g., one or more devices of transaction service provider system 108). In some non-limiting embodiments or aspects, one or more steps in process 300 may be performed (e.g., in whole, in part, etc.) by another device or group of devices that is independent of or includes transaction service provider system 108, such as merchant system 102 (e.g., one or more devices of merchant system 102), payment gateway system 104 (e.g., one or more devices of payment gateway system 104), acquirer system 106 (e.g., one or more devices of acquirer system 106), issuer system 110 (e.g., one or more devices of issuer system 110), and / or user device 112 (e.g., one or more devices of system of user device 112).

[0100] like Figure 3 As shown in , at step 302, the process 300 includes receiving an input sequence having a corresponding input at each input position in a plurality of input positions in an input order. For example, the transaction service provider system 108 may receive an input sequence having a corresponding input at each input position in a plurality of input positions in an input order. As an example, for a sequential recommendation task, let U be a user set, V be an item set, and S = (S 1 ,S 2 ,…,S |U|) is a set of user behaviors, each user u∈U can be associated with the item sequence in chronological order associated, where |S u | is the length, and is the item that user u has interacted with at time t. In this example, the transaction service provider system 108 may receive As an input sequence having a corresponding input at each input position in a plurality of input positions in an input order.

[0101] The sequential recommendation can be evaluated as the next item prediction. For example, for each user u, the transaction service provider system 108 can be based on the interaction history S u To find the time step |S u +1|Predict the next project As described in more detail herein.

[0102] Items may include any type of items, such as clothing items, electronic items, electronic game items, movie items, image items, word or text items, etc. User interaction with an item may include any type of user behavior with respect to the item, such as a user's evaluation of the item, a user's use of the item (e.g., a user clicking on an item representation in a web page, a user downloading a digital item, a user replaying a movie, etc.), etc.

[0103] like Figure 3 As shown in , at step 304, the process 300 includes processing the input sequence through the encoder neural network to generate a corresponding encoded representation of each of the inputs in the input sequence. For example, the transaction service provider system 108 can process the input sequence through the encoder neural network to generate a corresponding encoded representation of each of the inputs in the input sequence.

[0104] As previously described herein, due to efficient parallel training, transformers have been widely used for sequential recommendation, such as SASRec, BERT4Rec, TiSASRec, and Transformers4Rec. Referring also to FIG. 7 , which is a diagram of the model architecture of an existing transformer 700, the embedding layers (e.g., input embedding 702, output embedding 704, etc.), the transformer model (e.g., encoder 706, decoder 708, etc.), and the optimization of the existing transformer 700 are now described herein.

[0105] Transformer-based recommenders can maintain item embedding tables where d is the size of the embedding. For each sequence S u , the item embedding table can be converted into a fixed sequence (s1, s2, ..., s n), where n is the maximum length (e.g. by truncating or padding items to keep the most recent n items, etc.). n ) can be represented as an embedding that can be retrieved from the embedding table M

[0106] To preserve the temporal order of item sequences, we can further construct learnable position embeddings The item embedding and position embedding can be added as X = E + P. Synthetic Embedding Can be fed directly into any transformer-based recommender.

[0107] There is a transformer module at the top of the embedding layer, which includes a self-attention layer and a point-by-point feed-forward layer. For example, the transformer can have an encoder-decoder structure. The encoder 706 can map an input sequence of symbolic representations to a sequence of continuous representations. Given the continuous representation, the decoder 708 generates an output sequence of symbols one element at a time. At each step, the model is autoregressive, and when the next symbol is generated, the previously generated symbol is consumed as an additional input. The existing transformer 700 follows this overall architecture shown in the left and right halves of Figure 7, which uses stacked self-attention and point-by-point fully connected layers for both the encoder 706 and the decoder 708, respectively.

[0108] For example, the encoder 706 can be composed of a stack of N identical layers (e.g., N=6, etc.). Each layer can have two sublayers. The first sublayer can be a multi-head self-attention mechanism, and the second sublayer can be a simple position-by-position fully connected feedforward network. Residual connections can be used around each of the two sublayers, followed by layer normalization. That is, the output of each sublayer can be LayerNorm(x+Sublayer(x)), where Sublayer(x) is the function implemented by the sublayer itself. To facilitate these residual connections, all sublayers in the model and the embedding layer can produce outputs of the same dimension (e.g., 512, etc.).

[0109] For example, the decoder 708 can be composed of a stack of N identical layers (e.g., N=6, etc.). In addition to the two sub-layers in each encoder layer, the decoder 708 can also insert a third sub-layer that performs multi-head attention on the output of the encoder stack. Similar to the encoder, the decoder 708 can use residual connections around each sub-layer in the sub-layers, followed by layer normalization. The self-attention sub-layer in the decoder stack can be modified to prevent positions from paying attention to subsequent positions. This masking combined with the fact that the output embedding is offset by one position ensures that the prediction for a position in the input sequence can only rely on the known output at positions less than that position.

[0110] The self-attention layer can learn long-term dependencies within the sequence. The scaled dot product attention can be defined according to the following equation (1):

[0111]

[0112] in It is the output item representation; Q = XW Q , K = XW K and V=XW V They are query, key, and value respectively. are three projection matrices; and is a scaling factor. In practice, a multi-head self-attention layer and a point-by-point feed-forward layer may be employed to better capture item dependencies. For the sake of brevity, the details of the multi-head self-attention layer and the point-by-point feed-forward layer are omitted in this paper, and the transformer module or block of the existing transformer 700 is summarized as: (1) ←Transformer(H (1-1) ), where H (0) = X, because multi-head self-attention layers and point-wise feed-forward layers have been previously described in Wang-Cheng Kang and Julian McAuley (2018) in their paper “Self-attentive sequential recommendation” in ICDM 197–206 and in Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Kaiser and Illia Polosukhin (2017) in a paper titled “Attention is all you need”, NeurIPS 5998-6008, the disclosures of which are hereby incorporated by reference in their entirety.

[0113] After the stacked L transformer block, the next item (given the previous items) can be based on For example, the inner product can be used to predict the user's preference score for item i according to the following equation (2):

[0114]

[0115] in is the embedding of item i. The transformer can be fed with a sequence s = (s1, s2, …, s n ), and its expected output can be the same sequence o = (o1, o2, ..., o n), and the binary cross entropy loss can be used as the learning objective according to the following equation (3):

[0116]

[0117] where θ is the model parameter, corresponds to o t negative samples, and σ(·) is a S-type function.

[0118] A gated attention network (GAN) according to a non-limiting embodiment or aspect may provide a simpler yet more efficient layer than the existing transformer 700. Although the GAN may have quadratic complexity in sequence length, the GAN may be more suitable for detecting noisy sequences. For example and also with reference to Figure 4A , which is a diagram of a model architecture of a non-limiting embodiment or aspect of a transformer 400, which may include an embedding layer (e.g., input embedding 402, output embedding 404, etc.), an encoder 406, and / or a decoder 408. As an example, the transformer 400 may include a sequence transduction neural network for transducing an input sequence into an output sequence, the input sequence having a corresponding network input at each input position in a plurality of input positions in the input order, and the output sequence having a corresponding network output at each output position in a plurality of output positions in the output order. In such an example, the multi-head attention of the existing transformer 700 in the encoder 706 and the decoder 708 may be replaced with the GAN 450 in the encoder 406 and the decoder 408 of the transformer 400. It should be noted that, unless otherwise described herein, the architecture and functionality of the transformer 400 may be the same or similar to the architecture and functionality of the existing transformer 700; therefore, for the sake of brevity, the same or similar elements or functions of the transformer 400 previously described herein with respect to the existing transformer 700 may be omitted.

[0119] The encoder 406 (e.g., an encoder neural network, etc.) may include a sequence of one or more encoder subnetworks. Each encoder subnetwork may be configured to receive a corresponding encoder subnetwork input for each input position in a plurality of input positions, and to generate a corresponding encoder subnetwork output for each input position in a plurality of input positions. For example, the encoder 406 may be composed of a stack of N (e.g., N=6, etc.) identical encoder subnetworks or layers. Each layer may have two sublayers. The first sublayer may be a GAN 450, and the second sublayer may be a simple position-by-position fully connected feedforward network. Residual connections may be used around each of the two sublayers, followed by layer normalization. That is, the output of each sublayer may be LayerNorm(x+Sublayer(x)), where Sublayer(x) is a function implemented by the sublayer itself. To facilitate these residual connections, all sublayers in the model and the embedding layer may produce outputs of the same dimension (e.g., 512, etc.).

[0120] Also refer to Figure 4B , which is a diagram of the model architecture of GAN 450, each encoder sub-network may include an encoder gated attention layer (e.g., GAN 450, etc.), which is configured to receive a sub-network input for each input position in a plurality of input positions, and for each specific input position in the input order: applying a gated attention mechanism to the encoder sub-network input at the plurality of input positions to generate a corresponding output for the particular input position. Applying the gated attention mechanism may include: determining a shared representation Z based on the encoder sub-network input at the plurality of input positions, determining a query Q for the particular input position based on the shared representation, determining a key K for the plurality of input positions based on the shared representation, determining a value V for the plurality of input positions based on the shared representation, and using the determined query, key, and value to generate a corresponding output for the particular input position.

[0121] A shared representation Z may be determined by applying trainable variables to encoder subnetwork inputs at multiple input positions. A query Q for a particular input position may be determined by applying a query transform to the shared representation Z. A key K for multiple input positions may be determined by applying a key transform to the shared representation Z. A value V for multiple input positions may be determined by applying a value transform to the shared representation Z.

[0122] The GAN 450 may generate a corresponding output for a specific input position by: (i) determining a corresponding compatibility output for the specific input position by applying a compatibility function between a query Q for the specific input position and a key K generated for a plurality of input positions, and (ii) applying the learned difference M to the compatibility output for the specific input position to generate a corresponding attention value A for the specific input position. The GAN 450 may generate a corresponding output for the specific input position by applying a rectified linear unit (ReLU) activation function to the corresponding attention value A for the specific input position and the values ​​for the plurality of input positions.

[0123] For example, given an input X, a GAN 450 according to a non-limiting embodiment or aspect may be defined according to the following equation (4):

[0124]

[0125] where Z is a shared representation (s< <d) 3 , S q (·)and S k (·)and S v (·) is a cheap transformation that applies a per-dim scalar and an offset to Z (e.g., similar to a learnable variable in LayerNorms, etc.), M is a learnable difference that can be used to sparsify the attention map (e.g., a trainable variable, etc.), and Q, K, V and A are query, key, value and attention, respectively, and the softmax in the existing transformer 700 is simplified to the ReLU activation function in the transformer 400 according to a non-limiting implementation or aspect.

[0126] Each encoder sub-network can be trained according to an objective function that depends on the learned difference M and the desired attention capacity K. For example, the problem of training a sparse neural network can be naturally formulated as the following empirical risk minimization problem defined according to equation (5):

[0127]

[0128] where K is the attention capacity to be pruned. In this framework, sparse attention can be controlled by a single constraint, which avoids adjusting the pruning rate for each layer.

[0129] Each component of the learned variable or mask M can be regarded as a binary random variable, and the risk minimization problem defined according to equation (5) can be reparameterized with respect to the distribution of this random variable, and the risk minimization problem defined according to equation (5) can be relaxed to an expected loss minimization problem over the weight and probability space, which is continuous. This relaxation can be a very strict relaxation.

[0130] M ij can be considered as probability s ij will be 1 and 1-s ij A Bernoulli random variable, for example, M ij ~Bern(s ij ), where s ij ∈[0,1]. Assume that the variable M ij are independent, the distribution function of M may be because So the model size can be given by the probability s ij to control the sum of The discrete constraint ||M||1≤K can be transformed into Each of these ij ∈[0,1], which is continuous and convex. In this way, the risk minimization problem defined according to equation (5) can be relaxed to the expected loss minimization problem defined according to the following equation (6):

[0131]

[0132]

[0133] The expected loss minimization problem defined according to equation (6) can be solved using projected gradient descent. The difficulty lies in calculating the gradient of the expected loss with respect to the probability. Non-limiting embodiments or aspects of the present disclosure can use the Gumbel Softmax trick as described in the paper entitled "Categorical Reparameterization with Gumbel-Softmax" by Eric Jang, Shixiang Gu, and Ben Poole (2017) in ICLR to calculate the gradient, the disclosure of which is hereby incorporated by reference in its entirety, and the Gumbel Softmax trick can be used to calculate the gradient with respect to the weight and probability according to the following equation (7):

[0134]

[0135] where Ⅱ(A)∈{0,1} n is the indicator function; g0 and g1 are two random variables where each element is independent and identically distributed (iid) sampled from a Gumbel(0,1) distribution; Sampling examples; is an element-wise sigmoid function (e.g. for any etc.); and τ is a temperature annealing parameter that decreases linearly during training, where the precise choice of the decrease function helps the probabilities converge to a deterministic state.

[0136] like Figure 3 As shown in , at step 306, the process 300 includes processing the encoded representation through a decoder neural network to generate an output sequence having a corresponding output at each output position in a plurality of output positions in an output order. For example, the transaction service provider system 108 may process the encoded representation through a decoder neural network to generate an output sequence having a corresponding output at each output position in a plurality of output positions in an output order.

[0137] The decoder 408 (e.g., a decoder neural network, etc.) can be composed of a stack of N (e.g., N=6, etc.) identical decoder subnetworks or layers. Each layer can have three sublayers. The first sublayer can include a GAN 450, and the last sublayer can include a simple position-by-position fully connected feedforward network. In addition to these two sublayers, the decoder 408 can also insert a third sublayer between the two sublayers, which applies GAN 450 to the output of the encoder stack. Similar to the encoder 406, the decoder 408 can use residual connections around each sublayer in the sublayer, followed by layer normalization. The self-attention sublayer in the decoder stack can be modified to prevent positions from paying attention to subsequent positions. This masking combined with the fact that the output embedding is offset by one position ensures that predictions for positions in the input sequence can only rely on known outputs at positions less than that position.

[0138] For example, decoder 408 (e.g., a decoder neural network, etc.) may include a sequence of one or more decoder sub-networks, each decoder sub-network configured to receive a shifted version of an output sequence having a corresponding output at each of a plurality of output positions in the output order (e.g., shifted right by one position, etc.), and generate a corresponding decoder sub-network output for each of the plurality of output positions. Decoder 408 may include a plurality of GANs 450. The first GAN 450 may be configured to receive a shifted version of an output sequence (e.g., shifted right by one position, etc.), the output sequence having a corresponding output at each output position of a plurality of output positions in an output order, and for each specific output position in the output order: applying a gated attention mechanism to the shifted outputs at the plurality of output positions to generate a corresponding sub-layer output for the specific output position, wherein applying the gated attention mechanism comprises: determining a shared representation based on the shifted outputs at the plurality of output positions, determining a query for the specific output position based on the shared representation, determining a key for the plurality of output positions based on the shared representation, determining a value for the plurality of output positions based on the shared representation, and generating a corresponding output for the specific output position using the determined query, key, and value.

[0139] The second GAN 450 may be configured to receive the corresponding sub-layer output for the specific output position from the first GAN 450 and receive the corresponding encoded representation of each input in the input sequence. For example, for each specific output position in the output order, the second GAN 450 may: apply a gated attention mechanism to the corresponding sub-layer output for the specific output position from the first GAN 450 and / or the corresponding encoded representation of each input in the input sequence, wherein applying the gated attention mechanism includes: determining a shared representation based on the corresponding sub-layer output for the specific output position from the first GAN 450 and / or the corresponding encoded representation of each input in the input sequence, determining a query for the specific output position based on the shared representation, determining keys for multiple output positions based on the shared representation, determining values ​​for multiple output positions based on the shared representation, and generating a corresponding output for the specific output position using the determined query, key, and value.

[0140] like Figure 3 As shown in FIG. 3 , at step 308, process 300 includes providing an output sequence having a corresponding output at each output position in a plurality of output positions in an output order. For example, transaction service provider system 108 may provide (e.g., output, display, etc.) an output sequence having a corresponding output at each output position in a plurality of output positions in an output order. As an example, transaction service provider system 108 may provide (e.g., output, display, etc.) an output sequence having a corresponding output at each output position in a plurality of output positions in an output order. u And the time step |S u +1|Provide the next item for each user u In such an example, the next item may be provided to the user as a recommendation to the user (eg, as a recommended product or service to purchase, etc.).

[0141] Example Experiment

[0142] Experimental setup and empirical results are provided for experiments designed to answer the following research question: Do denoising transformers work better than transformers?

[0143] Simplified transformers according to non-limiting embodiments or aspects are evaluated on four datasets from three real-world applications. The datasets differ significantly in domain, platform, and sparsity: -Amazon: A series of datasets including a large corpus of product reviews crawled from Amazon.com. The top product categories on Amazon are treated as independent datasets. The experiment treats two categories: "Beauty" and "Games". This dataset is known for its high sparsity and variability. -Steam: We introduce a new dataset crawled from Steam (a large online video game distribution platform). The dataset contains 2,567,538 users, 15,474 games, and 7,793,069 English reviews spanning October 2010 to January 2018. The dataset also includes rich information such as the number of hours played by users, pricing information, media scores, categories, developers (etc.). -MovieLens: A benchmark dataset widely used to evaluate collaborative filtering algorithms. The experiment uses a version (MovieLens-1M) that includes 1 million user ratings.

[0144] The experiments follow the same preprocessing procedure as described in Angela Fan, Edouard Grave, and Armand Joulin (2020), “Reducing Transformer Depth on Demand with Structured Dropout”, ICLR, and Daniel Jakubovitz and Raja Giryes (2018), “Improving dnn robustness to adversarial attacks using jacobian regularization”, ECCV, 514–529, the disclosures of which are hereby incorporated by reference in their entirety. For each dataset, the experiments treat the presence of comments or ratings as implicit feedback (e.g., user-item interactions, etc.) and use timestamps to determine the sequential order of actions. The experiments discard users and items with fewer than 5 related actions. For partitioning, the experiments divide the historical sequence Su for each user u into three parts: (1) the most recent action |Su| for testing, (2) the second most recent action |Su|1 for validation, and (3) all remaining actions for training. It should be noted that during testing, the input sequence contains both training and validation behaviors. The data statistics are in Figure 5 The values ​​are listed in table 500 shown in FIG.

[0145] The baseline for the experiments can be a transformer as described in the paper titled “Time interval aware self-attention for sequential recommendation” by Jiacheng Li, Yujie Wang, and Julian McAuley (2020) in WSDM, pp. 322–330, the disclosure of which is hereby incorporated by reference in its entirety and which is the backbone framework on which many state-of-the-art algorithms are based.

[0146] The root mean square error (RSME) can be used to evaluate the simplified transformer according to non-limiting embodiments or aspects. To avoid heavy computations for all user-item pairs, the experiments follow the strategy in the paper titled “Time interval aware self-attention for sequential recommendation” by Jiacheng Li, Yujie Wang, and Julian McAuley (2020) in WSDM 322-330. For each user u, the experiment randomly extracts 100 negative items and rates these items using the ground truth items. Based on the ratings of these 101 items, RSME is evaluated. Figure 6 As shown in Table 600 in , a simplified converter according to a non-limiting embodiment or aspect outperforms existing converters.

[0147] Thus, non-limiting embodiments or aspects of the present disclosure may provide methods, systems, and / or computer program products that address the weaknesses of transformers when processing noisy sequences by providing a simple layer called a gated attention layer, which enables the use of a weaker single-headed attention with minimal quality loss, and using a probabilistic sparsification framework to further prune the noise and achieve high-quality sparse attention.

[0148] Although various embodiments or aspects have been described in detail for purposes of illustration and description, it will be understood that such detail is intended solely for that purpose and that the embodiments or aspects are not limited to the disclosed embodiments or aspects, but, on the contrary, are intended to cover modifications and equivalent arrangements within the spirit and scope of the appended claims. For example, it will be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect may be combined with one or more features of any other embodiment or aspect. Indeed, any of these features may be combined in a manner not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may be directly dependent on only one claim, the disclosure of possible implementations includes each dependent claim in combination with each other claim in the claim set.

Claims

1. A method comprising: receiving an input sequence having a corresponding input at each input position of a plurality of input positions in an input order; processing the input sequence by an encoder neural network to generate a respective encoded representation of each of the inputs in the input sequence, the encoder neural network comprising a sequence of one or more encoder sub-networks, each encoder sub-network configured to receive a respective encoder sub-network input for each of the plurality of input positions and to generate a respective encoder sub-network output for each of the plurality of input positions, and each encoder sub-network comprising: an encoder gated attention layer configured to receive the sub-network input for each input position in the plurality of input positions, and for each particular input position in the input order: applying a gated attention mechanism to the encoder sub-network inputs at the plurality of input positions to generate a corresponding output for the particular input position, wherein applying the gated attention mechanism comprises: determining a shared representation based on the encoder sub-network inputs at the plurality of input positions, determining a query for the particular input position based on the shared representation, determining keys for the plurality of input positions based on the shared representation, determining values ​​for the plurality of input positions based on the shared representation, and generating the corresponding output for the particular input position using the determined queries, keys, and values; and processing the encoded representation through a decoder neural network to generate an output sequence having a corresponding output at each of a plurality of output positions in an output order; and The output sequence is provided having the corresponding output at each output position of the plurality of output positions in the output order.

2. The method of claim 1, wherein the shared representation is determined by applying trainable variables to the encoder sub-network inputs at the plurality of input positions.

3. A method as claimed in claim 1, wherein the query for the specific input location is determined by applying a query transformation to the shared representation, wherein the key for the multiple input locations is determined by applying a key transformation to the shared representation, and wherein the values ​​for the multiple input locations are determined by applying a value transformation to the shared representation.

4. The method of claim 1, wherein the gated attention layer generates the corresponding output for the specific input position by: (i) determining the corresponding compatibility output for the specific input position by applying a compatibility function between the query for the specific input position and the keys generated for the multiple input positions, and (ii) applying the learned difference to the compatibility output for the specific input position to generate a corresponding attention value for the specific input position.

5. The method of claim 4, wherein the gated attention layer generates the corresponding output for the specific input position by applying a rectified linear unit (ReLU) activation function to the corresponding attention value for the specific input position and the values ​​for the multiple input positions.

6. A method as claimed in claim 4, wherein each encoder sub-network is trained according to an objective function that depends on the learned difference and the desired attention capacity.

7. A system comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to implement a sequence transduction neural network for transducing an input sequence having a corresponding network input at each input position of a plurality of input positions in an input order into an output sequence having a corresponding network output at each output position of a plurality of output positions in an output order, the sequence transduction neural network comprising: an encoder neural network configured to receive the input sequence and generate a respective encoded representation of each of the inputs in the input sequence, the encoder neural network comprising a sequence of one or more encoder sub-networks, each encoder sub-network configured to receive a respective encoder sub-network input for each of the plurality of input positions and generate a respective encoder sub-network output for each of the plurality of input positions, and each encoder sub-network comprising: an encoder gated attention layer configured to receive the sub-network input for each input position in the plurality of input positions, and for each particular input position in the input order: applying a gated attention mechanism to the encoder sub-network inputs at the plurality of input positions to generate a corresponding output for the particular input position, wherein applying the gated attention mechanism comprises: determining a shared representation based on the encoder sub-network inputs at the plurality of input positions, determining a query for the particular input position based on the shared representation, determining keys for the plurality of input positions based on the shared representation, determining values ​​for the plurality of input positions based on the shared representation, and generating the corresponding output for the particular input position using the determined queries, keys, and values; and A decoder neural network configured to receive the encoded representation and generate the output sequence.

8. The system of claim 7, wherein the shared representation is determined by applying trainable variables to the encoder sub-network inputs at the plurality of input positions.

9. A system as described in claim 7, wherein the query for the specific input position is determined by applying a query transformation to the shared representation, wherein the key for the multiple input positions is determined by applying a key transformation to the shared representation, and wherein the values ​​for the multiple input positions are determined by applying a value transformation to the shared representation.

10. A system as described in claim 7, wherein the gated attention layer generates the corresponding output for the specific input position by: (i) determining the corresponding compatibility output for the specific input position by applying a compatibility function between the query for the specific input position and the keys generated for the multiple input positions, and (ii) applying the learned difference to the compatibility output for the specific input position to generate a corresponding attention value for the specific input position.

11. The system of claim 10, wherein the gated attention layer generates the corresponding output for the specific input position by applying a rectified linear unit (ReLU) activation function to the corresponding attention value for the specific input position and the values ​​for the multiple input positions.

12. The system of claim 10, wherein each encoder sub-network is trained according to an objective function that depends on the learned differences and the desired attention capabilities.

13. A computer program product comprising a non-transitory computer readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to: receiving an input sequence having a corresponding input at each input position of a plurality of input positions in an input order; processing the input sequence by an encoder neural network to generate a respective encoded representation of each of the inputs in the input sequence, the encoder neural network comprising a sequence of one or more encoder sub-networks, each encoder sub-network configured to receive a respective encoder sub-network input for each of the plurality of input positions and to generate a respective encoder sub-network output for each of the plurality of input positions, and each encoder sub-network comprising: an encoder gated attention layer configured to receive the sub-network input for each input position in the plurality of input positions, and for each particular input position in the input order: applying a gated attention mechanism to the encoder sub-network inputs at the plurality of input positions to generate a corresponding output for the particular input position, wherein applying the gated attention mechanism comprises: determining a shared representation based on the encoder sub-network inputs at the plurality of input positions, determining a query for the particular input position based on the shared representation, determining keys for the plurality of input positions based on the shared representation, determining values ​​for the plurality of input positions based on the shared representation, and generating the corresponding output for the particular input position using the determined queries, keys, and values; and processing the encoded representation through a decoder neural network to generate an output sequence having a corresponding output at each of a plurality of output positions in an output order; and The output sequence is provided having the corresponding output at each output position of the plurality of output positions in the output order.

14. The computer program product of claim 13, wherein the shared representation is determined by applying trainable variables to the encoder sub-network inputs at the plurality of input positions.

15. A computer program product as described in claim 13, wherein the query for the specific input position is determined by applying a query transformation to the shared representation, wherein the keys for the multiple input positions are determined by applying a key transformation to the shared representation, and wherein the values ​​for the multiple input positions are determined by applying a value transformation to the shared representation.

16. A computer program product as described in claim 13, wherein the gated attention layer generates the corresponding output for the specific input position by: (i) determining the corresponding compatibility output for the specific input position by applying a compatibility function between the query for the specific input position and the keys generated for the multiple input positions, and (ii) applying the learned difference to the compatibility output for the specific input position to generate a corresponding attention value for the specific input position.

17. A computer program product as described in claim 16, wherein the gated attention layer generates the corresponding output for the specific input position by applying a rectified linear unit (ReLU) activation function to the corresponding attention value for the specific input position and the values ​​for the multiple input positions.

18. The computer program product of claim 16, wherein each encoder sub-network is trained according to an objective function that depends on the learned differences and the desired attention capabilities.

Images