Systems and methods for next basket recommendations using dynamic attribute modeling

Abstract

The embodiments described herein provide a framework of attention networks that model dynamic attributes with item and feature interactions. Specifically, the framework of attention networks first encodes the basket item sequence and dynamic attribute sequence with time-aware padding and time / month encoding in order to capture seasonal patterns (e.g., in app recommendations, outdoor activity apps are more suitable for summer time, while indoor activity apps are more suitable for winter). Then, the framework of attention networks applies a time-level attention module to the basket item sequence and dynamic user attribute sequence in order to capture per-basket-item and per-attribute temporal order patterns. Thereafter, a within-basket attention module is used on the items in each basket to capture the relevant information between items.

Description

[0001] Cross-referencing

[0002] The published text claims priority to U.S. Patent Application No. 17 / 112765, filed December 4, 2020, which in turn claims priority to Indian Provisional Application No. 202041035500, filed August 18, 2020, the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] The publicly available text generally deals with machine learning models and neural networks, and more specifically with next basket recommendation, which involves modeling dynamic attributes with high-order features and item interactions. Background Technology

[0004] Sequential recommendation is widely used in various applications, such as product recommendations, video recommendations, and music recommendations on e-commerce websites. In a sequential recommendation setup, a user has a series of historical interactions with different items (e.g., clicks, purchases). The recommendation mechanism predicts subsequent items based on the user's historical behavior. For example, by modeling sequential patterns from historical interactions, a sequential recommendation mechanism recommends items that the user is likely to interact with in the next time step. Existing sequential recommendation systems only consider the user's historical interaction items, ignoring the attributes of the items or the user themselves. Therefore, recommendations made by existing systems that ignore the attributes of the items or the user can sometimes be quite inaccurate or even irrelevant, leading to unsatisfactory user experiences. Attached Figure Description

[0005] Figure 1 A block diagram of an example recommender system that models dynamic attributes using item and feature interactions, based on an implementation described herein, is shown.

[0006] Figure 2A This illustrates an implementation scheme. Figure 1 A block diagram of an exemplary architecture for the recommendation module in [the context of the project], and Figure 2B It provides Figure 2A The diagram shows a block diagram detailing the structure of the module.

[0007] Figure 3 A simplified block diagram is provided to illustrate example operation of a multi-head self-attention block (MHSAB) according to the implementation described herein.

[0008] Figure 4 A simplified block diagram illustrating an example architecture of MHSAB according to the implementation described herein is provided.

[0009] Figure 5It is a simplified diagram of a computing device for implementing a recommendation system based on some implementation schemes, which models dynamic attributes with item and feature interactions that run through publicly available text descriptions.

[0010] Figure 6 This is a simplified logic flowchart illustrating a method for recommending the next basket using dynamic item attributes, based on some implementation schemes described herein.

[0011] Figure 7 A table summary statistics of an example training dataset is provided for the recommendation module 130 according to one implementation described herein.

[0012] Figure 8 A table is provided showing the overall results of a comparison with seven baseline methods based on one embodiment described herein.

[0013] Figure 9 A table is provided showing ablation studies of a recommended module based on one implementation described herein.

[0014] Figure 10 A visualization of the average timescale and in-attribute attention weights on an example training dataset based on an implementation described herein is provided.

[0015] In the accompanying drawings and appendices, elements with the same reference numerals have the same or similar functions. Detailed Implementation

[0016] Existing sequential recommendation systems primarily consider only the user's historical interactions with items, neglecting the attributes of items or users. In particular, since user interests and / or item attributes may change over time, recommendations that ignore the dynamic attributes of items or users may become significantly inaccurate over time.

[0017] Some existing systems can improve recommendation performance by incorporating static item attributes, such as product category and brand. However, for many applications, there are also dynamic user and item attributes that change over time. Such attributes provide important descriptions of shifts in user interests or changes in item attributes. For example, in app recommendation settings, there are dynamic attributes such as the amount of time a user spends on different apps, which can change daily and provide information about user preferences and needs across different apps. As another example, in exercise recommendation settings, a user's altitude, speed, and heart rate are relevant information that can change over time. Furthermore, seasonal patterns are often present in sequential recommendations. For example, a user who bought gift cards in the previous New Year is more likely to buy gift cards in the following New Year. As yet another example, swimwear is popular in the summer.

[0018] In some implementations, sequential recommendations can be made in the form of next-basket recommendations, where a user interacts with more than one item at each time step. For example, when ordering a meal online, people may purchase multiple products in the same shopping cart or order different dishes. Some existing systems can mine sequence patterns from a series of basket items. However, relationships between items within the same basket can also provide helpful information. Therefore, modeling sequential patterns (e.g., buying a phone and then a phone case) and relationships between items within the same basket (e.g., bread and butter are often grouped together) is important for understanding these relationships. For example, some items (e.g., bread and butter) are often grouped together. To capture relationships between items within the same basket, some existing systems use a correlation matrix based on the frequency of co-occurrence of items in the basket and then incorporate it into the modeling; however, this treats the correlations between items as static information rather than dynamic learning.

[0019] Beyond item interactions, higher-order interactions between different features are another important aspect of improving recommendation performance. For example, the movie "Frozen" could be recommended to a 10-year-old girl. The third-order features (gender: female, age: 10, genre: animation) provide useful information about this recommendation. Automatically learning these higher-order feature interactions, rather than relying on domain experts to manually design them, remains challenging.

[0020] Therefore, considering the need to provide sequential recommendations that capture users' temporal transaction patterns and in-basket relevance from historical user interaction items, as well as implicit dynamic preferences and higher-order feature interactions from dynamic user / item attributes, the implementation described in this paper provides an attentional network architecture for modeling dynamic attributes with item and feature interactions. Specifically, the attentional network framework first encodes the basket item sequence and dynamic attribute sequence using time-aware padding and time / month encoding to capture seasonal patterns (e.g., in app recommendation, outdoor activity apps are better suited for summer, while indoor activity apps are better suited for winter). Then, the attentional network framework applies temporal attention modules to the basket item sequence and dynamic user attribute sequence to capture basket items to basket items and attributes to attribute temporal order patterns. Subsequently, an in-basket attention module is used for items in each basket to capture relevant information between items. Self-attention is used to adaptively learn interactions between items during model training. Higher-order feature interactions can also be generated on attributes at each time step using self-attention modules.

[0021] As used herein, the term “network” can include any hardware- or software-based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented thereon or on it.

[0022] As used herein, the term "module" can include a hardware- or software-based framework that performs one or more functions. In some implementations, a module may be implemented on one or more neural networks.

[0023] Figure 1 A block diagram of an example recommender system demonstrating modeling dynamic attributes using item and feature interactions, according to the implementation described herein, is shown. Figure 100 illustrates a recommender system 130 that receives dynamic attributes 113a-n associated with user 110, dynamic attributes 114a-n associated with basket items 112, time-aware factor 116, etc.

[0024] For example, user dynamic attributes 113a-n may include attributes indicating a user's time-varying or dynamic attributes, such as age, location, application usage on the user's device, seasonal interests, etc. Time-aware factors 116 may include time in years, trends (e.g., movies showing in theaters at a certain time, etc.), etc. Dynamic attributes 114a-n of the past basket may include attributes of basket items, attributes of dynamic combinations of basket items (e.g., item A may frequently be combined with item B at a certain time of year, or may frequently be combined with item C at different times, etc.), and / or similar attributes.

[0025] Recommender system 130 can learn from historical interactions between users and items, dynamic attributes of users 113a-n, and dynamic attributes of items 114a-n, to recommend the next item 125. Specifically, recommender system 130 can obtain information about a set of users U with a total of |U| and a set of items V with a total of |V|. Each user or item has a set of attributes denoted by F, with a total number of attributes of |F|. For any user u∈U, such as user 110, the basket sequence... This indicates that within the time interval [0, T] u Project interactions during this period are sorted by time. Time period T u The time step can be different for each user, and T can be the maximum time step in each sequence, where t∈{1,2,3,…,T} is the first step size of the sequence of baskets. It is a set of items that user u interacts with at time step t. (Sequence) This represents the value of the dynamic user attribute for user u, sorted by time. These are all attribute values ​​of u at time step t. Specifically, This represents the category attribute value of user u at time t. This represents the numeric attribute value of user u at time t. Therefore, recommender system 130 can predict that user u (e.g., user 110) will be at time step t = T+1, given a maximum T-time history B. uand A u Basket project for information exchange.

[0026] Figure 2A This illustrates an implementation scheme. Figure 1 A block diagram 200a shows an exemplary architecture of the recommendation module 130 in the middle, while Figure 2B It provides Figure 2A Figure 200a shows a block diagram 200b detailing the structure of the modules shown. Figure 200a illustrates an attention network architecture for modeling dynamic attributes with item and feature interactions. The framework may include an input encoder (231a and 231b, or collectively 231), a temporal attention module (232a and 232b, or collectively 232), an in-basket module 233 and an in-attribute module 234, and a prediction module 235.

[0027] Input encoder 231 receives sparse representations of basket item sequence 203 and / or classification dynamic attribute sequence 204, encodes them into dense representations, and normalizes the numerical dynamic attributes. Specifically, at each time step, the optimal item sequence is received at input encoders 231a and 231b. and dynamic user attributes In some implementations, input encoders 231a and 231b can be two input encoders operating in parallel. In another implementation, input encoders 231a and 231b can be the same input encoder, configured to encode the basket item sequence 203 and the dynamic attribute sequence 204 simultaneously, alternately, or sequentially at each time step in any order.

[0028] The input encoder 231a can be used for each basket item sequence in the basket item sequence 203. We obtain the |V| vidothermal representation, thus representing the expression by... This represents information about the project group. Specifically, the vector index corresponds to the project ID, and if the project exists in the basket... If the item is in the basket, it is set to 1; otherwise, it is set to 0. Then, the input encoder 231a can use a concatenation-based lookup function to... Encoded as a latent representation 2^13, for example,

[0029]

[0030] The CONCAT-LOOKUP function first... Find the embedding vectors of the items in the array, and then concatenate all the vectors together to form a latent representation. |Vt| is The total number of items in the dataset. The embedding dimension for each item and each category attribute is D. This is the learnable embedding matrix used for lookup. After applying the "CONCAT-LOOKUP" function, the input encoder 231a is the basket. For each item i, a corresponding embedding vector is generated. Since the number of items in each basket varies, the maximum number of items in a basket is set to |V. max | and using vectors (q is the first embedding vector of Q) to fill the size smaller than |V max | Projects.

[0031] Similarly, for the dynamic user attributes in the dynamic attribute sequence 204, categorical and numerical attributes are modeled differently. Categorical attributes are represented as... |F cat | Dimensional heat vector representation. Using min-max normalization, the values ​​of numerical attributes are normalized to the range of -1 to 1, represented as: The input encoder 231b then represents the category attribute vector. Encoded as the following potential representation:

[0032]

[0033] in It is a learnable embedding matrix used for searching; and It is A t,cat The potential dense representation of .

[0034] In one implementation, when the lengths of the basket sequence and the user attribute sequence vary for different users, the maximum sequence length is set to T to obtain information from the latest T time steps. If the sequence length is greater than T, the sequence is truncated to the latest T time steps. If the sequence length is less than T, zero-padding can be applied to empty positions. Instead of padding the beginning of the sequence with zeros until the sequence length is T, the input encoder 231 pads the missed time steps with zeros. For example, for two sequences from months {first, second, third} and {first, third, fifth}, the maximum sequence length is 5. The initial padding ignores time-shift information. Instead, for the first sequence, zeros are padded to the end of the sequence (representing the fourth and fifth months), and for the second sequence, zeros are padded before the third and fifth months (representing the second and fourth months). Through this padding operation, the model can identify the differences between the two sequences. After padding, as well as The corresponding sequence represents the basket items, categorical attributes, and numerical attributes.

[0035] While padding techniques can produce longer sequences as input data compared to methods without padding, the actual duration of each time step (basket) depends on the application. For example, when the training data contains approximately four months of data, processing one basket per day is effective for training. Training per hour or per minute as a basket only produces sparse information for each basket.

[0036] In one implementation, the input encoder 231 embeds the position Add to the input sequence to utilize the order information of inputs 203 and 204, where p t This is the D-dimensional position embedding vector 213 at time t. Additionally, a monthly index embedding is added to the time-level attention modules 232a-b. A seasonal pattern was found, in which m t Let be the D-dimensional month index embedding vector at time t. The difference between positional embedding and month embedding is that the month index can be repeated every 12 time steps of the sequence. Therefore, the positional code and month code embeddings are related to the latent representation. The following connection is made:

[0037]

[0038] The basket sequence is represented as follows: Then the basket sequence L V The encoded representation 214 is sent from the input encoder 231a to the time-level attention module 232a.

[0039] Similarly, at input encoder 231b, the position index embedding 222 and the month index embedding 222 are added to...

[0040] and The potential representation 223 is used to obtain L respectively. Fcat and L Fnum The encoded representation 224 is then sent from the input encoder 231b to the time-level attention module 232b.

[0041] The time-level attention modules 232a-b are applied to the encoded basket sequence and user attribute sequence, respectively, to capture the user's sequential transaction patterns and implicit dynamic preferences. The time-level attention modules 232a-b can receive input sequence L. V L Fcat and L Fnum The dense tensor representation of . The tensor representation is reshaped into a 2D matrix, where . as well as Instead of taking the weighted sum of all items in the basket and all attribute values ​​each time.

[0042] The time-level attention modules 232a-b can each include multiple multi-head self-attention modules (MHSAB), which are applied to the 2D matrix L. V L Fcat and L Fnum To capture time-series patterns separately. For example, Figure 3 A simplified block diagram illustrating example operation of MHSAB according to the implementation described herein is provided. Specifically, the attention block can be described as mapping (query) 304 and a set of (key)-(value) pairs 306 and 308 to output 315. In the MHSA block, query 304, key 306, and value 308 are identical and are projected into multiple subspaces with different learned projection matrices. Here, query 304, key 306, and value 308 are first derived from L... (0) Input 302 projection (let L) (0) =L, any tensor representation from input encoder 231). Then the projection query 304 and projection key 306 are multiplied to obtain representation 310, which is then applied to projection value 308 to obtain self-attention output 315.

[0043] Figure 4 A simplified block diagram illustrating an example architecture of the MHSAB 400 according to the implementation described herein is provided. The MHASB 400 includes the MHSA 316 (which operates with...) Figure 3 Similar to Figure 300, the MHSA 316 consists of a normalization layer 331, a two-layer feedforward neural network (FFNN) 332, and another normalization layer 333. The MHSA 316 receives input 302 into multiple parallel self-attention (SA) blocks 320a-h, and the outputs from each SA block are concatenated to form output 318.

[0044] M (1) =MHSA(L (0) ,h)=[head1,head2,...,head h W concate ,

[0045]

[0046] Where SA is the self-attention mapping, and h is the number of subspaces. and The learning parameters (C = |F) cat +2|D), and C'=hC.

[0047] In one implementation, because the sequence is chronological, when making a prediction at time step t+1, the model can consider information from up to t time steps. To prevent the attention module from peeking into future step information, a causal mask can be added to the MHSA block to disable queries when j>i. i s and keys j All links between them.

[0048] In one implementation, to enhance the performance of representation learning for self-attention blocks, residual connections, dropout, layer normalization, and two fully connected layers with ReLU activation can be applied to the MHSA output 318.

[0049] M (1) =LayerNorm(Dropout(M (1) )+L (0) ),

[0050] L (1) =ReLU((M (1) W1+b1)W2+b2),

[0051] L (1) =LayerNorm(Dropout(L (1) )+M (1) ).

[0052] Therefore, the output 325 from the entire MHSAB can be obtained as follows:

[0053] L (1) =MHSAB(L (0) ,h).

[0054] Figure 4 Multiple (k>1) MHSABs shown can be stacked to capture more complex feature interactions:

[0055] L (k)＝ MHSAB(L (k-1) ,h k ), k>0

[0056] Where L (0) =L and h k It is the number of heads at the kth attention block.

[0057] Referring to Figure 2, the basket-in-module 233 and the attribute-in-attention module 234 are used to extract item interaction information and feature interaction information accordingly. Each of the basket-in-module 233 and the attribute-in-attention module 234 may include one or more stacks of MHSABs, and each stack of MHSABs includes one or more sequentially connected MHSABs. MHSABs can be similar to... Figure 4 Box 400 is shown.

[0058] Specifically, interactions between basket items have been shown to be useful for improving recommendations for the next basket. Furthermore, higher-order feature interactions have also shown improved prediction results. Therefore, MHSAB is applied to the basket and dynamic attributes at each time step t. This allows the model to mine interactions between basket items and generate higher-order feature interactions, which can help the model make better recommendations.

[0059] For example, for a categorical attribute sequence, the basket module 233 or the attribute module 234 receives the output representation matrix from the time-level attention module 232b. (L (k)Fcat (Abbreviation). Reshapes the output representation into a 3D tensor. Then, in the category user attribute representation L (k) MHSAB is applied to the 3D tensor L at each time step. (k) Therefore, the MHSAB stack updates the user attribute representation as follows:

[0060]

[0061] in It is the output of the time-level attention module at time t, and h k+1 It is the (k+1)th attention block.

[0062] Therefore, at time t, the classification attributes can be obtained as described above. Interactions between them. MHSAB can be stacked multiple times to capture higher-level categorical attribute interactions, such as... m>1. Similarly, basket items can be obtained at each time step t. Interactions and digital attributes Interactions between them.

[0063] Prediction module 235 receives and couples the outputs from basket-in and attribute-in-attention modules 233 and 234. To generate the final prediction 250. Specifically, in order to capture patterns of basket items and dynamic attributes, and to simultaneously incorporate item and attribute interaction information at time t, prediction module 235 includes a fully connected layer of feedforward neural network 236 applied to the concatenated input:

[0064]

[0065] in It is a high-level feature that contains time pattern information of basket items and dynamic attributes over the past time period t, as well as information on the interaction between items and attributes.

[0066] The prediction module 235 also receives a shared embedding table 230, such as the item embedding matrix Q, from the input encoder 231a, which provides a list of candidate items. This table is then used to predict the basket items at time step t+1. Specifically, the association prediction for item i is as follows:

[0067]

[0068] Where r i,t Let be the recommendation score of item i to be recommended, followed by the first t basket items and dynamic user attributes. Q is the item embedding matrix, which is shared for encoding basket items in the input encoder. Then the recommendation scores r of all i can be used. i,t To generate the next recommended item 250 for the next time step t+1. For example, using the softmax operation. 2

[0069] refer to Figure 2A During the training phase, the output r from the prediction module 235 i,t It is then sent to loss module 260 to calculate binary cross-entropy loss:

[0070]

[0071] Where σ is a sigmoid function The relevant score r of each item i in V at each time step t in each sequence. i,t User u's target basket item is B. u The transformed version, by The loss L can then be used, for example, to update modules 235, 234, 233, 232a-b, and 231a-b via backpropagation path 266 (shown in dashed lines).

[0072] Figure 5 This is a simplified diagram of a computing device 500 for implementing a recommendation system based on some implementation schemes. This recommendation system models dynamic attributes with interactions between items and features that run through publicly available text descriptions. Figure 5 As shown, computing device 500 includes a processor 510 coupled to memory 520. The operation of computing device 500 is controlled by processor 510. Furthermore, although computing device 500 is shown as having only one processor 510, it should be understood that processor 510 may represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), etc., within computing device 500. Computing device 500 may be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0073] Memory 520 may be used to store software executed by computing device 500 and / or one or more data structures used during operation of computing device 500. Memory 520 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tape, any other physical media with a perforated pattern, RAM, PROM, EPROM, FLASH-EPROM, any other memory chips or cartridges, and / or any other media suitable for a processor or computer to read from.

[0074] Processor 510 and / or memory 520 can be arranged in any suitable physical device. In some embodiments, processor 510 and / or memory 520 can be implemented on the same board, in the same package (e.g., system-in-package), on the same chip (e.g., system-on-a-chip), and / or the like. In some embodiments, processor 510 and / or memory 520 can include distributed, virtualized, and / or containerized computing resources. According to these embodiments, processor 510 and / or memory 510 can reside in one or more data centers and / or cloud computing facilities.

[0075] In some examples, memory 520 may include a non-transitory, tangible, machine-readable medium containing executable code that, when run by one or more processors (e.g., processor 510), enables the processors to perform methods described in further detail herein. For example, as shown, memory 520 includes instructions for recommendation module 130, which can be used to implement and / or simulate systems and models, and / or implement any methods further described herein. In some examples, recommendation module 130 may receive input 540, such as a basket sequence representing a user's item interaction records ordered chronologically. Data interface 515 may be any user interface that receives input performed by the user, or a communication interface that can receive or retrieve data sequences from a database. Recommendation module 130 may generate output 550, such as a first sequence of basket items and the next recommended item based on dynamic user attributes.

[0076] In some implementations, the recommendation module 130 includes an encoder 231, a time-level attention module 232, one or more in-basket and in-attribute self-attention modules 233, and a prediction module 234. Modules 231-234 in Figure 2A-2B This is further illustrated in the text.

[0077] The architecture of the recommended module 130, used to implement submodules 231-234, can be similar to... Figure 2A-2BFigures 200a-b are shown in the table. In some examples, it is recommended that module 130 and submodules 231-234 be implemented using hardware, software, and / or a combination of hardware and software.

[0078] Figure 6 This is a simplified logic flowchart illustrating a method for recommending the next basket using dynamic item attributes according to some embodiments described herein. One or more of processes 610-650 of method 600 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium, which, when run by one or more processors, causes one or more processors to execute one or more of processes 610-650. In some embodiments, method 600 may correspond to the method used by recommendation module 130.

[0079] In process 610, the input sequence of the past basket is received (e.g., Figure 2A-2B 203 in the middle) and sequences of dynamic user attributes (e.g., Figure 2A-2B (204 in the example). For instance, the input sequence of the past basket represents the user's item interaction records sorted by time.

[0080] In process 620, the input sequence of the past basket and the input sequence of the dynamic attributes are encoded into a dense representation, such as L. v L Fcat L Fnum For example, coding can be performed using embedded lookup and time-aware fill operations, as well as location or month index embeddings, as discussed in [the context of...]. Figure 2B The input encoders 231a-b are described in the document.

[0081] In process 630, temporal attention is applied to the dense representation to capture the user's sequential transaction patterns and implicit dynamic preferences, thereby generating a temporal attention matrix L. (k)V L (k)Fcat L (k)Fnum , k>0. Regarding Figure 2A-2B More details about time-level attention can be found in the time-level attention modules 232a-b.

[0082] In process 640, item interaction information and feature interaction information are extracted from the dense representation of interest. For example, stacks of MHSAB can be applied in parallel at each time step to the time-level attention matrices corresponding to the basket and dynamic attributes, respectively, so that the relevance between basket items and the generation of higher-order feature interactions are extracted to produce better recommendations. The resulting basket-based attention matrix and attribute attention matrix are L (k+m)V L (k+m)Fcat L (k+m)Fnum .about Figure 2A-2BThe self-attention modules 233 and 234 within baskets and attributes provide further details about self-attention.

[0083] In process 650, the next recommendation prediction is generated based on the basket item attention matrix and attribute attention matrix, indicating the extracted item interaction information and feature interaction information. About Figure 2A-2B The prediction module 235 in the document describes further details regarding the generation of the next recommendation prediction.

[0084] Example performance

[0085] Figure 7 A table summary statistics of an example training dataset is provided for the recommendation module 130 according to one implementation described herein. Figure 7 The dataset shown represents a collection of different domains with varying numbers of users, items, and attributes. The Enterprise Product Recommendation (EPR) dataset is a private dataset collected from a leading enterprise cloud platform. The task is to recommend products to enterprises. It contains 24 months of enterprise customer product purchase records (from January 2017 to December 2018) and 161 dynamic user attributes. Examples of dynamic attributes include enterprise product usage metrics, website behavioral metrics, sales information, and enterprise marketing campaigns.

[0086] The Santander Product Recommendation (SPR) 1 dataset is a public dataset about Santander Bank's product recommendations. It contains 18 months of user purchase information and user attributes. It also includes dynamic attributes. For example, "indrel_lmes" represents customer type such as primary customer, prospect, etc., and it can change from one type to another each month. "antiguedad" represents customer qualifications over several months.

[0087] The Ta-Feng dataset is a grocery shopping dataset containing transactions from November 2000 to February 2001. Each transaction is a basket of purchased items. Additionally, five attributes (AGE_GROUP, etc.) are provided. Although it does not contain dynamic attributes, the recommendation method described in this paper is still effective because it models the temporal patterns of user interactions with basket item sequences, higher-order item interactions, and higher-order attribute interactions.

[0088] Specifically, for each user u, the top K items from the ranked list of all items recommended in the next basket are selected. When making a prediction at time step t+1, time steps up to time step t are used. For the EPR dataset, the data from the first 20 months is used as the training set, and the data from the 21st month is used as validation data to tune the model. For testing, the best parameters are used to train on the data from the first 21 months. Data from three consecutive months (months 22-24) are processed as the test set. For the SPR dataset, the most frequent categorical attribute and the average of the numerical attribute are filled in based on the attribute of the missing value. The data from the first 17 months is used as the training data. The data is first split into training and validation sets, with 80% and 20% used for model tuning, and then all training data are used for training. The data from the 18th month is used for evaluation. For the Ta-Feng dataset, users who purchased less than 10 times and items purchased less than 10 times are removed. The data from the first 3 months is used as the training set, and the data from the last 0.5 months is used as the validation set to tune the model. The model is trained on the data from the first 3.5 months. Note that although the Ta-Feng dataset contains data for four months, the time pattern can include weekday / weekend patterns rather than seasonal patterns.

[0089] For the EPR and Ta-Feng datasets, evaluation metrics such as hit rate (HR@K), normalized discounted cumulative gain (NDCG@K), and median mean precision (MAP@K) are used. Accordingly, K is chosen as 23 and 7691 (all items) for EPR and Ta-Feng, respectively, based on MAP. Additionally, F1@5 and F1@10 are used as evaluation metrics on the Ta-Feng dataset. For the SPR dataset, the Kaggle challenge and MAP@7 are used for evaluation. Since the Kaggle submission only provides MAP@7 on the test set, this is the only metric reported for this dataset.

[0090] The following baseline methods are used for comparison: FMC, which applies Markov chains to the transition matrix of factorization terms; FPMC, which combines matrix factorization and first-order Markov chains of factorization to capture users' long-term preferences and item-by-item interactions; and SASRec, a self-attention-based model for sequential recommendation. This model takes a sequence of user actions as input and applies self-attention in the time domain to capture user action patterns to predict the next item. For a fair comparison, multi-head methods are used to enhance performance. Additional baseline models include BERT4Rec, which uses bidirectional self-attention to model user action sequences and employs a Cloze objective for sequential recommendation; DREAM, an RNN-based model where basket representations are aggregated via item embeddings through a merging operation—it generates the next basket item using the latest basket representation; Beacon, an RNN-based model that encodes the basket sequence while considering in-basket (item-in-basket) interactions; and FDSA+, an extension of the FDSA method where simple attention is applied to basket items at each time step and the loss function is updated in a manner similar to loss module 260.

[0091] For all baseline methods, if a cold start problem is encountered (e.g., users have no interaction history), a recommendation list is provided to these users based on popular items in the dataset. Since the original SASRec and BERT4Rec cannot handle basket item sequences by design, additional data preparation is performed on these models. Each basket item sequence, except for the last basket, is flattened, and then the last basket item is used as a label to generate training samples for each user.

[0092] AdamOptimizer was used to update the network with torque estimates β1 = 0.9 and β2 = 0.999. Self-attention layers were tuned according to {1, 2, 4}, and the number of heads on each attention block was tuned according to {1, 2, 4, 6}. The maximum sequence length was set to 12, 16, and 30 respectively in the EPR, SPR, and Ta-Feng datasets. All models were trained from scratch on a single NVIDIA Tesla K80 with a batch size of 256.

[0093] Figure 8A table showing the overall results of a comparison with seven baseline methods based on one implementation described herein is provided. The recommendation module 130 described herein significantly outperforms all baselines across all metrics. The relative percentage improvement ranges from 3.92% to 29.63%, demonstrating the effectiveness of recommendation module 130. Among the baseline methods, the method developed for next-basket recommendations (DREAM, Beacon) outperforms the method developed for general sequential recommendations (FMC, FPMC, SASRec, Bert4Rec). FDSA+ performs best on the EPR and SPR datasets, while Beacon performs best on the Ta-Feng dataset. The main reason is that FDSA+ utilizes attribute information, where EPR and SPR have more attributes compared to Ta-Feng. For the SPR dataset, the top model in the Kaggle challenge achieves 0.0314 in MAP@7. However, it employs heavy manual feature engineering to generate over 100 features (according to the Kaggle forum), which are time-consuming and overly specific to the dataset. It also integrates multiple classifiers to improve performance. All of these methods can be applied to the recommended module 130.

[0094] To understand the impact of the different components 231a-b, 232a-b, 233, 234, and 235 in recommendation module 130, a detailed ablation study was conducted using the EPR dataset. To understand the impact of different components, [further details were needed]. Figure 9 We conducted a detailed ablation study using the EPR dataset.

[0095] Figure 10 Provides a visualization of the average time level and intra-attribute attention weights on the SPR dataset. To understand how attention captures meaningful patterns, in Figure 10This paper explores the temporal attention weights and intra-attribute attention weights of sampled sequences from the SPR dataset. Four representative heatmaps with different layers and heads for a given input sequence are shown, where (a) and (b) are attention weights from two different layers (layers 1 and 4) of temporal basket attention, and (c) and (d) are from two different heads of the first intra-attribute layer. In (a) and (b), the x-axis is the position index of the basket sequence, and the y-axis is the time step. For example, in row 16, the temporal attention module considers baskets with position indices up to month 16. From (a) and (b), it is shown that attention varies across different layers. While the weights in layer 4 focus more on recent items, the weights in layer 1 pay more even attention to all previous history. This demonstrates the benefit of stacking more layers to capture different temporal patterns. In (c) and (d), each grid in the figure represents an interaction between two input features. From (c) and (d), it can be observed that the attention weights vary across different heads, and the module captures meaningful feature interactions. For example, in (c), position (11, 1) (marked by a red square) corresponds to the interaction feature value <“Foreigner Index”: No, “Customer Country Residence”: ES>, which makes sense because the bank is located in Spain, and customers living in Spain are not actually foreigners. In (d), position (9, 5) corresponds to the interaction feature value <“Customer Relationship Type”: Active, “New Customer Index”: 1>. This is also meaningful because new customers and active customers are typically highly correlated. These observations illustrate the usefulness of multi-headed attention.

[0096] Examples of computing devices, such as computing device 200, may include non-transitory, tangible, machine-readable media containing executable code that, when run by one or more processors (e.g., processor 210), enables the processors to execute the process of method 400. Common forms of machine-readable media that may include the process of method 400 include floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tapes, any other physical media with a perforated pattern, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media from which a processor or computer is adapted to read.

[0097] In illustrating inventive aspects, the embodiments, implementations, or applications described in this specification and the accompanying drawings should not be considered limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this specification and the claims. In some cases, well-known circuits, structures, or techniques are not shown or described in detail so as not to obscure the embodiments disclosed. Identical reference numerals denote identical or similar elements in two or more drawings.

[0098] This specification sets forth specific details of some embodiments consistent with the disclosed text. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be readily understood by those skilled in the art that some embodiments can be practiced without some or all of these specific details. The specific embodiments disclosed herein are illustrative and not restrictive. Those skilled in the art will recognize that while not specifically described herein, some other elements remain within the scope and spirit of the disclosed text. Furthermore, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless otherwise specifically described, or if one or more features render the embodiment ineffective.

[0099] While exemplary embodiments have been shown and described, a wide range of modifications, alterations, and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be employed without correspondingly using other features. Many variations, substitutions, and modifications will be recognized by those skilled in the art. Therefore, the scope of the invention should be limited only by the appended claims, and the claims should be interpreted broadly in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A system for recommending the next basket using dynamic item attributes, the system comprising: A communication interface that receives a basket of items that have been interacted with by the user and dynamic attributes associated with the user; Memory, which stores multiple processor-executable instructions; and One or more hardware processors that read and execute multiple processor-executable instructions to: The input encoder encodes the sequence of basket items indicating the basket of items and the sequence of dynamic attributes indicating the dynamic attributes associated with the user into a dense representation vector; The time-level attention module applies self-attention to generate a time-level attention matrix based on the dense representation vector, the time-level attention matrix indicating the time-level order pattern of the dynamic attributes associated with the user and the basket of items; Attention is applied by the basket-in-attention module to generate a basket-item attention matrix that indicates the feature interactions between the basket items based on the time-level attention matrix; Attention is applied by the in-attribute attention module to generate an attribute attention matrix based on the time-level attention matrix, which indicates the interaction between the dynamic attributes associated with the user; and The prediction module generates a prediction set of basket items for the next time step based on the recommendation scores calculated using the basket item attention matrix and the attribute attention matrix, wherein the prediction module includes a feedforward neural network, and the one or more hardware processors also read and execute a plurality of processor-executable instructions to: The basket item attention matrix from the basket attention module and the attribute attention matrix from the attribute attention module are concatenated to generate a concatenated matrix; and The recommended score is generated for each item using the feedforward neural network and based on the concatenated matrix.

2. The system of claim 1, wherein the input encoder is configured to encode each multi-hot representation vector corresponding to each basket item in the basket item sequence or each attribute in the dynamic attribute sequence as a basket item latent representation or an attribute latent representation, and The attributes mentioned therein can potentially represent either categorical or numerical attributes.

3. The system of claim 2, wherein the input encoder is further configured to: Generate position embedding vectors and month embedding vectors; and By adding the location embedding vector and the month embedding vector to the basket item latent representation or the attribute latent representation, a basket representation or a category attribute representation and a numerical attribute representation are generated.

4. The system of claim 1, wherein the input encoder is configured to perform a time-aware fill operation on the basket item sequence or the dynamic attribute sequence by: When the length of the corresponding sequence is less than the preset maximum sequence length, zeros are filled into the empty positions in the basket item sequence or the dynamic attribute sequence.

5. The system of claim 1, wherein the time-level attention module comprises one or more stacked multi-head self-attention blocks.

6. The system of claim 5, wherein the first multi-head self-attention block from the time-level attention module is configured as follows: The basket representation, categorical attribute representation, or numerical attribute representation from the input encoder is projected into the query, key, and value; and Multiple projected heads are connected in series.

7. The system of claim 6, wherein the first multi-head self-attention block also feeds the concatenated plurality of heads to a layer normalized and a fully connected layer with ReLU activation.

8. The system of claim 1, wherein the basket-in-attention module or the attribute-in-attention module comprises a stack of multiple multi-head self-attention blocks.

9. The system according to claim 1, further comprising: The loss calculation module is configured to calculate binary cross-entropy loss based on an S-shaped function of the recommended score.

10. A method for recommending the next basket using dynamic item attributes, the method comprising: Receive a basket of items that have been interacted with by the user and dynamic attributes associated with the user; The input encoder encodes the sequence of basket items indicating the basket of items and the sequence of dynamic attributes indicating the dynamic attributes associated with the user into a dense representation vector; The time-level attention module applies self-attention to generate a time-level attention matrix based on the dense representation vector, the time-level attention matrix indicating the time-level order pattern of the dynamic attributes associated with the user and the basket of items; Based on the time-level attention matrix, a basket item attention matrix is ​​generated that indicates the feature interactions between the basket items; Based on the time-level attention matrix, an attribute attention matrix is ​​generated that indicates the interaction between the dynamic attributes associated with the user; and Based on the recommendation scores calculated from the basket item attention matrix and the attribute attention matrix, a prediction set of basket items is generated for the next time step, wherein the method further includes: The basket item attention matrix and the attribute attention matrix are concatenated to generate a concatenated matrix; and The recommendation score is generated for each item based on the concatenated matrix using a feedforward neural network.

Images