A sequence recommendation method fusing item attribute information

Abstract

The application discloses a sequence recommendation method fusing item attribute information, and comprises the following steps: 1) data preprocessing and item attribute embedding; 2) extracting an interest category; 3) recommending an item; and 4) a training process. The method reduces the influence of model noise, eliminates the mixed correlation between heterogeneous attribute information, and improves the model performance.

Description

Technical Field

[0001] This invention relates to the field of sequence recommendation in data mining, specifically a sequence recommendation method that integrates item attribute information. Background Technology

[0002] Recommender systems play a crucial role in helping users filter valuable information from massive amounts of data. A key to a successful recommender system lies in the accurate modeling of user interests. Early recommender systems extracted user interests based on their entire historical behavior. However, a user's interests at a specific moment are influenced by their interests at previous moments and are dynamically changing. Based on this fact, some recommender algorithms model user history as a sequence, known as sequential recommendation. The goal of sequential recommendation is to predict the next item a user might be interested in based on their recent historical behavior. Its model takes a sequence of item IDs as input and outputs an interest vector.

[0003] Some studies have built enhanced sequence models by fusing other attribute information of items on top of the item ID sequence, resulting in significant improvements in recommendation performance. Existing research primarily uses multilayer perceptrons (MPBs) between the embedding and attention layers to fuse attribute information. However, attributes exhibit heterogeneity due to their varying data structures and element counts. ID embeddings represent overall item information, while attribute embeddings represent local item information, and there is a mixed correlation between the two. This method of fusing attribute information leads to a rank bottleneck in the attention matrix, limiting its expressive power. Furthermore, the fused embedding vectors are indivisible within the attention layer, inevitably increasing the complexity of model development and training. Specifically, early works like p-RNN typically use simple connections to directly inject auxiliary information into the item representation, resulting in a significant expansion of the embedding dimension. FDSA applies self-attention to both the item and attribute sequences, fusing the two feature vectors at the prediction layer. ICAI-SR fuses attribute information into the item representation before the attention layer. Both FDSA and ICAI-SR use MIBs to fuse heterogeneous attribute information, resulting in limitations in representational power, inflexible gradients, and complex development and training. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a sequence recommendation method that integrates project attribute information. This method can reduce the impact of model noise, eliminate the mixed correlation between heterogeneous attribute information, and improve model performance.

[0005] The technical solution to achieve the objective of this invention is:

[0006] A sequence recommendation method that integrates project attribute information includes the following steps:

[0007] 1) Data preprocessing and project attribute embedding:

[0008] Given a user's historical interactions with the system, a sequence recommendation task requires predicting the item to be interacted with next. Let u represent a user, and v represent the user's historical interactions. u The ordered tuples represented by the parentheses in formula (1) are:

[0009]

[0010] Where V j u This represents the j-th interaction performed by user u, also known as an action. Each interaction item is simply represented by an item ID and an item attribute, as shown in formula (2):

[0011] V j u =(ID) k At k (2),

[0012] ID k ∈Ι, representing the ID of the k-th item in the item vocabulary, Ι={ID 1 ID 2 ,…,ID m} is the identifier set of all project IDs to be considered in the system, and m is the total number of projects in the problem domain. J = {Attr} represents the attribute set of the k-th item in the project vocabulary. 1 Attr 2 ,…,Attr r} is the vocabulary of all project attributes in the system, and r is the total number of project attributes in the problem domain;

[0013] Original training sequence The length is uncertain and needs to be converted into a fixed-length sequence S = (S1, S2, ..., S...). n ), where n is the maximum length that the model can handle. If the sequence length is greater than n, the nearest n items are taken. If the sequence length is less than n, zeros are used to pad the left side of the sequence until the sequence length is n. Therefore, n is also the length of all fixed-length sequences after the transformation.

[0014] After converting the variable-length sequence into a fixed-length sequence, the original sequence is converted into an embedded sequence. Embedding refers to the embedding vector, which is the vector representation of item attributes in the recommendation algorithm. The embedding layer models different types of item attributes separately, converting the sequence into category attribute embedding vectors and ID embedding vectors for the item category attribute embedding vectors and ID embedding vectors, respectively, for the interest extraction module and the cross-prediction module; (1) ID attribute embedding:

[0015] Create ID attribute embedding matrix MID ∈R m×d Where m is the total number of items and d is the dimension of the embedding vector, we look up M based on the ID of the item in sequence S. ID The corresponding embedding vector in the sequence transforms S into an embedding sequence E. ID ∈R n×d The ID attribute of zero-padded items is embedded as a constant zero vector;

[0016] (2) Category attribute embedding:

[0017] Create category attribute embedding matrix M CATE ∈Rr ×d Where r is the total number of category elements and d is the dimension of the embedding vector, since the category attribute of an item is a set of category elements, which is a one-to-many relationship, it cannot be directly searched like the ID attribute embedding. Therefore, the category attribute of the i-th item in S needs to be multi-hot encoded and represented as a vector. e i If the value at position j is 1, then S i For a category element with index j, the category attribute embedding of the sequence is represented by formula (3), and the category attribute embedding of the zero-filled item is also a constant zero vector:

[0018]

[0019] (3) Position embedding:

[0020] The original attention mechanism does not include positional information of the sequence and cannot identify the order of items in the sequence. This deficiency can be remedied by adding positional encoding, which uses a learnable embedding matrix P∈R. n×d The fusion operations of position embedding, ID attribute embedding, and position embedding and category attribute embedding are all the same: the corresponding positions of the vectors are added together, as shown in formula (4):

[0021]

[0022] The category embedding representation after adding position embedding is as follows ID attribute embedding is represented as The position embedding matrices used for ID attribute embedding and category attribute embedding are not shared;

[0023] 2) Extracting interest categories:

[0024] The main structure of the interest extraction module is a multi-head sparse self-attention block. The multi-head sparse self-attention method is an extension of the multi-head self-attention method. The multi-head self-attention layer (MSA) and the multilayer perceptron (MLP) are connected together to form a multi-head attention block.

[0025] The interest extraction module embeds data from the input category. Extracting user category interests O CATE ∈R d ,

[0026] enter Use formula (5) to obtain the query vector Q, key vector K, and value vector V:

[0027] f (·) (x)=XW (·) (5),

[0028] in It is a learnable projection matrix, where i represents the i-th self-attention head, n represents the length of the fixed-length sequence, d is the dimension of the embedding vector, and h is the total number of self-attention heads. The attention score matrix A is obtained through formula (6):

[0029]

[0030] Explicit selection involves performing a sparsification operation on A, as shown in equation (7), where A ij Let t be the attention scores for item i and item j in the sequence. i Let k be the k-th maximum attention score for item i, where k is a hyperparameter:

[0031]

[0032] Then to Normalize as shown in formula (8):

[0033]

[0034] For standardized scores, due to the masking function The scores less than top-k are assigned negative infinity, and their normalized scores, i.e., probabilities, are approximately 0.

[0035] The aggregation operation for the value of the i-th head is shown in formula (9):

[0036] C i =A i V i (9),

[0037] Multi-head self-attention concatenates the outputs of each head and performs a linear projection to obtain the output, as shown in formula (10), W o It is a learnable projection matrix:

[0038] MSA(X) = Concat(C1, ..., C h W o (10)),

[0039] In a multilayer perceptron, between multi-head self-attention layers, FC(·) is a fully connected layer, and W∈R d×d b∈R d It is the learnable projection matrix and bias, and σ(·) is the activation function, as shown in Equation (11):

[0040] MLP(X)=FC(σ(FC(X))), FC(X)=XW+b (11),

[0041] Both multi-head self-attention and multilayer perceptrons use Residual Connections to prevent the vanishing gradient problem during training, use Layer Normalization to accelerate training, and concatenate them as a multi-head attention block. (l) G is the input of the l-th block. (l) The output of this multi-head attention layer is as shown in formula (12):

[0042] Input category embedding After passing through one or more multi-head sparse self-attention blocks, the output is O. CATE , representing the user's category interests, is used to calculate the auxiliary loss function and the cross-prediction module to predict interest items;

[0043] 3) Recommended projects:

[0044] The main structure of the cross-prediction module is a multi-head cross-attention block, which is composed of a cross-attention block and a multilayer perceptron connected in series. While self-attention methods aggregate sequences using intra-sequence information, cross-attention methods can introduce additional information to aggregate sequences. The cross-prediction module takes category interest O as input. CATE and ID embedding Using the cross-attention method, output the project interest O. ID ∈R d ;

[0045] The query vector for the cross-prediction module comes from the output O of the interest extraction module. CATE The key vector and value vector come from the ID embedding sequence.

[0046] As in formula (13):

[0047]

[0048] Linear transformation f (·) (x) is shown in formula (5), and its projection matrix is ​​not shared with the projection matrix of the sparse self-attention module;

[0049] The attention matrix calculation process is shown in formula (6). The multi-head aggregation operation and multilayer perceptron are consistent with the multi-head attention block, as shown in formulas (9), (10), (11), and (12). ID The inner product with the project ID embedding vector is used to measure a user's interest in the project; the larger the inner product, the greater the probability that the user is interested in the project.

[0050] 4) Training process:

[0051] The technical solution employs a binary cross-entropy loss function for model optimization, defining N as the number of training sequences, t as the time step in the sequence, and e[t+1]∈I. u As a positive example, it represents the items the user interacted with at time t+1. As a negative example, I represents an item that a randomly selected user did not interact with at time t+1. u It is the set of items that user u has interacted with, I is the set of all items, O[t] represents the interest at time t output by the model, lg is the logarithmic function, σ is the sigmoid function, and · is the inner product operation;

[0052] The loss function is defined by formula (14), and the sum of the interest of the model output at all time points of all sequences in the system and the binary cross-entropy loss of positive and negative examples is:

[0053]

[0054] To enable the interest extraction module to learn meaningful interest representations, the model uses an auxiliary loss function to convert the O(n) output of the interest extraction module into a single loss function. CATE The category embedding sequence e as a positive example CATE and the randomly selected class embedding sequence as a negative example Similarly, as input to the auxiliary loss function, the target loss function will use the O output of the cross-prediction module. ID ID embedding sequence e as a positive example ID and the randomly selected class embedding sequence as a negative example As an output, as shown in formula (15):

[0055]

[0056] The loss function of the final model is the target item loss L. target With auxiliary loss L aux In combination, β is a hyperparameter used to balance the target loss and the auxiliary loss, as shown in equation (1,6):

[0057] L = l target +β*L aux (16).

[0058] Compared with existing methods, this technical solution can produce the following beneficial effects:

[0059] Sequences inevitably contain noise, such as random user interactions. They may also contain multiple centers of interest. Failure to address these two issues will inevitably limit model performance. This technical solution employs an explicit self-attention sparsity method based on top-k selection. This method focuses only on the k items with the highest attention scores, effectively reducing the impact of noisy items and concentrating attention, resulting in a significant performance improvement compared to similar models.

[0060] Previous methods using multilayer perceptrons to fuse heterogeneous attribute information suffer from limited representational capabilities, inflexible gradients, and complex development and training. This technical solution employs a cross-attention method to fuse attribute information. Cross-attention is typically used as the decoder, in conjunction with self-attention as the encoder. In this model, the interest extraction module acts as the encoder, and the cross-prediction module acts as the decoder. The cross-attention method is non-intrusive to item embedding vectors and avoids the mixing and correlation between heterogeneous attribute information.

[0061] This method reduces the impact of model noise, eliminates the mixed correlation between heterogeneous attribute information, and improves model performance. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the model structure for an embodiment;

[0063] Figure 2 A comparison of the Benley model with different numbers of self-attention layers and cross-attention layers;

[0064] Figure 3 A comparison of the Benley model and the SASRec model under different numbers of attention layers;

[0065] Figure 4 This is a graph showing the comparison results of HR@20 and NDCG@20 metrics for this example model across different embedding dimensions and datasets;

[0066] Figure 5 This is a comparison of the HR metrics for this example model on the toys and lastFM datasets with different sparse self-attention K values. Detailed Implementation

[0067] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0068] Example:

[0069] Reference Figure 1 A sequence recommendation method that integrates project attribute information includes the following steps:

[0070] 1) Data preprocessing and project attribute embedding:

[0071] Given a user's historical interactions with the system, a sequence recommendation task requires predicting the item to be interacted with next. Let u represent a user, and v represent the user's historical interactions. u The ordered tuples represented by the parentheses in formula (1) are:

[0072]

[0073] Where V j u This represents the j-th interaction performed by user u, also known as an action. Each interaction item is represented by an item ID and an item attribute, as shown in formula (2):

[0074] V j u =(ID) k At k (2),

[0075] ID k ∈Ι, representing the ID of the k-th item in the item vocabulary, Ι={ID 1 ID 2 ,…,ID m} is the identifier set of all project IDs to be considered in the system, and m is the total number of projects in the problem domain. J = {Attr} represents the attribute set of the k-th item in the project vocabulary. 1 Attr 2 ,…,Attr r} is the vocabulary of all project attributes in the system, and r is the total number of project attributes in the problem domain;

[0076] Original training sequence The length is uncertain and needs to be converted into a fixed-length sequence S = (S1, S2, ..., S...). n ), where n is the maximum length that the model can handle. If the sequence length is greater than n, the nearest n items are taken. If the sequence length is less than n, zeros are used to pad the left side of the sequence until the sequence length is n. Therefore, n is also the length of all fixed-length sequences after the transformation.

[0077] After converting the variable-length sequence into a fixed-length sequence, the original sequence is converted into an embedded sequence. An embedding refers to an embedding vector, which is a vector representation of item attributes in the recommendation algorithm. The embedding layer models different types of item attributes separately, converting the sequence into category attribute embedding vectors and ID embedding vectors for the item category attribute embedding vectors and ID embedding vectors, respectively, which are used in the interest extraction module and the cross-prediction module.

[0078] (1) Embedding ID attribute:

[0079] Create ID attribute embedding matrix M ID ∈R m×d Where m is the total number of items and d is the dimension of the embedding vector, we look up M based on the ID of the item in sequence S. ID The corresponding embedding vector in the sequence transforms S into an embedding sequence E. ID ∈R n×d The ID attribute of zero-padded items is embedded as a constant zero vector;

[0080] (2) Category attribute embedding:

[0081] Create category attribute embedding matrix M CATE ∈R r×d r is the total number of item attributes, and d is the dimension of the embedding vector. Since the category attributes of an item are a set of category elements, which has a one-to-many relationship, they cannot be directly searched like ID attribute embeddings. Therefore, the category attributes of the i-th item in S must first be multi-hot encoded and represented as a vector. e i If the value at position j is 1, then S i For a category element with index j, the category attribute embedding of the sequence is represented by formula (3), and the category attribute embedding of the zero-filled item is also a constant zero vector:

[0082]

[0083] (3) Position embedding:

[0084] The original attention mechanism does not include positional information of the sequence and cannot identify the order of items in the sequence. This deficiency can be remedied by adding positional encoding, which uses a learnable embedding matrix P∈R. n×d The fusion operations of position embedding, ID attribute embedding, and position embedding and category attribute embedding are all the same: the corresponding positions of the vectors are added together, as shown in formula (4):

[0085]

[0086] The category attribute embedding after adding position embedding is represented as follows ID attribute embedding is represented as The position embedding matrices used for ID attribute embedding and category attribute embedding are not shared;

[0087] 2) Extracting interest categories:

[0088] The main structure of the interest extraction module is a multi-head sparse self-attention block. The multi-head sparse self-attention method is an extension of the multi-head self-attention method. The multi-head self-attention layer and the multilayer perceptron are connected together to form a multi-head attention block.

[0089] The interest extraction module embeds data from the input category. Extracting user category interests O CATE ∈R d ,

[0090] enter Use formula (5) to obtain the query vector Q, key vector K, and value vector V:

[0091] f (·) (x)=xW (·) (5),

[0092] in It is a learnable projection matrix, where i represents the i-th self-attention head, n is the length of the fixed-length sequence, d is the dimension of the embedding vector, and h is the total number of self-attention heads. The attention score matrix A is obtained through formula (6):

[0093]

[0094] Explicit selection involves performing sparsification on A, as shown in equation (7), Ai j Let t be the attention scores for item i and item j in the sequence. i Let k be the k-th maximum attention score for item i, where k is a hyperparameter:

[0095]

[0096] Then to Normalize as shown in formula (8):

[0097]

[0098] For standardized scores, due to the masking function The scores less than top-k are assigned negative infinity, and their normalized scores, i.e., probabilities, are approximately 0.

[0099] The aggregation operation for the value of the i-th head is shown in formula (9):

[0100] C i =A i V i (9),

[0101] Multi-head self-attention concatenates the outputs of each head and performs a linear projection to obtain the output, as shown in formula (10), W oIt is a learnable projection matrix:

[0102] MSA(X) = Concat(C1,...,C h W o (10),

[0103] In a multilayer perceptron, between multi-head self-attention layers, FC(·) is a fully connected layer, and W∈R d×d b∈R d It is the learnable projection matrix and bias, and σ(·) is the activation function, as shown in Equation (11):

[0104] MLP(X)=FC(σ(FC(X))),FC(X)=XW+b (11),

[0105] Both multi-head self-attention and multilayer perceptrons use Residual Connections to prevent the vanishing gradient problem during training, and use Layer Normalization to accelerate training, concatenating them as a multi-head attention block. (l) G is the input of the l-th block. (l) The output of this multi-head attention layer is as shown in formula (12):

[0106] Input category embedding After passing through one or more multi-head sparse self-attention blocks, the output is O. CATE , representing the user's category interests, is used to calculate the auxiliary loss function and the cross-prediction module to predict interest items;

[0107] 3) Recommended projects:

[0108] The main structure of the cross-prediction module is a multi-head cross-attention block, which is composed of a cross-attention block and a multilayer perceptron connected in series. While self-attention methods aggregate sequences using intra-sequence information, cross-attention methods can introduce additional information to aggregate sequences. The cross-prediction module takes category interest O as input. CATE and ID embedding Using the cross-attention method, output the project interest O. ID ∈R d ;

[0109] The query vector for the cross-prediction module comes from the output O of the interest extraction module. CATE The key vector and value vector come from the ID embedding sequence.

[0110] As in formula (13):

[0111]

[0112] Linear transformation f (·) (x) is shown in formula (5), and its projection matrix is ​​not shared with the projection matrix of the sparse self-attention module;

[0113] The attention matrix calculation process is shown in formula (6). The multi-head aggregation operation and multilayer perceptron are consistent with the multi-head attention block, as shown in formulas (9), (10), (11), and (12). ID The inner product with the project ID embedding vector is used to measure a user's interest in the project; the larger the inner product, the greater the probability that the user is interested in the project.

[0114] 4) Training process:

[0115] The binary cross-entropy loss function is used for model optimization, where N is the number of training sequences, t is the time step in the sequence, and e[t+1]∈I. u As a positive example, it represents the items the user interacted with at time t+1. As a negative example, I represents an item that a randomly selected user did not interact with at time t+1. u It is the set of items that user u has interacted with, I is the set of all items, O[t] represents the interest at time t output by the model, lg is the logarithmic function, σ is the sigmoid function, and · is the inner product operation;

[0116] The loss function is defined by formula (14), and the sum of the interest of the model output at all time points of all sequences in the system and the binary cross-entropy loss of positive and negative examples is:

[0117]

[0118] To enable the interest extraction module to learn meaningful interest representations, the model uses an auxiliary loss function to convert the O(n) output of the interest extraction module into a single loss function. CATE The category embedding sequence e as a positive example CATE and the randomly selected class embedding sequence as a negative example Similarly, as input to the auxiliary loss function, the target loss function will use the O output of the cross-prediction module. ID ID embedding sequence e as a positive example ID and the randomly selected class embedding sequence as a negative example As an output, as shown in formula (15):

[0119]

[0120] The loss function of the final model is the target item loss L. target With auxiliary loss L aux In combination, β is a hyperparameter used to balance the target loss and the auxiliary loss, as shown in equation (16):

[0121] L = L target +β*L aux (16).

[0122] This example uses four datasets from different domains with varying levels of sparsity. Table 1 summarizes the statistics for these datasets after preprocessing.

[0123] Table 1. Dataset Statistics Table

[0124]

[0125] The Beauty, Sports, and Toys datasets were obtained from the Amazon review dataset. Three subcategories were selected: "Beauty," "Sports," and "Toys," and the product category and brand were used as category attributes. The LastFM dataset contains social networks, tags, and music artist listening information from a 2K user set of the Last.fm online music system. Category attributes were set as labels.

[0126] For all datasets, interaction records are grouped by user and sorted in ascending order by interaction timestamp, while unpopular items and inactive users with fewer than 5 interaction records are filtered out. The Benley model's comparative results with different numbers of self-attention layers and cross-attention layers are as follows: Figure 2 As shown.

[0127] Evaluation indicators

[0128] Performance is evaluated using top-k hit rate (HR@k) and top-k normalized cumulative return (NDCG@k), which are widely used in related work. Results for HR@{10, 20} and NDCG@{10, 20} are reported. A leave-one-out strategy is used for evaluation. Specifically, for each user interaction sequence, the last item is used as test data, the items before the last are used as validation data, and the remaining data are used for training. This example calculates all metrics based on item ranking and reports the average score for all test users.

[0129] The comparison results of HR@20 and NDCG@20 metrics for this example model on different embedding dimensions and different datasets are as follows: Figure 4 As shown. The comparison results of the HR index of this example model on the toys and lastFM datasets with different sparse self-attention K values ​​are as follows. Figure 5 As shown.

[0130] Baseline method

[0131] This example model is compared with the following baseline methods:

[0132] Caser: A CNN-based model that uses horizontal and vertical convolutional filters to learn multi-level patterns and user preferences.

[0133] GRU4Rec: A session-based recommendation system that uses RNNs to capture sequence patterns.

[0134] BERT4Rec: A bidirectional self-attention network that uses a cloze test task to simulate user behavior sequences. SASRec: A sequence recommendation model based on self-attention methods that uses a multi-head attention mechanism to recommend the next item.

[0135] SASRecF: An extension of SASRecF, it concatenates representations of items and attributes as input to the model. FDSA: This model uses trivial attention and multi-head attention mechanisms to extract user interests and item features for sequence recommendation.

[0136] The comparison results of the Benley model and the SASRec model under different numbers of attention layers are as follows: Figure 3 As shown.

[0137] The comparison results of this example model with several baseline methods are shown in Table 2:

[0138] Table 2 compares the HR and NDCG metrics of this example model with other models on different datasets:

[0139]

Claims

1. A sequence recommendation method of fusion item attribute information, characterized by, Comprising the following steps: 1) Data preprocessing and item attribute embedding: Given a user's history of interactions with a system, the sequential recommendation task requires predicting the next item of interaction, where u denotes a user, V u The sequential tuples denoted by the parentheses in equation (1) are: wherein denotes the jth interaction, also referred to as behavior, of a user u, each item of which is simply represented by an item ID and item attributes, as in equation (2): where ID k ∈ I, represents the ID of the kth item in the item vocabulary, I = {ID 1 , ID 2 ,..., ID m} is the identification set of all item IDs to be considered in the system, m is the total number of items in the problem domain, represents the attribute set of the kth item in the item vocabulary, J = {Attr 1 , Attr 2 ,..., Attr r} is the vocabulary of all item attributes in the system, r is the total number of item attributes in the problem domain; Original training sequence The length is uncertain and needs to be converted into a fixed-length sequence S = (S1, S2, ..., S...). n ), where n is the maximum length that the model can handle. If the sequence length is greater than n, the nearest n items are taken. If the sequence length is less than n, zeros are used to pad the left side of the sequence until the sequence length is n. Therefore, n is also the length of all fixed-length sequences after the transformation. After converting the indefinite length sequence into a fixed length sequence, the original sequence is converted into an embedded sequence, embedding refers to embedding vectors, which are vector representations of item attributes in recommendation algorithms, and embedding layers model different types of item attributes separately, converting the sequence into category attribute embedding vectors and ID embedding vectors of items, which are used for interest extraction module and cross prediction module respectively; (1) ID attribute embedding: Creating an ID attribute embedding matrix M ID ∈ R m×d , m is the total number of items, d is the dimension of the embedding vectors, and M ID is the embedding matrix, where each row is the embedding vector for an item in the sequence S ID ∈ R n×d , the ID attribute embedding for a zero padding item is a constant zero vector; (2) Category attribute embedding: Creating category attribute embedding matrix M CATE ∈R r×d , r is the total number of item attributes, d is the dimension of embedding vector, since the category attribute of the item is a set of category elements, it is a one-to-many relationship, and cannot be directly looked up like ID attribute embedding. The category attribute of the i-th item in S needs to be Muti-Hot coded first, represented as a vector e i The j-th position value of the sequence is 1, then S i has a category element with number j, and the category attribute embedding of the sequence is represented by formula (3). The category attribute embedding of the zero padding item is also a constant zero vector: (3) Position embedding: The original attention mechanism does not contain the position information of the sequence, and cannot identify the order of the items in the sequence. The position encoding can be added to make up for this deficiency. The position encoding uses a learnable embedding matrix P ∈ R n×d The fusion operation of the position embedding and the ID attribute embedding and the fusion operation of the position embedding and the category attribute embedding are both vector addition corresponding to positions, as shown in equation (4): The class embedding after adding the position embedding is represented as The ID attribute embedding is represented as The position embedding matrix used by the ID attribute embedding and the class embedding are not shared; 2) Extracting interest categories: The main structure of the interest extraction module is a multi-head sparse self-attention block, which is an extension of the multi-head self-attention method. The multi-head self-attention layer and the multi-layer perceptron are connected in series to form a multi-head attention block. The interest extraction module extracts the category interest O of the user from the input category embedding CATE ∈R d , input The query vector Q, the key vector K, and the value vector V are obtained using formula (5):​ f (·) (x) = XW (·) (5), wherein is a learnable projection matrix, i represents the ith self-attention head, n is the length of the fixed-length sequence, d is the dimension of the embedding vector, h is the total number of self-attention heads, and A is the attention score matrix obtained by formula (6): Explicit selection is to do sparsification on A as shown in equation (7), A ij is the attention score for item i and item j in the sequence, t i is the k-th largest attention score for item i, k is a hyperparameter: Then the normalized value is calculated as shown in equation (8): (8) To standardize the scores, since the masking function Assign scores less than top-k a value of negative infinity, whose normalized score, i.e., probability, is 0; The aggregation operation of the value of the i-th head is shown in formula (9): C i = A i V i (9), Multi-head self-attention concatenates the output of each head and linearly projects it to get the output, as in equation (10), W o is a learnable projection matrix: MSA(X) = Concat(C1,..., C h )W o (10), MLP between multi-head self-attention layers, FC(·) is a fully connected layer, W e R d×d , b e R d is a learnable projection matrix and bias, and σ(·) is an activation function, as shown in equation (11): MLP(X)=FC(σ(FC(X))),FC(X)=XW+ b (11), Residual Connections method is used on both multi-head self-attention and multi-layer perceptron to prevent the problem of gradient vanishing during training, and Layer Normalization is used to accelerate training, and they are connected in series as a multi-head attention block, H (l) is the input of the l-th block, G (l) is the output of the multi-head attention layer of this block, as formula (12): input category embedding after passing through one or more multi-headed sparse self-attention blocks, output O CATE , representing the category interests of the user, for computing the auxiliary loss function and the interest item prediction by the cross-prediction module; 3) Recommended items: The main structure of the cross prediction module is a multi-head cross attention block, which is formed by connecting a cross attention block and a multi-layer perceptron in series. The self-attention method aggregates sequences using information within the sequence, while the cross-attention method can introduce additional information to aggregate sequences. The cross prediction module inputs the category interest O CATE and ID embedding Using the cross-attention method, the output item interest O ID ∈R d ; The cross-prediction module queries the vector from the output O of the interest extraction module CATE While the key vector and the value vector come from the ID embedding sequence As formula (13): Linear transformation f (·) (x) See Equation (5), whose projection matrix does not share with the projection matrix of the sparse self-attention module; The attention matrix calculation process is as formula (6), the multi-head aggregation operation and the multi-layer perception are consistent with the multi-head attention block, see formula (9), (10), (11), (12), O ID The inner product with the project ID embedding vector is used to measure the interest of the user in the project, and the greater the inner product, the greater the probability that the user is interested in the project; 4) Training process: The model is optimized using binary cross-entropy loss function, defining N as the number of training sequences, t as the time in the sequence, e[t+1]∈I u As a positive example, it represents the item that the user interacts at t+1 time, As a negative example, it represents the item that the user has not interacted at t+1 time, I u is the set of items that the user u has interacted with, I is the set of all items, O[t] represents the interest at t time output by the model, lg is the logarithmic function, σ is the sigmoid function, and · is the inner product operation; The loss function is defined as formula (14), and the sum of the binary cross-entropy loss of the interest of the model output at all times of all sequences in the system and the positive and negative examples is: To make the interest extraction module learn meaningful interest representation, the model uses a helper loss function, which takes as input the O CATE category embedding sequence e CATE as positive examples and randomly sampled category embedding sequences as negative examples As input to the helper loss function, similarly, the target loss function takes as input the O ID ID embedding sequence e ID as positive examples and randomly sampled category embedding sequences as negative examples As output, as formula (15): The loss function of the final model is the target item loss L target combined with the auxiliary loss L aux β is a hyperparameter for balancing the target loss and the auxiliary loss, as in equation (16): L = L target + β * L aux (16).

Images