A transformer-based item-behavior cross-sequence recommendation method

Abstract

The application claims a kind of item-behavior cross-sequence recommendation system based on Transformer, it is related to artificial intelligence and recommendation system field, including the following steps: 1) item-behavior sequence modeling, obtains user interaction item sequence and user interaction behavior sequence;2) sequence recommendation system modeling based on Transformer, by embedding representation query, convert the sparse item of one-hot into dense embedding representation vector, according to the embedding representation of sequence level, each item recommendation probability is calculated and the item with the highest recommendation probability is recommended to user;3) sequence representation modeling based on encoder, two kinds of sequence modeling are single sequence, and the embedding representation of sequence is learned;4) item-behavior cross-sequence fusion modeling based on self-attention mechanism.The application has important guiding significance in the research of multi-sequence recommendation system construction, sequence information modeling, multi-sequence fusion and the like.

Description

Technical Field

[0001] This invention relates to methods related to recommender systems and deep learning, specifically to feedforward neural networks, Transformers, and the field of deep learning. Background Technology

[0002] The superior performance and rich modeling methods of deep learning have revitalized it and brought breakthroughs to many fields. Among them, the learning task, loss function, and optimization method are three important foundational concepts in the field of deep learning.

[0003] Deep learning tasks can be categorized into classification and regression tasks based on the type of labels. When the labels are a finite number of discrete labels, and there is no clear numerical meaning between them, the task is a classification task. When the labels are defined as continuous numerical values ​​or have clear numerical meaning, the task is a regression task.

[0004] A loss function is a function that maps random events to non-negative real numbers to characterize their risk. Typically, the properties of the loss function are related to the learning task of deep learning; that is, the deep learning model can be indirectly optimized by minimizing the loss function. Different loss functions are chosen for different learning tasks to describe the model's optimization objective as accurately as possible. In deep learning, given independent and identically distributed training samples (X, y) and a deep learning model... The loss function L can be defined as follows: The formula is used to measure the difference between the probability distribution of the model output and the observed results; where D is a function that measures the probability distribution of the model and the observed results; w represents the model parameters; and p(y|X) is the probability of obtaining the observed result y under the sample feature X.

[0005] For regression problems, there are two common loss function selection schemes: L1 loss function and... and L2 loss function The two measures the estimated values ​​using different but similar methods. The distance to the observed value y is used as a metric. Deep learning models progressively minimize the model's loss function value through optimization algorithms, optimizing the model's loss function value to a local minimum or a global minimum.

[0006] For classification problems, 0-1 loss is a common measure of classification accuracy, as shown in formula (1):

[0007]

[0008] In the formula Let be the estimated value and y be the observed value. However, since discontinuous functions are not conducive to solving problems in deep learning, constructing a surrogate loss function to replace the 0-1 loss function is a common approach. For example, the cross-entropy loss function... Due to its excellent properties such as smoothness and unbiasedness, the cross-entropy loss function is often chosen in deep learning classification tasks to calculate the model error and to optimize the model.

[0009] After determining the loss function of the model, the optimizer continuously updates the model parameters to optimize the model in the direction of reducing the loss function. When the model converges to a certain minimum value, the update of the model parameters stops, and the model for online inference or offline testing is obtained.

[0010] A feedforward neural network (FNN) consists of an input layer, hidden layers, and an output layer.

[0011] Appendix Figure 1 It is a feedforward neural network with two hidden layers. Each node in each hidden layer is called a neuron. The neurons in each layer do not affect each other. Any two neurons in two adjacent layers have a connection. Each connection consists of a linear transformation and a nonlinear activation.

[0012] The linear transformation recombines the input data according to the weight matrix, as shown in formula (2):

[0013]

[0014] In the formula x l This represents the data after linear transformation and recombination; x represents the input data; W f This represents the weight matrix of the feedforward neural network; b f This represents the offset vector.

[0015] Nonlinear activation will provide a nonlinear transformation to the input data, as shown in Equation (3):

[0016] x a =Activate f (x l (3)

[0017] In the formula x a This represents the input after nonlinear activation; Activate f x represents a non-linear activation function; l This represents the data after being recombined through a linear transformation.

[0018] Common activation functions include ReLU(x) l ) = max(xl ,0),LeakyReLU(x l )=max(αx l ,z i ), wait.

[0019] As attached Figure 2 As shown in (a), the derivative of the ReLU function is constant and equal to 1 when the input value is greater than 0; otherwise, both the output value and the derivative value are 0. Therefore, when the input value is less than 0, no adjustment value is generated during backpropagation, which affects the training speed of the deep learning model.

[0020] As attached Figure 2 As shown in (b), LeakyReLU improves upon this by outputting relatively small function values ​​and derivative values ​​when the input value is less than 0.

[0021] As attached Figure 2 As shown in (c), since the derivative of the sigmoid function is located in the interval (0, 0.25], it will cause the gradient vanishing phenomenon after passing through multiple small activation function values ​​during the model learning process. Furthermore, since the exponential operation in sigmoid consumes relatively large computational resources, it is rarely used in feedforward neural networks.

[0022] As attached Figure 2 As shown in (d), the tanh function has a less severe gradient vanishing problem compared to the sigmoid function because its derivative is located in the interval (0,1]. However, since it also involves exponential calculation, it consumes more computational resources.

[0023] CN114780831A discloses a sequence recommendation method and system based on Transformer. The method includes: acquiring the current behavior data of a target user; inputting the current behavior data of the target user and the product information of each product in the product library into a sequence recommendation model to obtain a list of products to be recommended corresponding to the current behavior data; the list of products to be recommended is generated based on the products to be recommended in the product library; and performing a recommendation operation on the target user according to the list of products to be recommended.

[0024] Furthermore, in sequence recommendation using Transformers, some scholars have attempted to utilize network architecture search techniques to find better hyperparameters and superior model structures. However, this method does not delve deeply into the relationship between the two sequences generated during user interaction. Additionally, the two-stage approach combining architecture search and model training is more time-consuming. This invention focuses on the intrinsic relationship between the two sequences generated during user interaction, fusing them through the self-attention mechanism in Transformers and training the model using a one-stage modeling approach. Moreover, this invention emphasizes how to unify the various sequences generated during user interaction and utilize them to obtain better sequence-level embedding representations. Summary of the Invention

[0025] This invention aims to solve the problems of the prior art mentioned above. It proposes a Transformer-based cross-sequence recommendation method for items and behaviors. The technical solution of this invention is as follows:

[0026] A Transformer-based item-behavior cross-sequence recommendation method includes the following steps:

[0027] 1) Item-behavior sequence modeling steps: Obtain user interaction item sequences and user interaction behavior sequences, and model these two sequences;

[0028] 2) Modeling a sequence recommendation system based on Transformer: The sparse, one-hot items are transformed into dense embedding vectors by querying the embedding representation. Then, the encoder structure in Transformer is used to learn the embedding representation of the sequence, thereby obtaining a sequence-level embedding representation that can represent user preferences. Then, the normalized exponential function is used to calculate the recommendation probability of each item based on the sequence-level embedding representation and recommend the item with the highest recommendation probability to the user.

[0029] 3) Sequence representation modeling based on encoder: The two sequences are modeled as a single sequence and the embedding representation of the sequence is learned; the embedding representations of sequence elements generated during multiple user interactions are fused into a single sequence embedding representation through a self-attention mechanism, and the sequence-level embedding representation is learned using a feedforward neural network in deep learning;

[0030] 4) Item-behavior cross-sequence fusion modeling based on self-attention mechanism: multiple sequences of user interaction are fused into a single sequence. The element-level embedding representations of multiple sequences are fused into the element-level embedding representation of a single sequence using query vectors, key vectors, and value vectors based on self-attention.

[0031] Furthermore, step 1) of the item-behavior sequence modeling step involves obtaining user interaction item sequences and user interaction behavior sequences, and modeling these two sequences, specifically including:

[0032] The embedding representation query transforms unique user-interactive items and user interaction behaviors into corresponding dense embedding representation vectors. The similarity between the sequence embedding representation vector and the embedding representation vector of each item is calculated using a similarity function. The recommendation probability of each item is calculated using the Softmax function, and then the item with the highest recommendation probability is recommended to the user.

[0033] Furthermore, the cross-sequence encoder consists of a multi-head cross-sequence fusion module, a feedforward neural network, random deactivation, an activation function, and residual connections;

[0034] Furthermore, the multi-head cross-sequence self-attention fusion function MHAF utilizes a multi-head self-attention mechanism to fuse the embedded representation I′ of the user-interacted item sequence. j Embedded representation of user interaction behavior sequence B′ j In the multi-head self-attention mechanism, the query vector, key vector, and value vector are used to calculate the attention score from the query vector and key vector. The attention score is then used to perform a weighted summation on the value vector to obtain a new embedding representation. For the value vector, the fusion method will be processed in the manner of Transformer. The fusion of the query vector and key vector is divided into two categories: fusion before linear transformation and fusion after linear transformation.

[0035] Furthermore, the pre-linear transformation fusion involves fusing the two sequences before performing a linear transformation to obtain the embedded representation of the fused sequence; then, a linear transformation and matrix multiplication are performed on the fused embedded representation to obtain the attention matrix; the attention score is then obtained through the Softmax function; finally, the attention score is multiplied by the embedded representation of the user interaction item sequence after the linear transformation to obtain the final embedded representation.

[0036] Furthermore, the pre-linear transformation fusion includes additive fusion, multiplicative fusion, and splicing fusion;

[0037] Additive fusion adds the sequences to be fused element by element to obtain the fused embedding representation;

[0038] Multiplication fusion combines the embedded representations of user interaction item sequences and user interaction behavior sequences by multiplying corresponding elements of the sequences one by one.

[0039] The splicing fusion method uses concatenated embeddings to fuse sequences.

[0040] Furthermore, the fusion after linear transformation includes dual-QK fusion, dual-K fusion, dual-QK attention fusion, and triple-QK fusion.

[0041] The advantages and beneficial effects of this invention are as follows:

[0042] This invention utilizes both item sequences and behavior sequences generated by the user during interaction, mapping them to dense embedding vectors to reduce space complexity. Furthermore, it employs an attention mechanism to fuse the two embedding vectors, proposing multiple fusion methods for this purpose. Compared to using multiple models to model the two sequences separately, this invention uses only a single model to model both sequences, resulting in lower time complexity. Attached Figure Description

[0043] Appendix Figure 1 This is a schematic diagram of a feedforward neural network;

[0044] Appendix Figure 2 The graphs are of four activation functions;

[0045] Appendix Figure 3 This is a schematic diagram of the Transformer structure;

[0046] Appendix Figure 4 This is a schematic diagram of the encoder structure in Transformer;

[0047] Appendix Figure 5 A flowchart illustrating the framework of a Transformer-based cross-sequence recommendation system for items and behaviors, provided in an embodiment of the present invention.

[0048] Appendix Figure 6 A flowchart illustrating the framework of the encoding layer in a Transformer-based cross-sequence recommendation system, as provided in this embodiment of the invention.

[0049] Appendix Figure 7 A linear pre-fusion framework based on self-attention mechanism in a Transformer-based item-behavior cross-sequence recommendation system is provided for embodiments of the present invention.

[0050] Appendix Figure 8 The following is a linear post-fusion framework diagram based on self-attention mechanism in the Transformer-based item-behavior cross-sequence recommendation system provided in the embodiments of the present invention; wherein subgraph (a) is double QK fusion, subgraph (b) is double K fusion, subgraph (c) is double QK attention fusion, and subgraph (d) is triple QK fusion. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention.

[0052] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0053] This invention is a Transformer-based item-behavior cross-sequence recommendation system. The method uses PyTorch as the implementation platform and employs deep learning techniques to implement the sequence recommendation system algorithm. It models the item and behavior sequences generated during user interactions in real-world scenarios to obtain their dense embedding representations. Then, a self-attention mechanism is used to fuse various information with sequence characteristics, and an encoder is used to obtain embedding vectors representing user interaction sequence preferences. Finally, the recommendation probability calculated by combining the embedding representations of the items is the item with the highest recommendation probability for the user. This invention's recommendation system, by processing the sequence information generated during user browsing, can automatically model user preferences and thus provide users with items they may be interested in, thereby alleviating the information overload problem in internet applications.

[0054] This invention presents a Transformer-based cross-sequence recommendation system trained using deep learning optimization algorithms. This method improves recommendation system performance by fusing two different sequences. The fusion method using an attention mechanism can employ different approaches depending on the specific situation. Furthermore, the similarity metric function can be selected to suit the task scenario.

[0055] This invention has the following features: a) It models user-interacting items and user-interacting behaviors as embedded representation vectors, and uses matrix multiplication to calculate the similarity between the sequence's embedded representation vector and the item's embedded representation vector, thereby recommending items that the user may be interested in. b) It employs a Transformer encoder structure to learn sequence-level embedded representations that can represent user interests, and fuses the two sequences in the encoder's self-attention mechanism to improve model performance. c) It uses a self-attention mechanism to fuse the two sequences and proposes multiple attention-based fusion methods to adapt to different application scenarios. d) It uses a similarity metric to measure the similarity between the sequence-level embedded representation and the item embedded representation, thereby calculating more reliable recommendation results for the user. e) It implements this invention using Python and uses PyTorch for deep learning training and inference.

[0056] Furthermore, the Transformer discussed in this paper is a deep learning model based on a self-attention mechanism. This model constructs an encoder-decoder structure through self-attention to process sequential data in parallel, and its superior performance has made it popular in many fields. Compared to RNN structures such as GRU and LSTM, which read sequential data element by element and sequentially compute the state variables within the structure, the Transformer has no recursive structure. It can read all elements in the sequence at once, and all elements in the sequence can participate in network training simultaneously, which increases the computational efficiency of the network. Moreover, because the Transformer has no recursive structure, it also avoids the gradient dependency problem found in RNNs, resulting in higher training efficiency.

[0057] A schematic diagram of the Transformer model structure is attached. Figure 3 As shown, it consists of an encoder and a decoder, each of which is composed of residual connections, multi-head self-attention mechanisms, layer normalization, and feedforward neural networks.

[0058] The encoder is responsible for mapping the input to the hidden layer representation and outputting it to the decoder; the decoder then maps the hidden layer representation to the corresponding result according to the different learning tasks.

[0059] The encoder in a Transformer consists of several coding layers. Each coding layer outputs a code containing information about which parts of the input are related to each other, and this code is passed as input to the next coding layer. The decoder performs the opposite function, integrating the contextual information from the encoder output to generate the output sequence.

[0060] Attention mechanisms originated from research on human vision. Because human energy is limited, information is filtered before processing to select information with higher cognitive weight, thus requiring more effort to process it. Therefore, the main function of attention mechanisms is to identify the parts that need attention and allocate more resources to them. In Transformer, a self-attention mechanism is used to calculate the attention score between two elements in a sequence, and a multi-head attention mechanism is integrated to allow different heads to focus on different parts. The calculation formula is shown in formula (4):

[0061] MultiHead(Q,K,V)=Concat(head1,...,head h W O (4)

[0062] In the formula, MultiHead represents the multi-head attention mechanism calculation function; Q represents the query vector; K represents the key vector; V represents the value vector; Concat represents the concatenation operation; head i This represents the individual heads in a multi-head self-attention mechanism, where each head represents a self-attention mechanism, and h represents the number of heads; W O This represents the weight matrix.

[0063] The self-attention mechanism of each head in formula (4) is shown in formula (5):

[0064]

[0065] In the formula, softmax represents the normalized exponential function; d k The dimension of the K vector is used to stabilize the gradient during training. It's clear from the formula that the multi-head self-attention mechanism consists of multiple self-attention mechanisms and linear transformations. The result of each self-attention mechanism is obtained by weightedly combining the attention scores calculated from the linear transformation of the Query and Key vectors with the Value vector.

[0066] The encoder in a Transformer consists of multiple stacked coding layers, each composed of a multi-head self-attention mechanism and a fully connected layer. The coding layers are shown in the attached diagram. Figure 4 As shown. The encoding layer consists of positional encoding, multi-head self-attention mechanism, residual connections, and dense connections. Since the Transformer model does not have a recursive structure that expands with the sequence, positional encoding is needed to provide positional information of elements in the sequence to add sequential relationship information to the model. The added residual structure, to some extent, prevents gradient vanishing. The encoder is formally shown in Equation (6):

[0067] X out =Encoder(X) (6)

[0068] In the formula, X represents the input of the entire encoder.

[0069] Specifically, firstly, a dense embedding representation is extracted from the input sequence and positional encoding is added, as shown in Equation (7):

[0070] X e =Embedding_Lookup(X)+Positional-Encoding (7)

[0071] In the formula X eThis represents the sum of the dense representation sequence and the positional encoding; Embedding_Lookup represents the function to look up the embedding representation; X represents the input sequence; Positional-Encoding represents the positional encoding function.

[0072] Next, X e The embedded representation of the sequence dense representation is learned by inputting into the multi-head self-attention mechanism, as shown in Equation (8):

[0073] MSA = MultiHead(X) e ,X e ,X e (8)

[0074] In the formula, MultiHead represents the multi-head self-attention mechanism.

[0075] Next, the results MSA of the multi-head self-attention mechanism and the input X were compared. e Perform residual connections and layer normalization, as shown in formula (9):

[0076] X att =LayerNorm(X e +MSA) (9)

[0077] In the formula X att This represents the result after layer normalization; LayerNorm represents the layer normalization function.

[0078] Then the result of the layer normalization X is... att Perform a linear transformation, then a nonlinear activation, and then another linear transformation, as shown in formula (10):

[0079] X hid =W En1Atvt (X att W En2 (10)

[0080] In the formula X hid W represents the final result of the linear transformation. En1 and W En2 represents the weight matrix; Activate represents the activation function, usually the ReLU activation function.

[0081] Finally, the result X after level normalization is... att The result of the linear transformation X hid Perform residual connections and layer normalization, as shown in Equation (11):

[0082] X out =LayerNorm(X att +X hid (11)

[0083] In the formula X out This represents the embedded representation of the coding layer output; LayerNorm represents the layer normalization function.

[0084] First, define the sequence set. Here Let s represent the set containing all sequences, where s j Represents a set The j-th sequence in the set, where N represents the number of sequences in the set (i.e., ...). For each sequence s j , in Let j represent the sequence of user-interacted items in the j-th sequence, and Let j represent the sequence of user interaction behaviors for the j-th sequence. For each item sequence... Here Let n represent the k-th item in the j-th sequence, and n represent the k-th item in the j-th sequence. j Then it represents the length of the sequence of the j-th item (i.e., ),and Where I is the set of all items. For each user behavior sequence Here Let n represent the k-th user action in the j-th sequence, and n represent the user action in the j-th sequence. j Then it represents the length of the j-th user interaction sequence (i.e. Obviously there is and Here, B is the set of all user behaviors.

[0085] The goal of the model established in this invention is to achieve the desired result from a given sequence s. j To predict the items that users will interact with n+1 To achieve this goal, sequence s j It will be input into the model and used based on the generated matching score. To obtain candidate items i n+1 Where i n+1 ∈I. Based on the matching score The model will provide the top-k items with the highest scores, and the items with the highest scores will be recommended to the user.

[0086] The model proposed in this invention is attached. Figure 5As shown in the figure, solid lines represent data flow; solid-line boxes represent specific deep learning modules; dashed-line boxes represent enlarged structural diagrams of a certain deep learning module; colored rectangles represent different embedding representations; and uncolored rectangles represent different modules in the model. To improve the performance of the sequence recommendation system, the model described in this invention includes two sequences: user interaction item sequences and user interaction behavior sequences.

[0087] First, to transform the sparse one-hot representations of user interaction item sequences and user interaction behavior sequences into dense embedding representation vectors, the model described in this chapter will use embedding... I and Embedding B Find user interaction item sequences User interaction behavior sequence The dense embedding representation vector is shown in Equation (12):

[0088]

[0089] In the formula Represents a sequence of user-interacted items Embedded representation; n j Indicates the length of the sequence; dim I The dimension of the item embedding vector; Embedding I By looking up the item embedding representation matrix Retrieve the embedded representation of the item with the corresponding number; Represents a sequence of user interaction behaviors The embedding representation, where dim B The dimension of the embedding vector representing user interaction behavior; Embedding B By looking up the embedding representation matrix from user behavior The embedded representation of the user interaction behavior corresponding to the number is obtained.

[0090] Next, the embedding representation vectors of all elements in the sequence learned by the encoder are obtained through the encoder constructed by the L-layer cross-sequence encoding layer, as shown in Equation (13):

[0091] S CSE =CSEncoder(I′) j ,B′ j In formula (13) This represents the output of the cross-sequence encoder, which is composed of the outputs of the L-layer cross-sequence coding layers.

[0092] Then take out S CSE The nth layer of the Lth layer j Each element serves as the embedding representation vector for the entire sequence, as shown in equation (14):

[0093] s cs =GetOut(S CSE (14)

[0094] In the formula This represents the embedding representation of the entire sequence; GetOut represents the above-mentioned vector extraction operation.

[0095] Then, through the similarity function f sim Calculate the sequence embedding representation vector s cs With E I The similarity score for each element is calculated, and the recommendation probability for each item is obtained by applying a normalized exponential function, softmax. As shown in equation (15):

[0096]

[0097] In the formula Let be a vector of size 1×||I||, with values ​​ranging from [0,1]. Typically, the similarity function can be Euclidean distance, cosine similarity, etc. This chapter uses equation (16) to quickly calculate the similarity between vectors:

[0098]

[0099] To train the model proposed in this invention, the model uses the cross-entropy function as the loss function L, as shown in equation (17):

[0100]

[0101] And all parameters of the model are trained by minimizing the loss function L, as shown in equation (18):

[0102] Θ = arg min L (18)

[0103] In the formula, Θ represents all parameters in the model.

[0104] The cross-sequence encoder shown in equation (13) will be described in detail below. This module consists of a multi-head cross-sequence fusion module, a feedforward neural network, random deactivation, an activation function, and residual connections. Its structure is shown in the attached figure. Figure 6 As shown.

[0105] First, the embedding representation of the item sequence I′ is... j Embedded representation of user interaction behavior B′ j The input is fed into the multi-head attention mechanism and cross-sequence fusion module, as shown in Equation (19):

[0106] S fus =MHAF(I′)j ,B′ j In formula (19) It represents the embedding representation of the sequence after fusing the sequence of user-interacted items and the sequence of user-interacted behaviors; MHAF represents the multi-head cross-sequence fusion function.

[0107] Next, the embedding representation S of the fused sequence is... fus Embedded representation of the sequence of items interacting with the user I′ j Perform residual connections and layer normalization, as shown in equation (20):

[0108] S 1stLN =LayerNorm(I′) j +S fus (20)

[0109] In the formula The layer normalization fusion sequence is represented by LayerNorm; the layer normalization function is represented by LayerNorm.

[0110] Then the layer-normalized fusion sequence S 1stLN The input is fed into the feedforward neural network, as shown in equation (21):

[0111] S act =Activate(S 1stLN W CSE1 +b CSE1 ) (twenty one)

[0112] In the formula This represents the fused sequence after passing through the feedforward neural network; Activate represents the activation function, which is the ReLU activation function in this invention. Represents the weight matrix; This represents the offset vector.

[0113] Finally, the fused sequence S after passing through the feedforward neural network is... act The input is densely connected and subjected to nonlinear activation and random deactivation. This is then followed by a layer-normalized fusion sequence S. 1stLN Perform residual connections, and then perform layer normalization, as shown in equation (22):

[0114] S 2ndLN =LayerNorm(S 1stLN +Dropout(S act W CSE2 +b CSE2 )) (twenty two)

[0115] In the formula This represents the fused sequence after normalization by this layer; LayerNorm represents the layer normalization function; Dropout represents the random deactivation function; Represents the weight matrix; This represents the offset vector.

[0116] Finally, various schemes for the multi-head cross-sequence self-attention fusion function MHAF, as shown in equation (19), will be described in detail. This module utilizes a multi-head self-attention mechanism to fuse the embedded representation I′ of user-interacted item sequences. j Embedded representation of user interaction behavior sequence B′ j In the multi-head self-attention mechanism, the query vector, key vector, and value vector are used to calculate an attention score from the query and key vectors. This attention score is then used to weight and sum the value vectors to obtain a new embedding representation. Therefore, the value vector is processed in the manner described in this invention, following the approach used in Transformers. The fusion of query and key vectors is broadly categorized into two types: fusion before linear transformation and fusion after linear transformation.

[0117] The overall framework for fusion before linear transformation is attached. Figure 7 As shown, the model performs sequence fusion before linear transformation of the two sequences to obtain the embedded representation of the fused sequence; then, it performs linear transformation and matrix multiplication on the fused embedded representation to obtain the attention matrix; then, it obtains the attention score through the Softmax function; finally, it multiplies the result with the embedded representation of the linearly transformed user interaction item sequence to obtain the final embedded representation.

[0118] For the fusion function F before linear transformation, there are usually three choices.

[0119] Additive fusion adds the sequences to be fused element by element to obtain the fused embedding representation. Additive fusion is shown in equation (23):

[0120] S fus =I′ j +B′ j In equation (23) The fused embedding representation is represented as Q and K in equation (4), and the linearly transformed embedding representation of the user interaction item sequence is represented as V in equation (4) to obtain the additive fusion; this fusion method needs to satisfy dim I =dim B =dim.

[0121] Multiplicative fusion fuses the embedded representations of the user interaction item sequence and the user interaction behavior sequence by multiplying corresponding elements of the sequence. Multiplicative fusion is shown in equation (24):

[0122] S fus =I′ j ⊙B′ j In equation (24) The fused embedding representation is represented as Q and K in equation (4), and the embedded representation of the linearly transformed user interaction item sequence is represented as V in equation (4) to obtain the multiplicative fusion; this fusion method needs to satisfy dim I =dim B =dim.

[0123] The splicing fusion is performed by fusing sequences through concatenated embeddings. The splicing fusion is shown in equation (25):

[0124]

[0125] In the formula The fused embedding representation is represented as Q and K in equation (4), and the embedded representation of the linearly transformed user interaction item sequence is represented as V in equation (4), thus obtaining the splicing fusion.

[0126] Appendix Figure 8 (a) illustrates the dual QK fusion proposed in this invention. This model takes the input user-interactive item sequence I′ as an example. j User interaction behavior sequence B′ j Linear transformations are performed on both query and key vectors to obtain their respective query and key vectors. These vectors are then fused with the corresponding vectors of another sequence using a fusion function F. Matrix multiplication is then performed on the fused query and key vectors, and the attention score matrix is ​​obtained using the Softmax function. Finally, this matrix is ​​multiplied by the value vector to obtain the sequence embedding representation S. fus The specific calculation process is shown below.

[0127] First, analyze the user interaction item sequence I′ j User interaction behavior sequence B′ j Perform a linear transformation, and then use the fusion function F to obtain the fused query vector Fus. Q As shown in equation (26):

[0128] Fus Q =F(I′) j W IQ +b IQ ,B′ j W BQ +b BQ (26)

[0129] In the formula Fus Q This represents the merged query vector, whose dimensions may vary depending on the fusion function F. These represent the weight matrices corresponding to the user interaction item sequence and the user interaction behavior sequence, respectively; Let dim represent the offset vectors corresponding to the user interaction item sequence and the user interaction behavior sequence, respectively. h This is to unify the dimension of the embedded representation of user interaction behavior with the dimension of the embedded representation of user interaction items, and the dimension of the hidden state.

[0130] Secondly, the user interaction item sequence I′ j User interaction behavior sequence B′ j Perform a linear transformation, and then use the fusion function F to obtain the fused key vector Fus. K As shown in equation (27):

[0131] Fus K =F(I′) j W IK +b IK ,B′ j W BK +b BK (27)

[0132] In the formula Fus K This represents the fused key vector, whose dimensions may vary depending on the fusion function F; These represent the weight matrices corresponding to the user interaction item sequence and the user interaction behavior sequence, respectively; These represent the offset vectors for the corresponding user interaction item sequence and user interaction behavior sequence, respectively.

[0133] Then, the user interaction item sequence I′ j A linear transformation is performed to obtain the value vector V, as shown in equation (28):

[0134] V = I′ j W IV +b IV (28)

[0135] In the formula Represents a value vector; Represents the weight matrix; This represents the offset vector.

[0136] Finally, the query vector Fus is obtained through matrix multiplication. Q and key vector Fus K The attention matrix is ​​obtained by using the Softmax function to get the attention score of each sequence element relative to other elements, and then multiplying it with the value vector V to obtain the fused representation S. fus As shown in equation (29):

[0137]

[0138] In the formula Represents the key vector Fus K The dimension of gradient is used to stabilize gradients during training.

[0139] As attached Figure 8 As shown in (b), this figure illustrates the dual-K fusion proposed in this invention. This model fuses the Key vectors of two sequences using a fusion function F, then multiplies them with the Query vector of the user-interacted item sequence and calculates the attention score matrix using the Softmax function. Finally, it multiplies this matrix with the Value vector to obtain the sequence's embedding representation S. fus The specific calculation process is shown below.

[0140] First, the embedding representation I′ of the user interaction item sequence is... j A linear transformation is performed to obtain the query vector Q, as shown in equation (30):

[0141] Q = I′ j W IQ +b IQ (30)

[0142] In the formula A query vector representing a sequence of items interacted with by the user; Represents the weight matrix; This represents the offset vector.

[0143] Secondly, the embedding representation I′ of the user interaction item sequence. j Embedded representation of user interaction behavior sequence B′ j Perform linear transformations on each, and then fuse the two embedded representations after the linear transformations using a fusion function F, as shown in equation (31):

[0144] Fus K =F(I′) j W IK +b IK ,B′ j W BK +b BK (31)

[0145] In the formula This represents the merged Key vector; the fusion function F can be one of the aforementioned addition, multiplication, or concatenation fusion methods. These represent the weight matrices corresponding to the user interaction item sequence and the user interaction behavior sequence, respectively; These represent the offset vectors corresponding to the user interaction item sequence and the user interaction behavior sequence, respectively.

[0146] Then, the embedding representation I′ of the user interaction item sequence is... jA linear transformation is performed to obtain the value vector V, as shown in equation (32):

[0147] V = I′ j W IV +b IV (32)

[0148] In the formula Represents a value vector; Represents the weight matrix; This represents the offset vector.

[0149] Finally, the query vector Q and the key vector Fus are obtained through matrix multiplication. K The attention matrix is ​​obtained by using the Softmax function to get the attention score of each sequence element relative to other elements, and then multiplying it with the value vector V to obtain the fused representation S. fus As shown in equation (33):

[0150]

[0151] In the formula This is the final embedded representation of the fusion.

[0152] Appendix Figure 8 (c) Demonstrates the proposed dual-QK attention fusion. This model fuses the Query and Key vectors of two sequences obtained through a feedforward neural network using two fusion functions F. Then, it calculates an attention matrix by combining the fused Query vector with the Key vector of the user-interacted item sequence, and calculates an attention matrix by combining the fused Key vector with the Query vector of the user behavior sequence. Finally, it adds these two attention matrices and passes the Softmax function to obtain the attention score matrix, which is then multiplied by the Value vector to obtain the sequence embedding representation Sf. us The specific calculation process is shown below.

[0153] First, analyze the user interaction item sequence I′ j User interaction behavior sequence B′ j Perform linear transformations on each vector to obtain the corresponding query vector, as shown in equation (34):

[0154]

[0155] In the formula and These represent the query vector embeddings of the linearly transformed user interaction item sequence and user interaction behavior sequence, respectively. and These represent the weight matrices corresponding to the user interaction item sequence and the user interaction behavior sequence, respectively; These represent the offset vectors corresponding to the user interaction item sequence and the user interaction behavior sequence, respectively.

[0156] Then, the user interaction item sequence I′ j User interaction behavior sequence B′ j Performing linear transformations on each yields the corresponding key vectors, as shown in equation (35):

[0157]

[0158] In the formula and The key vector embeddings represent the linearly transformed sequences of user-interacted items and user-interacted behaviors, respectively. and These represent the weight matrices corresponding to the user interaction item sequence and the user interaction behavior sequence, respectively; These represent the offset vectors corresponding to the user interaction item sequence and the user interaction behavior sequence, respectively.

[0159] Secondly, the user interaction item sequence I′ j A linear transformation is performed to obtain the value vector of the user-interacted item sequence, as shown in equation (36):

[0160] V = I′ j W IV +b IV (36)

[0161] In the formula This represents the value vector obtained after a linear transformation. Represents the weight matrix; This represents the offset vector.

[0162] Then, for the query vector Q respectively I and Q B To perform fusion, the key vector K I and K B The fusion is performed as shown in equation (37):

[0163]

[0164] In the formula and represents the merged query vector and key vector, respectively; F represents the fusion function.

[0165] Finally, the merged query vector Fus Q The key vector K of the sequence of items interacted with by the user I Multiplying them yields the attention matrix, which is then used to generate the query vector Q of the user interaction sequence. B and the fused key vector Fus KThe two attention matrices are multiplied to obtain the attention matrix; then the two attention matrices are added together and the Softmax function is used to obtain the attention score matrix; finally, the matrix is ​​multiplied with the value vector V to obtain the fused embedding representation, as shown in equation (38):

[0166]

[0167] In the formula This represents the final fused embedding representation.

[0168] As attached Figure 8 As shown in (d), this represents the three-QK fusion proposed in this invention. For a user interaction behavior sequence, it is fused with a user interaction item sequence using a fusion function F, and then its attention matrix is ​​calculated; for a user interaction item sequence, its attention matrix is ​​directly calculated; then the attention matrix between the two sequences is calculated; next, the three attention matrices are added together and the attention score matrix is ​​calculated using the Softmax function, and finally multiplied by the Value vector to obtain the embedding representation S of the sequence. fus The specific calculation process is shown below.

[0169] First, the embedding representation of the user interaction item sequence is I′. j Embedded representation of user interaction behavior sequence B′ j The fusion process yields the embedded representation B of the fused user interaction behavior sequence. fus As shown in equation (39):

[0170] B fus =F(I′) j ,B′ j (39)

[0171] In the formula F represents the merged sequence vector of user interaction behaviors; F represents the fusion function.

[0172] Secondly, the embedding representation I′ of the user interaction item sequence. j Perform a linear transformation to obtain the query vector Q of the user-interacted item sequence. innerI and bond vector K innerI As shown in equation (40):

[0173]

[0174] In the formula and These represent the query vector and key vector of the user-interacted item sequence after linear transformation, respectively. and These represent the weight matrices respectively; and These represent offset vectors.

[0175] Then, the embedding representation B of the user interaction behavior sequence fus Perform a linear transformation to obtain the query vector Q of the user interaction behavior sequence. innerB and bond vector K innerB As shown in equation (41):

[0176]

[0177] In the formula and These represent the query vector and key vector of the user interaction behavior sequence after linear transformation, respectively. and Represents the weight matrix; and This represents the offset vector.

[0178] Furthermore, the user interaction item sequence embedding representation I′ j Perform a linear transformation to obtain the query vector Q for the cross-attention mechanism. inter And embed the user interaction behavior sequence into a representation B′ j Perform a linear transformation to obtain the key vector K of the cross-attention mechanism. inter As shown in equation (42):

[0179]

[0180] In the formula The query vector representing the cross-attention mechanism; The key vector representing the cross-attention mechanism; and Represents the weight matrix; and This represents the offset vector.

[0181] Next, we will embed the user interaction item sequence into an I′ representation. j A linear transformation is performed to obtain the value vector, as shown in equation (43):

[0182] V = I′ j W IV +b IV (43)

[0183] In the formula The value vector representing the cross-attention mechanism; Represents the weight matrix; This represents the offset vector.

[0184] Finally, the query vector Q of the user interaction item sequence is... innerI and bond vector K innerIMultiply to obtain the attention matrix within the user-interacting item; then multiply the query vector Q of the user interaction sequence. innerB and bond vector K innerB Multiplying yields the attention matrix within the user interaction behavior; the query vector Q of the cross-attention mechanism is then used. inter and bond vector K inter Multiplying these three attention matrices yields the cross-attention matrix; then, the three attention matrices are summed and exponentially normalized to obtain the attention score matrix. Finally, this matrix is ​​multiplied by the value vector V to obtain the fused representation S. fus As shown in equation (44):

[0185]

[0186] In the formula This represents the final fused embedding representation.

[0187] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0188] The above embodiments should be understood as illustrative only and not as limiting the scope of protection of the present invention. After reading the description of the present invention, those skilled in the art can make various alterations or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A Transformer-based cross-sequence recommendation method for items and behaviors, characterized in that, Includes the following steps: 1) Item-behavior sequence modeling steps: Obtain user interaction item sequences and user interaction behavior sequences, and model these two sequences; 2) Modeling a sequence recommendation system based on Transformer: The sparse, one-hot items are transformed into dense embedding vectors by querying the embedding representation. Then, the encoder structure in Transformer is used to learn the embedding representation of the sequence, thereby obtaining a sequence-level embedding representation that can represent user preferences. Then, the normalized exponential function is used to calculate the recommendation probability of each item based on the sequence-level embedding representation and recommend the item with the highest recommendation probability to the user. 3) Sequence representation modeling based on encoder: The two sequences are modeled as a single sequence and the embedding representation of the sequence is learned; The sequence element embeddings generated during multiple user interactions are fused into a single sequence embedding representation using a self-attention mechanism, and a feedforward neural network in deep learning is used to learn the sequence-level embedding representation. 4) Item-behavior cross-sequence fusion modeling based on self-attention mechanism: multiple sequences of user interactions are fused into a single sequence. The element-level embedding representations of multiple sequences are fused into the element-level embedding representation of a single sequence using query vectors, key vectors, and value vectors based on self-attention. Step 1) Item-Behavior Sequence Modeling Step: Obtain user interaction item sequences and user interaction behavior sequences, and model these two sequences, specifically including: The query transforms unique user-interactive items and user interactions into corresponding dense embedding vectors. A similarity function is used to calculate the similarity between the sequence embedding vector and the embedding vector of each item. The Softmax function is then used to calculate the recommendation probability of each item, and the item with the highest recommendation probability is recommended to the user. The process of transforming unique user-interactive items and user-interactive behaviors into corresponding dense embedding representation vectors through embedding representation queries specifically includes: First, define the sequence set. , here Let represent the set containing all sequences, where Represents a set The Middle A sequence, This represents the number of sequences in the set, for each sequence. , ,in Indicates the first A sequence of user-interacting items, and Indicates the first A sequence of user interaction behaviors; for each item sequence , , here Indicates the first The first sequence an item, and Then it means the first The length of the sequence of items, and ,in It is a collection of all items; for each user behavior sequence , , here Indicates the first The first sequence Individual user behavior, and Then it means the first The length of a user interaction sequence is Obviously there is ,and , here It is a collection of all user behaviors; Find user interaction item sequences User interaction behavior sequence The dense embedding representation vector is shown in Equation (12): (12) In the formula Represents a sequence of user-interacted items Embedded representation; Indicates the length of the sequence; The dimension of the item embedding vector; By looking up the item embedding representation matrix Retrieve the embedded representation of the item with the corresponding number; Represents a sequence of user interaction behaviors The embedding representation, where The dimension of the embedding vector representing user interaction behavior; By looking up the embedding representation matrix from user behavior Obtain the embedded representation of the user interaction behavior corresponding to the ID; Next, through the The encoder constructed by the cross-sequence encoding layer obtains the embedding representation vectors of all elements in the sequence learned by the encoder, as shown in Equation (13): (13) In the formula This represents the output of the cross-sequence encoder, which is... It is formed by concatenating the outputs of the cross-sequence coding layers; Then take it out The The first layer Each element serves as the embedding representation vector for the entire sequence, as shown in equation (14): (14) In the formula An embedding representation of the entire sequence; This indicates the above vector extraction operation; Then through the similarity function Calculate the sequence embedding representation vector and Similarity score for each element And through the normalized exponential function The recommended probability of obtaining each item As shown in equation (15): (15) In the formula ; For size A vector whose value range is . The similarity function uses equation (16) to quickly calculate the similarity between vectors: (16)。 2. The Transformer-based cross-sequence recommendation method according to claim 1, characterized in that, The cross-sequence encoder consists of a multi-head cross-sequence fusion module, a feedforward neural network, random deactivation, an activation function, and residual connections. First, the item sequence is embedded in the representation. Embedded representation of user interaction behavior The input is fed into the multi-head attention mechanism and cross-sequence fusion module, as shown in Equation (19): (19) In the formula This represents the embedding representation of a sequence that combines a sequence of user-interacted items and a sequence of user-interacted behaviors. This represents a multi-head cross-sequence fusion function; Next, the embedding representation of the fused sequence is... Embedded representation of user-interacting item sequences Perform residual connections and layer normalization, as shown in equation (20): (20) In the formula The fused sequence after normalization of the representation layer; Indicates the layer normalization function; Then the fusion sequence after layer normalization The input is fed into the feedforward neural network, as shown in equation (21): (21) In the formula This represents the fused sequence after passing through a feedforward neural network; This represents the activation function. Represents the weight matrix; Represents the offset vector; Finally, the fused sequence after passing through the feedforward neural network is... The input is densely connected and subjected to nonlinear activation and random deactivation; then it is fused with a layer-normalized sequence. Perform residual connections, and then perform layer normalization, as shown in equation (22): (22) In the formula This represents the fused sequence after normalization at this layer; Indicates the layer normalization function; Represents the random deactivation function; Represents the weight matrix; This represents the offset vector.

3. The item-behavior cross-sequence recommendation method based on Transformer according to claim 2, characterized in that, The multi-head cross-sequence self-attention fusion function Fusing embedded representations of user-interacted item sequences using multi-head self-attention mechanism Embedded representation of user interaction behavior sequences In the multi-head self-attention mechanism, the query vector, key vector, and value vector are used to calculate the attention score from the query vector and key vector. The attention score is then used to perform a weighted summation on the value vector to obtain a new embedding representation. For the value vector, the fusion method will be processed in the manner of Transformer. The fusion of the query vector and key vector is divided into two categories: fusion before linear transformation and fusion after linear transformation.

4. The Transformer-based cross-sequence recommendation method according to claim 3, characterized in that, The pre-linear transformation fusion involves fusing the two sequences before performing a linear transformation to obtain the embedded representation of the fused sequence. Then, a linear transformation and matrix multiplication are performed on the fused embedded representation to obtain the attention matrix. The attention score is then obtained through the Softmax function. Finally, the attention score is multiplied by the embedded representation of the user interaction item sequence after the linear transformation to obtain the final embedded representation.

5. The Transformer-based cross-sequence recommendation method according to claim 4, characterized in that, The pre-linear transformation fusion includes additive fusion, multiplicative fusion, and splicing fusion; Additive fusion adds the sequences to be fused element by element to obtain the fused embedding representation; Multiplication fusion combines the embedded representations of user interaction item sequences and user interaction behavior sequences by multiplying corresponding elements of the sequences one by one. The splicing fusion method uses concatenated embeddings to fuse sequences.

6. The Transformer-based cross-sequence recommendation method according to claim 4, characterized in that, The fusion after linear transformation includes dual-QK fusion, dual-K fusion, dual-QK attention fusion, and triple-QK fusion.

Images