Recommendation method and system based on self-attention sequence

Abstract

The invention relates to a recommendation method and system based on a self-attention sequence, and the method comprises the following steps: A, obtaining the text information and user behavior data of a commodity, and constructing a training set; b, a deep learning network model is constructed, the deep learning network model uses commodity text information to calculate and obtain commodity embedding, and the behavior pattern and interest evolution of a user are more comprehensively understood by combining a text pre-training model and time coding based on a self-attention mechanism; training a deep learning network model by using the training set to obtain a trained deep learning network model; and step C, inputting commodity and user data into the trained deep learning network model, and outputting the score ranking of the current user on the target commodity. The method and the system are favorable for improving the prediction accuracy of the recommendation system.

Description

Technical Field

[0001] The present invention belongs to the field of recommendation systems, and in particular relates to a self-attention sequence recommendation method and system. Background Art

[0002] As an advanced form of recommendation system, the sequential recommendation system focuses on using the sequence data of users' historical behaviors to predict their future behaviors. The core of this system is to understand and utilize the time series characteristics of user behaviors to provide more personalized and accurate recommendations. Compared with traditional recommendation systems, the sequential recommendation system places more emphasis on capturing the temporal dynamics and evolutionary patterns of user behaviors, and can effectively reflect users' current interests and preferences.

[0003] However, researchers face multiple challenges in building efficient sequential recommendation systems. One of the main challenges is how to effectively process and parse large amounts of sequential data, especially when parsing the complex interactions between users and products. Traditional methods often ignore detailed information in user behavior sequences, such as text descriptions and timestamp information, which are crucial for understanding users' specific interests. In addition, focusing only on user behavior patterns while ignoring the temporal context in which the behavior occurs may lead to an inaccurate understanding of the user's current interests. Summary of the invention

[0004] The purpose of the present invention is to provide a self-attention sequence recommendation method and system, which are conducive to improving the accuracy of recommendation system prediction.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is: a self-attention sequence recommendation method, comprising the following steps:

[0006] Step A: Obtain product text information and user behavior data to build a training set;

[0007] Step B: construct a deep learning network model G, wherein the deep learning network model G uses the product text information to calculate the product embedding, and more comprehensively understands the user's behavior pattern and interest evolution by combining the text pre-training model and the time coding based on the self-attention mechanism; train the deep learning network model G using the training set to obtain a trained deep learning network model;

[0008] Step C: Input the product and user data into the trained deep learning network model G, and output the current user's rating ranking for the target product.

[0009] Furthermore, the step A specifically includes the following steps:

[0010] Step A11: Build a text dictionary of product descriptions and create a text description for each product, including all relevant text information of the product, to ensure that the model has enough information to generate accurate product embeddings;

[0011] Step A12: Preprocess the user interaction sequence: construct the user interaction sequence based on the user's historical interaction information in Represents the text dictionary content corresponding to the i-th item interacted by user u, represents the time point of the i-th item interacted by user u;

[0012] Step A13: Convert each user's interaction sequence into a set of fixed-length item sequences S = (s1, s2, ..., s L ), where s i That is, the user's interactive information Where L represents the maximum sequence length. If the user's interaction sequence length is greater than L, we use the most recent L interactions. If the user's interaction sequence is less than L, we pad the items on the left of the item sequence until the length is L. For timestamp sequences, we use the first timestamp to fill.

[0013] Furthermore, the step B specifically includes the following steps:

[0014] Step B1: Divide the time interval into different intervals and assign an embedding vector to each interval; by adjusting the maximum relative time interval r max The time interval is adjusted by using the scaling factor φ, and all relative time intervals are clipped and scaled to the range [0, k], thereby obtaining the scaled relative time interval sequence T s ; Create a trainable interval matrix Embedding used for subsequent calculation of timestamp;

[0015] Step B2: Use the pre-trained BERT model to extract features of the product description text and convert the features into the initial embedding representation; during the model training process, the embedding representation remains unchanged; apply the self-attention layer and the fully connected layer to extract information to form the embedding matrix E for each product I ; In order to capture the position information of the sequence, a learnable position embedding matrix P is introduced into the product embedding matrix E I In order to enhance the model's ability to recognize location information, E I The decoder will be used in the model to further extract and fuse information to ensure that the model can understand and process complex commodity text data;

[0016] Step B3: Define a window function F win , the window function F win is an adjusted Hamming window function, the weight is calculated by relative position and a trainable parameter w; the window function F win It is used to capture the relative distance between different elements in the sequence. The weight depends on the relative position of different elements in the sequence. For each element i in the sequence, the weights of other elements j related to the element are calculated. These weights are calculated by the window function F. win Then, using these weights and the embedding of other elements, we can calculate the weighted timestamp embedding. Apply the above calculation process to all timestamps and convert the timestamp sequence T s Convert to a time embedding matrix E that preserves continuous dependencies T ;

[0017] Step B4: Apply the residual attention module (RAB) as the encoder. The RAB module uses the temporal embedding matrix E T Perform self-attention operation, where E T At the same time, it serves as a query, key, and value, so that the model can adaptively emphasize or suppress information at a specific time point, thereby capturing temporal dependencies and contextual relevance. After the self-attention operation, the RAB module introduces nonlinearity through two fully connected layers, which enhance the expressiveness of the model through the ReLU activation function. In order to improve the generalization ability of the model and prevent overfitting, the RAB module also applies layer normalization and dropout techniques. Layer normalization standardizes the outputs of different neurons to stabilize the learning process and make training more efficient. Dropout improves the robustness and generalization ability of the model by temporarily discarding some connections in the neural network at random.

[0018] Step B5: The decoder processes and integrates temporal and sequence data by connecting two residual attention modules (RAB) in series, where the low-level RAB module is used to capture the relationship between all items in the sequence and receives the embedding matrix E I As query, key and value, to capture the direct commodity relationship within the sequence; the high-level RAB module uses the output of the low-level RAB module as query and the output of the encoder as key and value input to further process and refine the information; after each RAB module, the model introduces nonlinearity by adding a feedforward neural network (FFN) and uses layer normalization and dropout techniques for regularization;

[0019] Step B6: Use the decoder to obtain the user interest vector r i Perform a dot product operation on the embedding of the potential item to obtain the interest score r i,t ; The cross entropy function is used to calculate the prediction loss to measure the difference between the predicted probability distribution and the actual label; then, according to the target loss function, the parameter gradients in the deep learning network model are calculated by the back propagation method, and these parameters are updated using the stochastic gradient descent method, so as to gradually approach the minimum value of the loss function to make the model prediction more accurate;

[0020] Step B7: When the loss value generated by the deep learning network model is less than the set threshold or reaches the maximum number of iterations, the training of the deep learning network model G is terminated.

[0021] Furthermore, the step B1 specifically includes the following steps:

[0022] Step B11: Divide the time interval: First, evenly divide the time interval into different intervals to convert the continuous time information into discrete form, making it easier for the model to process;

[0023] Step B12: Assigning temporal embedding vectors: Assign a unique embedding vector to each temporal interval; these embedding vectors represent the characteristics of each temporal interval, enabling the model to recognize and utilize temporal information;

[0024] Step B13: Convert the timestamp into a time embedding matrix that maintains continuous correlation; convert the timestamp sequence into a relative time interval sequence, T r =(0,t2-t1,t3-t1,…,t L -t1);

[0025] Step B14: Adjust and scale the time interval: Use the maximum relative time interval r max and scaling factor φ to adjust the time interval; Represents the maximum relative time interval, where R u is the set of relative time intervals of all users, and the scale factor φ is a hyperparameter that scales the maximum relative time interval to a reasonable range;

[0026] Step B15: Scale all relative time intervals to [0, k] to obtain the scaled relative time interval sequence T s , as shown below:

[0027]

[0028] in, represents the time point of user s's i-th interaction, k is the number of time intervals; through the above calculation, the original discrete time is mapped to one of the k time intervals;

[0029] Step B16: Create a trainable embedding matrix Used to represent time intervals, where d is the dimension size;

[0030] Step B17: k embeddings correspond to k time intervals; after the timestamp is mapped to the i-th of the k intervals through the above calculation, its timestamp embedding is mapped to the matrix M T The corresponding e i .

[0031] Furthermore, the step B2 specifically includes the following steps:

[0032] Step B21: Use the pre-trained BERT model to convert the content in the product description text dictionary into the initial embedding of each product; generate the initial embedding matrix It remains unchanged during model training and serves as the basis for subsequent steps, where N represents the number of products;

[0033] Step B22: Get the initial embedding matrix E BERT After that, the embedding matrix is ​​further processed using the self-attention layer and the fully connected layer to obtain a trainable embedding matrix The self-attention mechanism helps the model identify and emphasize the most important parts of the text, thereby better understanding the relationship between products and their key attributes;

[0034] Step B23: Construct the behavior sequence of each user and add the learnable position embedding matrix P to the embedding matrix of the product sequence:

[0035]

[0036] Furthermore, in step B3, the time embedding matrix that maintains continuous dependencies is calculated using weights Each row is calculated as follows:

[0037]

[0038] Among them, e j From the matrix M I , weight The calculation using the window function is as follows:

[0039]

[0040]

[0041] Where w is the size of the window function,

[0042] The generated time embedding matrix E T It is then used as the query, key, and value in the subsequent residual attention module (RAB).

[0043] Furthermore, the step B4 specifically includes the following steps:

[0044] Step B41: Construct a time-based embedding matrix E T The residual attention module (RAB) of ; this RAB module uses the temporal embedding matrix as query, key and value;

[0045] Step B42: Use a multi-head attention mechanism to calculate multiple attentions; the multi-head attention mechanism MH is defined as follows:

[0046]

[0047]

[0048] Among them, W o , is the parameter to be learned, n h is the number of attention heads; Concat means concatenating the matrices, and SDPA means scaled dot product attention, which is defined as follows:

[0049]

[0050] Among them, Q, K, and V represent query, key, and value, respectively, and the scaling factor Indicates the dimensions of Q, K, and V;

[0051] Step B43: Apply residual connection in RAB module, which is defined as follows:

[0052] H = LayerNorm(MH(Q,K,V)+Q)

[0053] Among them, LayerNorm represents normalization to reduce the problem of gradient disappearance;

[0054] Step B44: The entire process of the RAB module is defined as follows:

[0055] O=RAB(Q,K,V)

[0056] Step B45: The encoder block receives the time information matrix E from the timestamp embedding module T ; E T Feedback to RAB to understand the temporal relationship of each interaction in different contexts:

[0057] E' T =RAB(E T ,E T ,E T )

[0058] Step B46: After each RAB, two fully connected layers with ReLU activation are used to give the model nonlinearity; layer normalization is applied, and dropout is used for regularization:

[0059] FFN(E′ T ) = ReLU(E′) T W1+b1)W2+b2

[0060] O E =E′ T +Dropout(FFN(LayerNorm(E′ T )))

[0061] Among them, W * ∈R d×d , b * ∈R d All are trainable parameters;

[0062] Step B47: The entire process of the encoder is defined as follows:

[0063] O E =EB(E T ),O E ∈R L×d .

[0064] Furthermore, the step B5 specifically includes the following steps:

[0065] Step B51: The decoder obtains the embedding matrix E in step 2 I and the matrix O in step 4 E ; Two consecutive RAB modules are connected in series in the decoder. The low-level RAB module is used to capture the relationship between all commodities in the sequence and receives the item embedding matrix E I As query, key and value:

[0066]

[0067] Step B52: The high-level RAB module receives the output query from the low-level RAB module and the matrix O output by the encoder in step B4. E As key and value; the high-level RAB module further integrates and processes the commodity information and time information from the lower level; at this level, K=O E , V=O E , so the combination of project information and time information is E PT :

[0068]

[0069] Step B53: Enhance performance using the same method as step B46:

[0070]

[0071] Step B54: The entire process of the decoder is defined as follows:

[0072]

[0073] Step B55: Stack Q encoder blocks and P decoders respectively to learn more complex interactive transformations; each decoder takes the output of the last encoder block as input; the qth block in the encoder and the pth block in the decoder are defined as:

[0074]

[0075]

[0076] in, q∈[0,Q], p∈[0,P].

[0077] Furthermore, the step B6 specifically includes the following steps:

[0078] Step B61: Generate the user's interest vector O through the final output of the decoder D ;These interest vectors integrate users’ historical interactions and time patterns, reflecting users’ preferences for various products;

[0079]

[0080] in, Yes D The tth row of M I ∈R L×d Represents the embedding matrix of the project, represents the embedding of the i-th item; ri,t It is the preference score of item i as the next item given the first t items. The higher the preference score, the greater the possibility that item i will be interacted.

[0081] Step B62: Define o=(o1, o2, ... o L ) is the expected output of the model, where o i is defined as follows:

[0082]

[0083] in, <pad>is a filler item, and the user interaction sequence is defined as Each user’s interaction sequence is converted into a set of fixed-length sequences S = (s1, s2, …, s L );

[0084] Step B63: Generate negative samples using negative sampling; for each o i , randomly select an item o' i , As negative samples;

[0085] Step B64: Use the cross entropy loss function to calculate the prediction loss of the model; compare the difference between the model's predicted score and the actual user feedback, and adjust the model parameters accordingly; this process is implemented through back propagation and stochastic gradient descent to reduce the prediction error and improve the accuracy and generalization ability of the model. The specific implementation formula is as follows:

[0086]

[0087] in,

[0088] The present invention also provides a self-attention sequence recommendation system, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0089] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a method and system for sequential recommendation based on self-attention, which introduces a text-based pre-training model and a time coding mechanism, and uses the text pre-training model to more deeply understand the complex relationship between users and commodities; at the same time, combined with the time coding mechanism, including the time-aware self-attention mechanism, it can effectively capture the changes in user interests over time, thereby providing more dynamic and timely recommendations. This method combines text pre-training models and time coding, which not only enhances the model's ability to understand user behavior, but also improves the accuracy and relevance of recommendations, so that the sequential recommendation system can more comprehensively understand the user's behavior patterns and interest evolution, thereby achieving significant progress in providing personalized recommendations. The application of this method provides new perspectives and possibilities for the development of recommendation systems, which is of great significance for improving user experience and recommendation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 is a flow chart of a method implementation of an embodiment of the present invention;

[0091] Figure 2 is a flow chart of constructing a commodity text dictionary in an embodiment of the present invention;

[0092] Figure 3 It is an architecture diagram of the deep learning network model in an embodiment of the present invention. DETAILED DESCRIPTION

[0093] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0094] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0095] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0096] like Figure 1 As shown, this embodiment provides a self-attention sequence recommendation method, including the following steps:

[0097] Step A: Obtain product text information (including name, brand, category) and user behavior data (including user ID, product ID, interaction time) to build a training set;

[0098] Step B: construct a deep learning network model G, wherein the deep learning network model G uses the product text information to calculate the product embedding, and more comprehensively understands the user's behavior pattern and interest evolution by combining the text pre-training model and the time coding based on the self-attention mechanism; train the deep learning network model G using the training set to obtain a trained deep learning network model;

[0099] Step C: Input the product and user data into the trained deep learning network model G, and output the current user's rating ranking for the target product.

[0100] Figure 2 1 is a flowchart of constructing a commodity text dictionary in this embodiment. In this embodiment, the specific implementation steps of step A are as follows.

[0101] Step A11: Build a text dictionary for product descriptions and create a text description for each product, including all relevant text information about the product, including name, brand, and category, to ensure that the model has enough information to generate accurate product embeddings.

[0102] Step A12: Preprocess the user interaction sequence: construct the user interaction sequence based on the user's historical interaction information in Represents the text dictionary content corresponding to the i-th item interacted by user u, Indicates the time point when user u interacts with the i-th item.

[0103] Step A13: Convert each user's interaction sequence into a set of fixed-length item sequences S = (s1, s2, ..., s L ), where s i That is, the user's interactive information Where L represents the maximum sequence length. If the user's interaction sequence length is greater than L, we use the most recent L interactions. If the user's interaction sequence is less than L, we pad the items on the left of the item sequence until the length is L. For timestamp sequences, we use the first timestamp to fill.

[0104] In this embodiment, the specific implementation steps of step B are as follows.

[0105] Step B1: Divide the time interval into different intervals and assign an embedding vector to each interval; by adjusting the maximum relative time interval r max The time interval is adjusted by using the scaling factor φ, and all relative time intervals are clipped and scaled to the range [0, k], thereby obtaining the scaled relative time interval sequence T s ; Create a trainable interval matrix The embedding used for subsequent calculation of timestamps.

[0106] Step B2: Use the pre-trained BERT model to extract features of the product description text and convert the features into the initial embedding representation; during the model training process, the embedding representation remains unchanged; apply the self-attention layer and the fully connected layer to extract information to form the embedding matrix E for each product I ; Furthermore, in order to capture the position information of the sequence, a learnable position embedding matrix P is introduced into the product embedding matrix E I In order to enhance the model's ability to recognize location information, E I It will be used in the decoder of the model to further extract and fuse information to ensure that the model can understand and process complex commodity text data.

[0107] Step B3: Define a window function F win , the window function F win is an adjusted Hamming window function, the weight is calculated by relative position and a trainable parameter w; the window function F win It is used to capture the relative distance between different elements in the sequence. The weight depends on the relative position of different elements in the sequence. For each element i in the sequence, the weights of other elements j related to the element are calculated. These weights are calculated by the window function F. win Then, using these weights and the embedding of other elements, we can calculate the weighted timestamp embedding. Apply the above calculation process to all timestamps and convert the timestamp sequence T s Convert to a time embedding matrix E that preserves continuous dependencies T .

[0108] Step B4: Apply the residual attention module (RAB) as the encoder. The RAB module uses the temporal embedding matrix E T Perform self-attention operation, where E T It serves as a query, key, and value at the same time, so that the model can adaptively emphasize or suppress information at a specific time point, thereby capturing temporal dependencies and contextual relevance. After the self-attention operation, the RAB module introduces nonlinearity through two fully connected layers, which enhance the expressiveness of the model through the ReLU activation function. In order to improve the generalization ability of the model and prevent overfitting, the RAB module also applies layer normalization and dropout technology. Layer normalization standardizes the outputs of different neurons to stabilize the learning process and make training more efficient. Dropout improves the robustness and generalization ability of the model by temporarily discarding some connections in the neural network at random.

[0109] Step B5: The decoder processes and integrates temporal and sequence data by connecting two residual attention modules (RAB) in series, where the lower-level RAB module is used to capture the relationship between all items in the sequence and receives the embedding matrix E I As query, key and value, to capture the direct commodity relationship within the sequence; the higher-level RAB module uses the output of the lower-level RAB module as query and the output of the encoder as key and value input to further process and refine the information; after each RAB module, the model introduces nonlinearity by adding a feedforward neural network (FFN) and uses layer normalization and dropout techniques for regularization.

[0110] Step B6: Use the decoder to obtain the user interest vector r i Perform a dot product operation on the embedding of the potential item to obtain the interest score r i,t ; The cross entropy function is used to calculate the prediction loss to measure the difference between the predicted probability distribution and the actual label; then, according to the target loss function, the parameter gradients in the deep learning network model are calculated by the back propagation method, and the stochastic gradient descent (SGD) method is used to update these parameters, so as to gradually approach the minimum value of the loss function to make the model prediction more accurate.

[0111] Step B7: When the loss value generated by the deep learning network model is less than the set threshold or reaches the maximum number of iterations, the training of the deep learning network model G is terminated.

[0112] Figure 3 is the architecture diagram of the deep learning network model in this embodiment. Figure 3 As shown in the figure, after the training set generates an interaction sequence containing text information and interaction time information in step A, the text information in the interaction sequence is processed by the text embedding module in step B2 to obtain the product embedding matrix. The mutual time information is processed by steps B1 and B3 to obtain the timestamp embedding matrix. Then, the timestamp embedding is passed through the encoder module in step B4 to obtain the time information, and the product embedding and the output of the encoder part are sent as the input of the decoder in step B5 to obtain the final interest, and then the prediction layer in step B6 is passed to obtain the final output, and reverse regression is performed for training.

[0113] In this embodiment, step B1 specifically includes the following steps:

[0114] Step B11: Divide the time interval: First, divide the time interval evenly into different intervals; the purpose of this step is to convert the continuous time information into a discrete form to make it easier for the model to process.

[0115] Step B12: Assigning temporal embedding vectors: Assign a unique embedding vector to each time interval; these embedding vectors represent the characteristics of each time interval, enabling the model to recognize and utilize temporal information.

[0116] Step B13: Convert the timestamps into a time embedding matrix that maintains continuous correlation; Since we only care about the relative relationship between timestamps, we first convert the timestamp sequence into a relative time interval sequence, T r =(0,t2-t1,t3-t1,…,t L -t1).

[0117] Step B14: Adjust and scale the time interval: Use the maximum relative time interval r max and scaling factor φ to adjust the time interval; Represents the maximum relative time interval, where R u is the set of relative time intervals of all users, and the scale factor φ is a hyperparameter that scales the maximum relative time interval to a reasonable range.

[0118] Step B15: Scale all relative time intervals to [0, k] to obtain the scaled relative time interval sequence T s , as shown below:

[0119]

[0120] in, represents the time point of user s's i-th interaction, k is the number of time intervals; through the above calculation, the original discrete time can be mapped to one of the k time intervals.

[0121] Step B16: Create a trainable embedding matrix Used to represent time intervals, where d is the dimension size.

[0122] Step B17: k embeddings correspond to the k time intervals above; after the timestamp is mapped to the i-th of the k intervals through the above calculation, its timestamp embedding is mapped to the matrix M T The corresponding e i .

[0123] In this embodiment, step B2 specifically includes the following steps:

[0124] Step B21: Use the pre-trained BERT model to convert the content in the product description text dictionary into the initial embedding of each product; generate the initial embedding matrix It remains unchanged during model training and serves as the basis for subsequent steps, where N represents the number of products.

[0125] Step B22: Get the initial embedding matrix E BERT After that, the embedding matrix is ​​further processed using the self-attention layer and the fully connected layer to obtain a trainable embedding matrix The self-attention mechanism helps the model identify and emphasize the most important parts of the text, thereby better understanding the relationship between products and their key attributes.

[0126] Step B23: Construct the behavior sequence of each user and add the learnable position embedding matrix P to the embedding matrix of the product sequence:

[0127]

[0128] This learnable position embedding is different from the fixed position embedding in Transformer, which provides more flexible position information and better performance; these steps together enhance the model's understanding of product characteristics, so that the final product embedding can more comprehensively and accurately represent the characteristics of each product.

[0129] In this embodiment, in step B3, the time embedding matrix that maintains continuous dependencies is calculated using weights Each row is calculated as follows:

[0130]

[0131] Among them, e j From the matrix M I , weight The calculation using the window function is as follows:

[0132]

[0133]

[0134] Where w is the size of the window function,

[0135] The generated time embedding matrix E T It is then used as the query, key, and value in the subsequent residual attention module (RAB).

[0136] In this embodiment, step B4 specifically includes the following steps:

[0137] Step B41: Construct a time-based embedding matrix E T The residual attention module (RAB) is based on the temporal embedding matrix E T The design of RAB allows the model to distinguish useful temporal information in different contexts; this RAB module uses the temporal embedding matrix as query, key, and value.

[0138] Step B42: In order to enable the model to jointly focus on information from different representation subspaces at different positions, a multi-head attention mechanism (MH) is used to calculate multiple attentions; the multi-head attention mechanism MH is defined as follows:

[0139]

[0140]

[0141] Among them, W o , is the parameter to be learned, n h is the number of attention heads; Concat means concatenating the matrices, and SDPA means scaled dot product attention, which is defined as follows:

[0142]

[0143] Among them, Q, K, and V represent query, key, and value, respectively, and the scaling factor Represents the dimensions of Q, K, and V. The scaling factor is introduced to avoid the gradient being too small, especially when the dimensions are large.

[0144] Step B43: Residual connections are applied in the RAB module, which not only solves the degradation problem when the neural network is deepened, but also helps to maintain the original information of the project when we use the decoder to introduce the temporal information of the encoder; this mechanism is defined as follows:

[0145] H = LayerNorm(MH(Q,K,V)+Q)

[0146] Among them, LayerNorm represents normalization to reduce the problem of gradient disappearance.

[0147] Step B44: For simplicity, the entire process of the remaining RAB modules is defined as follows:

[0148] O=RAB(Q,K,V)

[0149] Step B45: The encoder block receives the time information matrix E from the timestamp embedding module T ; E T Feedback to RAB to understand the temporal relationship of each interaction in different contexts:

[0150] E' T =RAB(E T ,E T ,E T )

[0151] Step B46: After each RAB, two fully connected layers with ReLU activation are used to give the model nonlinearity; to stabilize and accelerate neural network training, layer normalization is applied and dropout is used for regularization:

[0152] FFN(E′ T ) = ReLU(E′) T W1+b1)W2+b2

[0153] O E =E′ T +Dropout(FFN(LayerNorm(E′ T )))

[0154] Among them, W * ∈R d×d , b * ∈R d are all trainable parameters.

[0155] Step B47: For simplicity, the entire process of the encoder is defined as follows:

[0156] O E =EB(E T ),O E ∈R L×d .

[0157] In this embodiment, step B5 specifically includes the following steps:

[0158] Step B51: The decoder obtains the embedding matrix E in step 2 I and the matrix O in step 4 E ; Two consecutive RAB modules are used in series in the decoder. The lower-level RAB module is used to capture the relationship between all commodities in the sequence and receive the item embedding matrix E I As query, key and value:

[0159]

[0160] Step B52: The higher-level RAB module receives the output query from the lower-level RAB module and the matrix O output by the encoder in step B4. E As key and value; the role of this layer is to further integrate and process the commodity information and time information from the lower layer to provide richer and deeper analysis; at this layer, the input Q, K, and V of RAB are different. K=O E , V=O E , so the combination of project information and time information is E PT :

[0161]

[0162] Step B53: Enhance performance using the same method as step B46:

[0163]

[0164] Step B54: For simplicity, the entire process of the decoder is defined as follows:

[0165]

[0166] Step B55: Stack Q encoder blocks and P decoders respectively to learn more complex interactive transformations; each decoder takes the output of the last encoder block as input; the qth block in the encoder and the pth block in the decoder are defined as:

[0167]

[0168]

[0169] in, q∈[0,Q], p∈[0,P].

[0170] In this embodiment, step B6 specifically includes the following steps:

[0171] Step B61: Generate the user's interest vector O through the final output of the decoder D ;These interest vectors integrate users’ historical interactions and time patterns, reflecting users’ preferences for various products;

[0172]

[0173] in, Yes D The tth row of M I ∈R L×d Represents the embedding matrix of the project, represents the embedding of the i-th item; r i,t It is the preference score of item i as the next item given the first t items. The higher the preference score, the greater the possibility that item i will be interacted.

[0174] Step B62: Define o=(o1, o2, ... o L ) is the expected output of the model, where o i is defined as follows:

[0175]

[0176] in, <pad>is a filler item, and the user interaction sequence is defined as Each user’s interaction sequence is converted into a set of fixed-length sequences S = (s1, s2, …, s L ).

[0177] Step B63: In sequential recommendation, a ranked list of items needs to be provided based on the user's historical interaction sequence; however, since the user's historical interaction sequence is implicit data, the preference score r cannot be directly optimized. i,t , so negative sampling is used to generate negative samples; for each o i , randomly select an item o' i , as negative samples.

[0178] Step B64: Use the cross entropy loss function to calculate the prediction loss of the model; compare the difference between the model's predicted score and the actual user feedback, and adjust the model parameters accordingly; this process is implemented through optimization techniques such as back propagation and stochastic gradient descent, the purpose of which is to reduce the prediction error and improve the accuracy and generalization ability of the model. The specific implementation formula is as follows:

[0179]

[0180] in,

[0181] This embodiment also provides a self-attention sequence recommendation system, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0182] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0183] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0184] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0185] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0186] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.< / pad> < / pad>

Claims

1. A self-attention sequence recommendation method, characterized in that: The following steps are involved: Step A: Obtain product text information and user behavior data to build a training set; Step B: construct a deep learning network model G, wherein the deep learning network model G uses the product text information to calculate the product embedding, and more comprehensively understands the user's behavior pattern and interest evolution by combining the text pre-training model and the time coding based on the self-attention mechanism; train the deep learning network model G using the training set to obtain a trained deep learning network model; Step C: Input the product and user data into the trained deep learning network model G, and output the current user's rating ranking for the target product.

2. The self-attention sequence recommendation method according to claim 1, characterized in that: The step A specifically comprises the following steps: Step A11: Build a text dictionary of product descriptions and create a text description for each product, including all relevant text information of the product, to ensure that the model has enough information to generate accurate product embeddings; Step A12: Preprocess the user interaction sequence: construct the user interaction sequence based on the user's historical interaction information in Represents the text dictionary content corresponding to the i-th item interacted by user u, represents the time point of the i-th item interacted by user u; Step A13: Convert each user's interaction sequence into a set of fixed-length item sequences S = (s1, s2, ..., s L ), where s i That is, the user's interactive information Where L represents the maximum sequence length. If the user's interaction sequence length is greater than L, we use the most recent L interactions. If the user's interaction sequence is less than L, we pad the items on the left of the item sequence until the length is L. For timestamp sequences, we use the first timestamp to fill.

3. The self-attention sequence recommendation method according to claim 1, characterized in that: The step B specifically comprises the following steps: Step B1: Divide the time interval into different intervals and assign an embedding vector to each interval; by adjusting the maximum relative time interval r max The time interval is adjusted by using the scaling factor φ, and all relative time intervals are clipped and scaled to the range [0, k], thereby obtaining the scaled relative time interval sequence T s ; Create a trainable interval matrix Embedding used for subsequent calculation of timestamp; Step B2: Use the pre-trained BERT model to extract features of the product description text and convert the features into the initial embedding representation; during the model training process, the embedding representation remains unchanged; apply the self-attention layer and the fully connected layer to extract information to form the embedding matrix E for each product I ; In order to capture the position information of the sequence, a learnable position embedding matrix P is introduced into the product embedding matrix E I In order to enhance the model's ability to recognize location information, E I The decoder will be used in the model to further extract and fuse information to ensure that the model can understand and process complex commodity text data; Step B3: Define a window function F win , the window function F win is an adjusted Hamming window function, the weight is calculated by relative position and a trainable parameter w; the window function F win It is used to capture the relative distance between different elements in the sequence. The weight depends on the relative position of different elements in the sequence. For each element i in the sequence, the weights of other elements j related to the element are calculated. These weights are calculated by the window function F. win Then, using these weights and the embedding of other elements, we calculate the weighted timestamp embedding, apply the above calculation process to all timestamps, and convert the timestamp sequence T s Convert to a time embedding matrix E that preserves continuous dependencies T ; Step B4: Apply the residual attention module (RAB) as the encoder. The RAB module uses the temporal embedding matrix E T Perform self-attention operation, where E T At the same time, it serves as a query, key, and value, so that the model can adaptively emphasize or suppress information at a specific time point, thereby capturing temporal dependencies and contextual relevance. After the self-attention operation, the RAB module introduces nonlinearity through two fully connected layers, which enhance the expressiveness of the model through the ReLU activation function. In order to improve the generalization ability of the model and prevent overfitting, the RAB module also applies layer normalization and dropout techniques. Layer normalization standardizes the outputs of different neurons to stabilize the learning process and make training more efficient. Dropout improves the robustness and generalization ability of the model by temporarily discarding some connections in the neural network at random. Step B5: The decoder processes and integrates temporal and sequence data by connecting two residual attention modules (RAB) in series, where the low-level RAB module is used to capture the relationship between all items in the sequence and receives the embedding matrix E I As query, key and value, to capture the direct commodity relationship within the sequence; the high-level RAB module uses the output of the low-level RAB module as query and the output of the encoder as key and value input to further process and refine the information; after each RAB module, the model introduces nonlinearity by adding a feedforward neural network (FFN) and uses layer normalization and dropout techniques for regularization; Step B6: Use the decoder to obtain the user interest vector r i Perform a dot product operation on the embedding of the potential item to obtain the interest score r i,t ; The cross entropy function is used to calculate the prediction loss to measure the difference between the predicted probability distribution and the actual label; then, according to the target loss function, the parameter gradients in the deep learning network model are calculated by the back propagation method, and these parameters are updated using the stochastic gradient descent method, so as to gradually approach the minimum value of the loss function to make the model prediction more accurate; Step B7: When the loss value generated by the deep learning network model is less than the set threshold or reaches the maximum number of iterations, the training of the deep learning network model G is terminated.

4. The self-attention sequence recommendation method according to claim 3, characterized in that: The step B1 specifically comprises the following steps: Step B11: Divide the time interval: First, evenly divide the time interval into different intervals to convert the continuous time information into discrete form, making it easier for the model to process; Step B12: Assigning temporal embedding vectors: Assign a unique embedding vector to each temporal interval; these embedding vectors represent the characteristics of each temporal interval, enabling the model to recognize and utilize temporal information; Step B13: Convert the timestamp into a time embedding matrix that maintains continuous correlation; convert the timestamp sequence into a relative time interval sequence, T r =(0,t2-t1,t3-t1,…,t L -t1); Step B14: Adjust and scale the time interval: Use the maximum relative time interval r max and scaling factor φ to adjust the time interval; Represents the maximum relative time interval, where R u is the set of relative time intervals of all users, and the scale factor φ is a hyperparameter that scales the maximum relative time interval to a reasonable range; Step B15: Scale all relative time intervals to [0, k] to obtain the scaled relative time interval sequence T s , as shown below: in, represents the time point of user s's i-th interaction, k is the number of time intervals; through the above calculation, the original discrete time is mapped to one of the k time intervals; Step B16: Create a trainable embedding matrix Used to represent time intervals, where d is the dimension size; Step B17: k embeddings correspond to k time intervals; after the timestamp is mapped to the i-th of the k intervals through the above calculation, its timestamp embedding is mapped to the matrix M T The corresponding e i .

5. The self-attention sequence recommendation method according to claim 4, characterized in that: The step B2 specifically comprises the following steps: Step B21: Use the pre-trained BERT model to convert the content in the product description text dictionary into the initial embedding of each product; generate the initial embedding matrix It remains unchanged during model training and serves as the basis for subsequent steps, where N represents the number of products; Step B22: Get the initial embedding matrix E BERT After that, the embedding matrix is ​​further processed using the self-attention layer and the fully connected layer to obtain a trainable embedding matrix The self-attention mechanism helps the model identify and emphasize the most important parts of the text, thereby better understanding the relationship between products and their key attributes; Step B23: Construct the behavior sequence of each user and add the learnable position embedding matrix P to the embedding matrix of the product sequence:

6. The self-attention sequence recommendation method according to claim 5, characterized in that: In step B3, the weights are used to calculate the temporal embedding matrix that maintains the continuous dependency. Each row is calculated as follows: Among them, e j From the matrix M I , weight The calculation using the window function is as follows: Where w is the size of the window function, The generated time embedding matrix E T It is then used as the query, key, and value in the subsequent residual attention module (RAB).

7. The self-attention sequence recommendation method according to claim 6, characterized in that: The step B4 specifically comprises the following steps: Step B41: Construct a time-based embedding matrix E T The residual attention module (RAB) of ; this RAB module uses the temporal embedding matrix as query, key and value; Step B42: Use a multi-head attention mechanism to calculate multiple attentions; the multi-head attention mechanism MH is defined as follows: Among them, W o , is the parameter to be learned, n h is the number of attention heads; Concat means concatenating the matrices, and SDPA means scaled dot product attention, which is defined as follows: Among them, Q, K, and V represent query, key, and value, respectively, and the scaling factor Indicates the dimensions of Q, K, and V; Step B43: Apply residual connection in RAB module, which is defined as follows: H = LayerNorm(MH(Q,K,V)+Q) Among them, LayerNorm represents normalization to reduce the problem of gradient disappearance; Step B44: The entire process of the RAB module is defined as follows: O=RAB(Q,K,V) Step B45: The encoder block receives the time information matrix E from the timestamp embedding module T ; E T Feedback to RAB to understand the temporal relationship of each interaction in different contexts: E′ T =RAB(E T ,It T ,It T ) Step B46: After each RAB, two fully connected layers with ReLU activation are used to give the model nonlinearity; layer normalization is applied, and dropout is used for regularization: FFN(E′ T )=ReLU(E′ T W1+b1)W2+b2 THEIR E =E′ T +Dropout(FFN(LayerNorm(E′ T ))) Among them, W * ∈R d×d , b * ∈R d All are trainable parameters; Step B47: The entire process of the encoder is defined as follows: The E =EB(E T ),O E ∈R L×d 。 8. The self-attention sequence recommendation method according to claim 7, characterized in that: The step B5 specifically comprises the following steps: Step B51: The decoder obtains the embedding matrix E in step 2 I and the matrix O in step 4 E ; Two consecutive RAB modules are connected in series in the decoder. The low-level RAB module is used to capture the relationship between all commodities in the sequence and receives the item embedding matrix E I As query, key and value: Step B52: The high-level RAB module receives the output query from the low-level RAB module and the matrix O output by the encoder in step B4. E As key and value; the high-level RAB module further integrates and processes the commodity information and time information from the lower level; at this level, K=O E , V=O E , so the combination of project information and time information is E PT : Step B53: Enhance performance using the same method as step B46: Step B54: The entire process of the decoder is defined as follows: Step B55: Stack Q encoder blocks and P decoders respectively to learn more complex interactive transformations; each decoder takes the output of the last encoder block as input; the qth block in the encoder and the pth block in the decoder are defined as: in, q∈[0,Q], p∈[0,P].

9. The self-attention sequence recommendation method according to claim 8, characterized in that: The step B6 specifically comprises the following steps: Step B61: Generate the user's interest vector O through the final output of the decoder D ;These interest vectors integrate users’ historical interactions and time patterns, reflecting users’ preferences for various products; in, Yes D The tth row of M I ∈R L×d Represents the embedding matrix of the project, represents the embedding of the i-th item; r i,t It is the preference score of item i as the next item given the first t items. The higher the preference score, the greater the possibility that item i will be interacted. Step B62: Define o=(o1, o2, ... o L ) is the expected output of the model, where o i is defined as follows: in, <pad>is a filler item, and the user interaction sequence is defined as Each user’s interaction sequence is converted into a set of fixed-length sequences S = (s1, s2, …, s L );< / pad> Step B63: Generate negative samples using negative sampling; for each o i , randomly select an item o' i , As negative samples; Step B64: Use the cross entropy loss function to calculate the prediction loss of the model; compare the difference between the model's predicted score and the actual user feedback, and adjust the model parameters accordingly; this process is implemented through back propagation and stochastic gradient descent to reduce the prediction error and improve the accuracy and generalization ability of the model. The specific implementation formula is as follows: in, 10. A self-attention sequence recommendation system, characterized in that: The method comprises a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the method steps as described in any one of claims 1 to 9 can be implemented.

Images