A BERT-based sequence recommendation method and system

Abstract

This invention discloses a BERT-based sequence recommendation method and system. The method comprises: combining adversarial learning and contrastive learning based on temporal attention on the BERT module to obtain a DACRec framework; constructing the DACRec framework; obtaining a dataset and using the dataset to train the DACRec framework to obtain a trained DACRec framework; and using the trained DACRec framework to predict forwarding results and obtain recommendation results. The system includes a framework construction module, a training module, and a prediction module. By using this method, timestamp information and popularity bias can be used to improve recommendation accuracy. This invention has broad application in the field of recommendation systems.

Description

Technical Field

[0001] The present invention relates to the field of recommendation systems, and in particular to a BERT-based sequence recommendation method and system. Background Art

[0002] Recommender systems are an important information filtering technology designed to accurately describe user interests to alleviate information overload and improve user experience. In most applications, user interests are dynamic and evolve over time. Capturing the dynamics of continuous user behavior is crucial for making appropriate recommendations. Sequential recommendation tasks are a widely-attended task in recommendation systems. They predict which items a user may be interested in by analyzing historical user behavior sequences. Compared to other recommendation methods, sequential recommendation systems have stronger dynamics and personalization capabilities because they can capture the patterns of user interests changing over time.

[0003] Most traditional sequential recommenders rank items based on interaction timestamps and focus on mining sequential patterns to predict the next item likely to be interacted with by a user. Markov chain methods can capture short-term item transitions in recommendations in highly sparse environments, but are not suitable for complex scenarios. Recurrent neural networks, with their "memory" capabilities for capturing user preferences, have also been used in this context, but they require large amounts of data to outperform simpler baseline models. To address the limitations of Markov chain and RNN-based models, some methods apply self-attention mechanisms to sequential recommendation problems, but these methods do not consider the influence of timestamps. Other methods employ graph neural networks to explore more complex item transition patterns in user sequences, but these methods rely heavily on sufficient interaction data and semantically rich sequences, making them difficult to address sparsity and other issues in recommendation. Deep neural networks suffer from the long-tail problem in collaborative filtering (CF)-based recommendation tasks (recommendation systems tend to recommend popular items while ignoring less popular ones, causing popular items to become increasingly popular and less popular items to become less popular). This not only affects user experience but also reduces the share of platform revenue derived from high-profit, niche products. From a data perspective, the interaction frequency of projects is unevenly distributed (typically with a long tail). The interaction data between users and projects often exhibits a long tail distribution, meaning that the top projects account for the majority of interactions. Recommendation models trained on this long-tail data not only inherit biases but also amplify them. To make matters worse, the feedback loop of the recommendation system further exacerbates this Matthew effect.

[0004] Real-world recommendation datasets often contain a large amount of sparse data in the user-item interaction matrix. This data sparsity prevents models from fully utilizing this user-item interaction information. This data sparsity also causes the representations learned by the model to be influenced by a large number of popular items, resulting in popularity bias in both the learned representations and the recommendation results. However, data sparsity is not always a disadvantage, as some items exhibit higher popularity due to their superior intrinsic quality. Blindly pursuing unbiased learning can eliminate beneficial patterns in the data, thereby reducing recommendation accuracy and user satisfaction. Therefore, how to leverage popularity bias to improve recommendation accuracy is a key issue that needs to be addressed. Summary of the Invention

[0005] In order to solve the above technical problems, the purpose of the present invention is to provide a BERT-based sequence recommendation method and system, which can improve the accuracy of recommendations by utilizing timestamp information and popularity deviation.

[0006] The first technical solution adopted by the present invention is: a BERT-based sequence recommendation method, comprising the following steps:

[0007] Based on the BERT module, adversarial learning and contrastive learning are combined based on temporal attention to obtain the DACRec framework;

[0008] Obtain a data set and use the data set to train the DACRec framework to obtain a trained DACRec framework;

[0009] The trained DACRec framework is used to predict the advancement results and obtain the recommendation results.

[0010] Furthermore, the DACRec framework includes a time series BERT module, a contrastive learning loss module, and an adversarial learning debiasing module, which specifically include:

[0011] Build a time series BERT module based on learnable time encoding and self-attention modules;

[0012] Construct a contrastive learning loss module based on data augmentation and contrastive learning modules;

[0013] Construct an adversarial learning debiasing module based on the adversarial learning module.

[0014] Through this preferred step, the DACRec framework can fully utilize timestamp information and solve the problems of poor recommendation performance and popularity deviation caused by data sparsity.

[0015] Furthermore, the step of obtaining a data set and using the data set to train the DACRec framework to obtain a trained DACRec framework specifically includes:

[0016] Time-encode the data set to obtain a time-encoded vector;

[0017] Embed the time encoding vector into the original sequence input of the time series BERT module to obtain the sequence input of the time series BERT module;

[0018] Perform data expansion on the same source sample to obtain several augmented samples;

[0019] Set different augmented sample indices in the sequence input of the time series BERT module and input them into the contrastive learning loss module to calculate the supervised contrast loss of different augmented samples;

[0020] Calculate the loss of the model before data expansion and fuse the coefficients with the supervised contrast loss to obtain the DACRec framework loss;

[0021] The sequence input and sequence output of the time series BERT module are input into the adversarial learning debiasing module for adversarial learning to obtain the probability distribution of the posterior distribution;

[0022] Backpropagation is performed based on the probability distribution of the DACRec framework loss and posterior distribution to obtain the parameter set of the DACRec framework.

[0023] Furthermore, the sequence input of the time series BERT module is expressed as follows:

[0024]

[0025]

[0026]

[0027] in, represents the sequence input, represents the interaction sequence embedding of user u, represents the interaction sequence position embedding of user u, Indicates time code, t indicates timestamp, d T represents the time dimension, ω i and θ i represents the learnable parameters, represents the sequence input of the time series BERT module, S u Represents the historical behavior sequence of user u.

[0028] Furthermore, the supervised contrast loss is expressed as follows:

[0029]

[0030]

[0031] Among them, L sup represents the supervised contrast loss, i represents the index, also called the anchor, |Φ(i)| represents the set cardinality of all positive indices in the multi-view batch different from i, and the symbol represents the inner product. Enc represents the encoder, represents the true sample, represents negative samples, and τ represents the parameter used to optimize contrastive learning.

[0032] Furthermore, the probability distribution of the posterior distribution is expressed as follows:

[0033]

[0034] Where P represents the data distribution, θ and φ represent the parameters of the DACRec framework, and q φ (z u，t |S u，[1：t] ||p(z u，t )) are set as independent normal distributions and parameterized by neural networks. represents the marginal log-likelihood, and KL represents the KL divergence.

[0035] The second technical solution adopted by the present invention is: a BERT-based sequence recommendation system, comprising:

[0036] A framework building module that combines adversarial learning and contrastive learning based on temporal attention on the basis of the BERT module to obtain the DACRec framework;

[0037] The training module is used to obtain a data set and use the data set to train the DACRec framework to obtain a trained DACRec framework;

[0038] The prediction module uses the trained DACRec framework to predict the advancement results and obtain the recommendation results.

[0039] The beneficial effects of the method and system of the present invention are as follows: the present invention proposes a DACRec framework based on the BERT model, which adds the timestamp information contained in the recommendation sequence to the sequence embedding to fully utilize the timestamp information; uses contrastive learning for data expansion processing to solve the problem of poor recommendation performance caused by sequence sparsity; uses adversarial learning methods to learn enhanced embedding representations that are aware of popularity bias, and ultimately achieves the use of timestamp information and popularity bias to improve the accuracy of sequence recommendations. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1This is a flowchart of the steps of a BERT-based sequence recommendation method of the present invention;

[0041] Figure 2 This is a structural block diagram of a BERT-based sequence recommendation system of the present invention;

[0042] Figure 3 This is a DACRec framework diagram of a BERT-based sequence recommendation method of the present invention;

[0043] Figure 4 This is a training prediction flowchart of a BERT-based sequence recommendation method of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.

[0045] like Figure 1 As shown, the present invention provides a BERT-based sequence recommendation method, which includes the following steps:

[0046] S1. Based on the BERT module, adversarial learning and contrastive learning are combined based on temporal attention to obtain the DACRec framework.

[0047] S1.1. Construct a time series BERT module based on learnable temporal encoding and self-attention modules.

[0048] Specifically, refer to Figure 3 , taking into account long-term or short-term preferences by tandemly combining a learnable temporal encoding method and a self-attention scheme, because temporal characteristics are implicitly reflected by training time parameters without explicitly designing UNIX timestamp features. The timestamp information encoding is connected to the sequence input representation of the BERT module to obtain temporal self-attention, so that the weight becomes the sum of self-attention and temporal attention.

[0049] S1.2. Construct a contrastive learning loss module based on data augmentation and contrastive learning modules;

[0050] Specifically, for supervised learning, due to the existence of labels, it is known that there are multiple samples belonging to the same class, and unsupervised contrast loss cannot handle it. However, generalizing the positive number to any number will lead to a choice between multiple possible functions; in a batch, let i∈i≡{1...2N} be the index of any augmented sample, and let z pis the index of another augmented sample from the same source sample, and contrastive learning is achieved by calculating the supervised contrast loss of different augmented samples.

[0051] S1.3. Construct an adversarial learning debiasing module based on the adversarial learning module.

[0052] Specifically, the adversarial learning model is introduced into the BERT module, and the model is trained under the adversarial BERT module. The sequence input of the time series BERT module and the sequence output of the time series BERT module are used as the true input and predicted input of the adversarial learning, respectively.

[0053] S2. Obtain a data set and use the data set to train the DACRec framework to obtain a trained DACRec framework;

[0054] Specifically, recommendation datasets were obtained from MovieLens 1M and Steam respectively; the MovieLens dataset contains rating data of multiple users on multiple movies, as well as movie metadata and user attribute information; the Steam dataset contains rating data of multiple users on the Steam platform for multiple games, ranging from 3A masterpieces to small independent games. These datasets are often used as test datasets for recommendation systems and machine learning algorithms. In particular, many well-known papers in the field of recommendation systems are based on this dataset. For each user, we randomly selected 100 negative items that were not in their interaction history and ranked them together with the real positive items. We also constructed popular negative sampling and all negative sampling based on popularity sorting. Popular negative sampling is to sort the items by popularity and select the items among them as negative items, which acts as a kind of "hard negative sampling."

[0055] We then preprocessed the recommendation dataset, setting the maximum sequence length for ML-1M and Steam to 200 and 15, respectively, and treating each review or rating as an implicit positive interaction. To ensure dataset quality, we removed items with an interaction count of 5 or less and users with a history length of 5 or less.

[0056] The preprocessed dataset is divided so that for each user, the last item is used as the test item, the second most recent item is used as the validation item, and the remaining items are used as the training set.

[0057] S2.1. Time-code the data set to obtain a time-coded vector.

[0058] Specifically, the time code of timestamp t is defined as follows:

[0059]

[0060] in, Indicates time code, t indicates timestamp, d T represents the time dimension, ω i and θ i represents the learnable parameters, S u Represents the historical behavior sequence of user u.

[0061] S2.2. Embed the time encoding vector into the original sequence input of the time series BERT module to obtain the sequence input of the time series BERT module;

[0062] Specifically, the time encoding vector Connected to the original sequence input, the weight of the sequence input becomes the sum of self-attention and temporal attention, and its sequence input also has the temporal self-attention attribute. The sequence input of the time series BERT module is expressed as follows:

[0063]

[0064]

[0065] in, represents the sequence input, represents the interaction sequence embedding of user u, represents the interaction sequence position embedding of user u, Indicates time code, Represents the sequence input to the time series BERT module.

[0066] S2.3. Perform data expansion on the same source sample to obtain several augmented samples;

[0067] Specifically, in each minibatch, two data augmentation methods are randomly selected from the following four data augmentation methods. After data augmentation, the two augmented sentences from the same sentence are considered to be positive samples, and all other augmented samples from the same minibatch are considered to be negative samples of these two positive samples.

[0068] Delete words: Randomly delete some words in the sentence and replace them with [DEL].

[0069] Delete interval: select and delete from the interval and use [DEL] to replace;

[0070] Rearrange the order: randomly sample several pairs of words from the sentence and then replace their positions;

[0071] Synonym replacement: Randomly select some words from the sentence and replace them with synonyms of these words.

[0072] S2.4. Set different augmented sample indices in the sequence input of the time series BERT module and input them into the contrastive learning loss module to calculate the supervised contrast loss of different augmented samples;

[0073] Specifically, in a batch, let i∈i≡{1...2N} be the index of any augmented sample, and let z p For the index of another augmented sample from the same source sample, the supervised contrast loss of different augmented samples is calculated, which is expressed as follows:

[0074]

[0075]

[0076] Among them, L sup represents the supervised contrast loss, i represents the index, also called the anchor, |Φ(i)| represents the set cardinality of all positive indices in the multi-view batch different from i, and the symbol represents the inner product. Enc represents the encoder, represents the true sample, represents negative samples, and τ represents the parameter used to optimize contrastive learning.

[0077] S2.5. Calculate the loss of the model before data augmentation and fuse the coefficients with the supervised contrast loss to obtain the DACRec framework loss.

[0078] Specifically, the calculation expression of the DACRec framework loss is as follows:

[0079] L p =CrossEntropyLoss(logits,labels)

[0080] L=β*L sup +(1-β)*L p

[0081] Where L represents the DACRec framework loss, L sup represents the supervised contrast loss, L p represents the loss of the model before data augmentation, β represents an adjustable parameter, β∈{0, 1}.

[0082] S2.6. Input the sequence input of the time series BERT module and the sequence output of the time series BERT module into the adversarial learning debiasing module to perform adversarial learning and obtain the probability distribution of the posterior distribution;

[0083] Specifically, the sequence input and sequence output of the time series BERT module are input into the adversarial learning model, and the adversarial learning model is trained under the adversarial BERT framework. The adversarial learning BERT uses a complex inference model to generate various probability distributions close to the true posterior distribution, which is expressed as follows:

[0084]

[0085] Where P represents the data distribution, θ and φ represent the parameters of the DACRec framework, and q φ (z u，t |S u，[1：t] ||p(z u，t )) are set as independent normal distributions and parameterized by neural networks. represents the marginal log-likelihood, and KL represents the KL divergence.

[0086] Sequence BERT trains maximum likelihood on variational evidence lower bounds to estimate intractable marginal log-likelihoods Bundle Assume that is an independent normal distribution and parameterize it with a neural network. In this case, It is easy to calculate, because p(z u，t )and is normally distributed, and kL can be calculated explicitly. But it also makes the model heavily dependent on z u，t , which limits the quality of the inference model, so The higher the quality, the more flexible the representation ability is, and the closer it is to the true posterior.

[0087] S2.7. Backpropagate based on the probability distribution of the DACRec framework loss and posterior distribution to obtain the parameter set of the DACRec framework.

[0088] Specifically, back propagation is performed based on the DACRec framework loss and the probability distribution of the posterior distribution, and the model parameters of the DACRec framework are continuously adjusted to reduce the DACRec framework loss and make the probability distribution of the posterior distribution close to the true posterior, thus obtaining the parameter set of the DACRec framework.

[0089] S2.8. Verify the parameter set of the DACRec framework using the validation set to obtain the final parameter set of the DACRec framework;

[0090] Specifically, refer to Figure 4 The verification step is consistent with the S3 training step. Its purpose is to verify that the parameter set learned by the DACRec framework training can also have excellent recommendation accuracy under the verification set, and to re-adjust the parameters that can be further optimized and adjusted to obtain the final parameter set of the DACRec framework.

[0091] S2.9. Test the final parameter set of the DACRec framework using the test set to obtain the trained DACRec framework.

[0092] Specifically, the testing step is consistent with the S3 training step, and its purpose is to test whether the DACRec framework containing the final parameter set can maintain excellent recommendation accuracy in actual tests.

[0093] S3. Use the trained DACRec framework to predict the advancement results and obtain the recommendation results.

[0094] Specifically, the trained DACRec framework is combined with the user's historical recommendation data preferences to predict the content that the user may like and obtain recommendation results.

[0095] like Figure 2 As shown in FIG, a BERT-based sequence recommendation system includes:

[0096] A framework building module that combines adversarial learning and contrastive learning based on temporal attention on the basis of the BERT module to obtain the DACRec framework;

[0097] The training module is used to obtain a data set and use the data set to train the DACRec framework to obtain a trained DACRec framework;

[0098] The prediction module uses the trained DACRec framework to predict the advancement results and obtain the recommendation results.

[0099] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0100] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A BERT-based sequence recommendation method, characterized in that: The following steps are involved: Based on the BERT module, adversarial learning and contrastive learning are combined based on temporal attention to obtain the DACRec framework; Obtain a data set and use the data set to train the DACRec framework to obtain a trained DACRec framework; Use the trained DACRec framework to predict the advancement results and obtain the recommendation results; The DACRec framework includes a time series BERT module, a contrastive learning loss module, and an adversarial learning debiasing module, which specifically includes: Build a time series BERT module based on learnable time encoding and self-attention modules; Construct a contrastive learning loss module based on data augmentation and contrastive learning modules; Construct an adversarial learning debiasing module based on the adversarial learning module; The step of obtaining a data set and using the data set to train the DACRec framework to obtain a trained DACRec framework specifically includes: Time-encode the data set to obtain a time-encoded vector; Embed the time encoding vector into the original sequence input of the time series BERT module to obtain the sequence input of the time series BERT module; Perform data expansion on the same source sample to obtain several augmented samples; Set different augmented sample indices in the sequence input of the time series BERT module and input them into the contrastive learning loss module to calculate the supervised contrast loss of different augmented samples; Calculate the loss of the model before data expansion and fuse the coefficients with the supervised contrast loss to obtain the DACRec framework loss; The sequence input and sequence output of the time series BERT module are input into the adversarial learning debiasing module for adversarial learning to obtain the probability distribution of the posterior distribution; Backpropagation is performed based on the probability distribution of the DACRec framework loss and posterior distribution to obtain the parameter set of the DACRec framework.

2. The BERT-based sequence recommendation method according to claim 1, characterized in that: The sequence input of the time series BERT module is expressed as follows: in, represents the sequence input, represents the interaction sequence embedding of user u, represents the interaction sequence position embedding of user u, Indicates time code, t indicates timestamp, d T represents the time dimension, ω i and θ i represents the learnable parameters, represents the sequence input of the time series BERT module, S u Represents the historical behavior sequence of user u.

3. The BERT-based sequence recommendation method according to claim 2, characterized in that: The supervised contrast loss is expressed as follows: Among them, L sup represents the supervised contrast loss, i represents the index, also called the anchor, |Φ(i)| represents the set cardinality of all positive indices in the multi-view batch different from i, and the symbol represents the inner product. Enc represents the encoder, represents the true sample, and τ represents the parameter used to optimize contrastive learning.

4. The BERT-based sequence recommendation method according to claim 3, characterized in that: The probability distribution of the posterior distribution is expressed as follows: Where P represents the data distribution, θ and φ represent the parameters of the DACRec framework, and q φ (z u,t |S u,[1:t] ||p(z u,t )) are set as independent normal distributions and parameterized by neural networks. represents the marginal log-likelihood, and KL represents the KL divergence.

5. A BERT-based sequence recommendation system, characterized in that Used to perform the BERT-based sequence recommendation method according to claim 1, comprising: A framework building module that combines adversarial learning and contrastive learning based on temporal attention on the basis of the BERT module to obtain the DACRec framework; The training module is used to obtain a data set and use the data set to train the DACRec framework to obtain a trained DACRec framework; The prediction module uses the trained DACRec framework to predict the advancement results and obtain the recommendation results.

Images