Project embedding general pre-training method based on sequence modeling

Abstract

The invention discloses a project embedding general pre-training method based on sequence modeling, which is realized through a pre-training model, and the pre-training model comprises a preprocessing layer, an embedding layer and a mixed perception attention layer. According to the invention, a highly universal pre-training model is innovatively designed and realized, so that a project embedding universal pre-training method based on sequence modeling is formed, and a project interaction sequence can be constructed by using interaction data; and multi-source information such as project titles, project descriptions, sequence positions and interaction time is integrated into a sequence modeling framework to generate semantic enhanced project embedding, so that the problem of data sparseness in a downstream recommendation system is effectively relieved, and the performance upper limit of a recommendation model is improved.

Description

[Technical field]

[0001] The present invention relates to the technical field of pre-training methods, in particular to the technical field of a general pre-training method for item embedding based on sequence modeling. [Background technology]

[0002] Sequential recommendation models can analyze users' historical behavior sequences, identify their access patterns, and provide personalized next-hop recommendations. Under the architecture of sequential recommendation systems, the core of data modeling is the deep integration of time dimension and sequence order. Time-aware recommendations focus on the precise modeling of absolute timestamps, aiming to capture the dynamic evolution of user preferences and item popularity over time. Traditional sequential recommendations usually mine user behavior patterns from item sequences sorted by interaction time to predict the next interaction item, such as SASRec (W.-C.Kang and J.McAuley.Self-Attentive Sequential Recommendation.in:Proceeding of the 18th IEEE International Conference on Data Mining(ICDM),2018,pp.197-206.). Recent studies, such as TiSASRec (J.Li,Y.Wang and J.McAuley.Time Interval Aware Self-Attention for Sequential Recommendation.in:Proceedings ofthe 13th International Conference on WebSearch and Data Mining (ICDM), 2020, pp. 322–330.), has demonstrated the importance of considering time intervals (which not only preserves the integrity of sequence information, but also incorporates the subtle effects of time intervals to enrich the depth and breadth of sequence recommendations).

[0003] The rise of pre-training technology in the field of recommendation systems has opened up new ways to improve recommendation accuracy and generalization ability; the mainstream pre-training paradigms can be summarized into two categories; one of the pre-training paradigms first uses a pre-trained model to learn the latent representation of item attributes, and then combines it with a sequential recommendation model for user preference modeling, such as BERT4Rec (F.Sun, J.Liu, J.Wu, et al. BERT4Rec: Sequential Recommendation with BidirectionalEncoder Representations from Transformer.in: Proceedings of the 28th ACMInternational Conference on Information and Knowledge Management, 2019, pp.1441–1450.); this strategy strengthens the personalized features of items through item descriptions and promotes deep personalization of recommendation services; the other pre-training paradigm directly learns item representations from user behavior sequences to model user embeddings and seamlessly integrates them into sequential recommendation models, such as PrepRec (J.Wang, P.Rathi and H.Sundaram.APre-trained Sequential Recommendation Framework: Popularity Dynamics for Zero-shot Transfer.2024.); This approach simplifies the recommendation process while ensuring efficient and accurate recommendation results.

[0004] Although existing pre-trained models perform well in zero-shot and few-search recommendation scenarios, they mainly rely on multimodal information related to items and ignore the key role of interaction data; in addition, multimodal information itself is scarce, while interaction data constitutes the backbone of the recommendation system. [Summary of the invention]

[0005] The purpose of the present invention is to solve the problems in the prior art; the present invention innovatively designs and implements a highly general pre-training model, thereby forming a general pre-training method for project embedding based on sequence modeling, which can use interaction data to construct project interaction sequences, and integrate multi-source information such as project titles, project descriptions, sequence positions and interaction times into the sequence modeling framework to generate semantically enhanced project embeddings, effectively alleviating the data sparsity problem in downstream recommendation systems and improving the performance upper limit of recommendation models.

[0006] To achieve the above object, the present invention proposes a general pre-training method for item embedding based on sequence modeling, which is implemented by a pre-training model, wherein the pre-training model includes a pre-processing layer, an embedding layer and a hybrid perceptual attention layer, and the training process includes the following steps:

[0007] Step 1, using the preprocessing layer to perform data preprocessing in sequence through item sequence extraction, time conversion and text processing, wherein the item sequence extraction is to construct an access sequence about the item based on the interaction time and each element in the access sequence is a user who interacts with the item in chronological order, the time conversion is to convert the timestamp into fine-grained time information, and the text processing is to simplify the text vocabulary;

[0008] Step 2: Using the embedding layer to convert discrete features and text features into low-dimensional dense vectors;

[0009] Step 3: Utilize the hybrid-aware attention layer to capture the complex collaborative relationship between sequence users based on multi-source information and through the self-attention mechanism in sequence modeling to obtain project embedding.

[0010] Preferably, in step 1, the item sequence extraction converts each sequence into a fixed length n and obtains an item interaction sequence S = {u1, u2, ..., u n} and the interaction time series T = {t1, t2, ..., t n}, the fine-grained time information is the year, month, day, hour, minute and second time information, and the simplified text vocabulary is to delete useless characters in the project name and description and / or perform stemming and lemmatization operations.

[0011] Preferably, in step 2, an initialized user embedding matrix E is first given u ∈R |U|×d , position embedding matrix E p ∈R n×d , year embedding matrix E y ∈R 5000×d , monthly embedding matrix E mon ∈R 24×d , daily embedding matrix E d ∈R 62×d , when embedding matrix E h ∈R 48×d , embedding matrix E min ∈R 120×d and the second embedding matrix E sec ∈R 120×d And d is used to represent the dimension of the embedding vector, and then the embedding layer is used to obtain the user embedding m∈R in the item interaction sequence through the embedding search operation n×d, position embedding and time embedding and input the project name and description into the SBERT model to obtain the project name embedding N∈R n×d and the item description embedding D∈R n×d , the position embedding and time embedding include value vector embedding and key vector embedding, the value vector embedding includes p V ∈R n×d ,y V ∈R n×d ,mon V ∈R n×d 、day V ∈R n×d 、h V ∈R n×d 、min V ∈R n×d and sec V ∈R n×d , the key vector embedding includes p K ∈R n×d ,y K ∈R n×d ,mon K ∈R n×d 、day K ∈R n×d 、h K ∈R n×d 、min K ∈R n×d and sec K ∈R n×d .

[0012] Preferably, in step 3, the hybrid perceptual attention layer implements the hybrid perceptual attention process by first performing self-attention calculation on the interaction sequence, then applying residual connection to retain the key information in the original user embedding, and finally integrating the processing result with the project text information.

[0013] Preferably, in step 3, given a project interaction sequence M=(m1, m2, ..., e n ) and use m j represents the embedding of the jth interacting user, and uses formula (2) to generate a new embedding sequence Z = (z1, z2, ..., z n );

[0014]

[0015] Where W V ∈R d×d represents the input projection matrix; embeddings representing where the user interacted; and Respectively represent fine-grained temporal embedding; a ij Represents the weight coefficient.

[0016] As a preference, formula (3) is used to calculate the a according to the Softmax function. ij ;

[0017]

[0018] In the formula, e ij Represents the attention weights calculated based on user embedding, location embedding, year embedding, month embedding, day embedding, hour embedding, minute embedding, and second embedding.

[0019] As a preference, formula (4) is used to express the e ij ;

[0020]

[0021] Where W Q ∈R d×d and W K ∈R d×d denote the query projection matrix and the key vector of the interacting user respectively; and The key vector embeddings representing fine-grained interaction time respectively; is the normalization factor.

[0022] Preferably, in step 3, z is calculated using formula (5). j Add linear transformation;

[0023] FFN(z j )=max(0,z j W1+b1)W2+b2 (5);

[0024] Where W1, W2∈R d×d Represents the projection matrix of two-layer linear transformation; b1, b2∈R d Represents the deviation term.

[0025] Preferably, in step 3, formula (6), formula (7) and formula (8) are used to calculate z j Perform layer normalization, residual connection and dropout regularization;

[0026] z j =z j +Dropout(FFN(LayerNorm(z j ))) (6);

[0027]

[0028] Q = N + D + V (8);

[0029] In the formula, m i represents the original user embedding; Avg(·) represents the average pooling operation; N represents the item name embedding; D represents the item description embedding; V represents the preliminary item embedding obtained by the self-attention layer.

[0030] As a preference, given an item k, the item interaction sequence S k ∈S, for each expected positive user u_i∈S^k, randomly select a negative user To generate a pair of training samples O = (u i ,u j ) and use binary cross entropy as the loss function, and use formula (9), formula (10) and formula (11) to optimize the pre-trained model;

[0031] R i，k =q k m i T (9);

[0032]

[0033] In the formula, m i represents the embedding of candidate user i; q k represents the embedding of the current item k; represents the learnable embedding; λ represents the regularization weight; ||.|| F represents the Frobenius norm.

[0034] It should be pointed out that in the present invention, the pre-trained model is a model for item embedding modeling, rather than a model for user preference modeling; this alleviates the data sparsity problem in traditional recommendations to a certain extent; specifically, when the number of items accessed by a user is small, due to insufficient interaction data, traditional user modeling methods may find it difficult to accurately characterize user preferences; the model proposed in the present invention focuses on item embedding, and in addition to traditional interaction information, further integrates text modal information such as item names and descriptions for item modeling, thereby overcoming the problem of data sparsity to a certain extent; in addition, the pre-trained model is a general model that can be seamlessly integrated with downstream recommendation models to provide effective initial input item embedding for recommendation tasks.

[0035] Beneficial effects of the present invention:

[0036] The present invention innovatively proposes a universal pre-training model based on interaction information and item description for modeling item embedding (the pre-training model adopts the idea of ​​sequence modeling to learn item embedding, aiming to optimize the performance of downstream recommendation tasks by generating effective item embeddings using multi-source and multi-modal information); in the modeling process, the pre-training model can extract item interaction sequences according to interaction timestamps, and comprehensively consider information such as item name, item description, sequence order and interaction time to learn item embeddings (this strategy stems from a deep insight into the current mainstream trend, while existing pre-training models tend to learn embeddings of intrinsic attributes of items to serve the prediction tasks of recommendation systems; we have changed this traditional paradigm and shifted the focus to interaction information as a more accessible data source, capturing and enhancing item embedding representations through the idea of ​​sequence modeling); experimental results show that the innovative strategy of the present invention significantly improves the performance of different downstream recommendation models on dense and sparse data sets, verifying its effectiveness and generalization ability in practical applications; in general, the beneficial effects of the present invention can be summarized as follows:

[0037] 1) Item embedding based on the idea of ​​sequence modeling: This paper creates a general pre-training method that uses the idea of ​​sequence modeling to generate effective item embeddings, and this method can be seamlessly integrated with traditional recommendation models, thereby greatly improving recommendation performance;

[0038] 2) Improving sequence modeling performance by using fine-grained time information: The present invention introduces fine-grained interaction time information into project sequence modeling, so as to better understand the overall trend of projects and achieve more accurate project embedding;

[0039] 3) Enriching item embeddings by fusing multimodal information: The present invention cleverly integrates interaction information and text information into the item modeling process, enriches the embedding representation of items, and promotes the learning of semantic item embeddings.

[0040] The features and advantages of the present invention will be described in detail through embodiments in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0041] Figure 1 It is a framework diagram of a pre-training model of a general pre-training method for embedding items based on sequence modeling of the present invention;

[0042] Figure 2 This is the parameter sensitivity experiment result diagram of comparative example 3 on the Software dataset;

[0043] Figure 3 This is a graph of the parameter sensitivity experiment results of comparative example 3 on the ML-1M dataset. [Specific implementation method]

[0044] Embodiment 1:

[0045] See also Figure 1 The present invention provides a general pre-training method for project embedding based on sequence modeling, which is implemented by a pre-training model. The pre-training model includes a pre-processing layer, an embedding layer and a hybrid perception attention layer, and the training process includes the following steps:

[0046] Step 1: Use the preprocessing layer to perform data preprocessing in sequence through item sequence extraction, time conversion and text processing. The item sequence extraction is to construct an access sequence about the item based on the interaction time and let each element in the access sequence be a user who interacts with the item in chronological order. The time conversion is to convert the timestamp into fine-grained time information. The text processing is to simplify the text vocabulary:

[0047] The pre-trained model uses user-item interaction data as the main data source and generates item interaction sequences based on the interaction time. Since the lengths of item interaction sequences are different, this embodiment converts each sequence into a fixed length n (if the sequence length is greater than n, only the most recent n interactions are considered and other interactions are deleted; if the sequence length is less than n, the left end of the sequence is padded with pseudo user labels “0” until the length reaches n). Similarly, the corresponding interaction time series is also processed using the same method. Finally, we can obtain the item interaction sequence S = {u1, u2, ..., u n} and the interaction time series T = {t1, t2, ..., t n};

[0048] Time conversion is the process of converting timestamps into fine-grained time information, i.e., year, month, day, hour, minute, and second, in order to capture the temporal correlation and seasonal trends in item sequences (some items may be more popular during specific time periods, such as weekends and holidays, etc., and this information is not directly reflected in the original timestamp); in user-item interaction data, each interaction is associated with a timestamp; since the timestamp refers to the total number of seconds from 00:00:00 GMT on January 1, 1970 to the present, the numerical representation of the timestamp is usually very large (if it is directly input into the embedding layer for time embedding, it will lead to inefficiency); for this reason, the present invention proposes an effective time conversion strategy , the timestamp can be first decomposed from a single numerical value into specific years, months, days, hours, minutes, and seconds (for example, the specific time of "October 1, 2021 00:00:00" represented by the timestamp "1633046400000" is decomposed into "2021", "10", "1", "0", and "0"), and then these decomposed time elements are input as independent features into the embedding layer (where each fine-grained time unit is regarded as a category, and its corresponding embedding vector represents the unique behavior pattern of the time unit); this method can not only reduce the number of time elements and make the time encoding process more efficient, but also provide a deeper understanding of the impact of time factors on project popularity;

[0049] In the text processing process, the present invention effectively reduces the size of the text vocabulary by deleting useless characters (such as line breaks "\n", HTML tags "", etc.) in project names and descriptions and / or performing stemming and lemmatization operations, so that the model pays more attention to the text content itself, ensures the accuracy and effectiveness of text embedding, and improves the generalization ability of the model;

[0050] Step 2: Use the embedding layer to convert discrete features (such as user ID, sequence position, and fine-grained time information) and text features (such as item name and item description) into low-dimensional dense vectors (vectorization captures the potential relationship between features, making it easier for the model to handle high-dimensional sparse data):

[0051] First, given the initialized user embedding matrix E u ∈R |U|×d , position embedding matrix E p ∈R n×d , year embedding matrix E y ∈R 5000×d , monthly embedding matrix E mon ∈R 24×d , daily embedding matrix E d ∈R 62×d , when embedding matrix E h ∈R 48×d , embedding matrix E min∈R 120×d and the second embedding matrix E sec ∈R 120×d And d is used to represent the dimension of the embedding vector, and then the embedding layer is used to obtain the user embedding m∈R in the item interaction sequence through the embedding search operation n×d , position embedding and time embedding (position embedding and time embedding include value vector embedding (p V ∈R n×d ,y V ∈R n×d ,mon V ∈R n×d 、day V ∈R n×d 、h V ∈R n×d 、min V ∈R n×d and sec V ∈R n×d ) and key vector embedding (p K ∈R n×d ,y K ∈R n×d ,mon K ∈R n×d , day K ∈R n×d ,h K ∈R n×d , min K ∈R n×d , sec K ∈R n×d )); Finally, the project name and description are input into the SBERT model (J. Liao, S. Li, Z. Yang, et al. LLaRA: Large Language-Recommendation Assistant. in: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2024, pp. 1785-1795) to obtain the project name embedding N∈R n×d and the item description embedding D∈R n ×d ;

[0052] Step 3: Use the hybrid-aware attention layer to capture the complex collaborative relationship between sequence users based on multi-source information and through the self-attention mechanism in sequence modeling to obtain project embedding:

[0053] The essence of the attention mechanism is that its output depends only on the relevant parts of the input (in other words, different inputs contribute differently to the output); this contribution is the attention weight and can be calculated using the scaled dot product measurement method as shown in formula (1);

[0054]

[0055] Where Q, K, and V represent query, key, and value, respectively (in the self-attention mechanism, these three inputs are usually represented by the same object);

[0056] If the attention weights are calculated based only on time information, the model will be transformed into a traditional self-attention method; the present invention introduces more dimensions of information into the hybrid-aware attention layer for item embedding learning, including fine-grained interaction time (by converting timestamps into fine-grained time format as a key component of the attention network, the model's ability to capture and understand subtle differences in the time dimension is enhanced), the sequence order of users (in sequence modeling, the order of sequence elements is important; by assigning a unique position identifier to each user in the item interaction sequence, different interaction events can be distinguished even if the timestamps are the same), and item names and descriptions (item names and descriptions contain rich semantic information, which can help the model understand the characteristics of different items, thereby providing semantic-aware item embedding);

[0057] The implementation process of the hybrid perceptual attention mechanism is to first perform self-attention calculation on the interaction sequence (aiming to capture the intrinsic correlation between interacting users), then apply residual connections to retain the key information in the original user embedding to ensure effective transmission and enhancement of information, and finally integrate the processing results with the project text information to achieve comprehensive utilization of cross-modal information;

[0058] Specifically, given a project interaction sequence M = (m1, m2, ..., e n ) and use m j represents the embedding of the jth interacting user, and uses formula (2) to generate a new embedding sequence Z = (z1, z2, ..., z n );

[0059]

[0060] Where W V ∈R d×d represents the input projection matrix; embeddings representing where the user interacted; and Respectively represent fine-grained temporal embedding; a ij represents the weight coefficient and is calculated using formula (3) according to the Softmax function;

[0061]

[0062] In the formula, e ij represents the attention weight calculated based on user embedding, location embedding, year embedding, month embedding, date embedding, hour embedding, minute embedding and second embedding; Formula (4) is used to represent the e ij ;

[0063]

[0064] Where W Q ∈R d×d and W K ∈R d×d denote the query projection matrix and the key vector of the interacting user respectively; and The key vector embeddings representing fine-grained interaction time respectively; is the normalization factor (used to avoid the problem of too large inner product value caused by large vector dimension);

[0065] Although the hybrid perceptual attention layer can use information such as user, location and time to calculate the attention weight, the calculation process is implemented through linear combination. Therefore, after each self-attention layer, the present invention also adds two linear transformations to make the model more flexible. Specifically, formula (5) is used to transform z j Add linear transformation;

[0066] FFN(z j )=max(0,z j W1+b1)W2+b2 (5);

[0067] Where W1, W2∈R d×d Represents the projection matrix of two-layer linear transformation; b1, b2∈R d represents the bias term; although the linear transformation method is the same between different users, they use different parameters in each layer;

[0068] After stacking the self-attention layer and the forward propagation layer, problems such as overfitting, gradient disappearance, and the need for more training time often occur; here, the present invention uses layer normalization, residual connection, and dropout regularization techniques to solve these problems;

[0069] z j =z j +Dropout(FFN(LayerNorm(z j ))) (6);

[0070] To avoid overfitting, the concept of residual connection is introduced here; as shown in formula (7), the original user embedding is linearly transformed, and then each element is added to the output of the self-attention layer, and an average pooling operation is used to obtain the preliminary item embedding;

[0071]

[0072] In the formula, m i represents the original user embedding; Avg(·) represents the average pooling operation;

[0073] Finally, as shown in formula (8), the preliminary item embedding is fused with the embedding of the item name and description to obtain the item embedding containing rich semantic information (here, the sum pooling method is used for fusion);

[0074] Q = N + D + V (8);

[0075] Where N represents the item name embedding; D represents the item description embedding; and V represents the preliminary item embedding obtained by the self-attention layer.

[0076] In addition, the pre-trained model can be optimized as follows:

[0077] After multiple layers of self-attention, the embedding vectors obtained from each layer are stacked to obtain the comprehensive embedding of the project; as shown in formula (9), in order to predict the next interactive user of the current project, the present invention uses vector dot product to calculate the prediction score of each candidate user for the current project;

[0078] R i，k =q k m i T (9);

[0079] In the formula, m i represents the embedding of candidate user i; q k represents the embedding of the current item k;

[0080] In the session-based recommendation scenario, user-item interaction is implicit feedback data, and the model training process cannot directly optimize the prediction score; the goal of recommendation is to provide a ranked user list, so the present invention uses negative sampling technology to optimize user ranking; specifically, as shown in formula (10), given the item interaction sequence S of item k, k ∈S, for each expected positive user u_i∈S^k, randomly select a negative user To generate a pair of training samples O = (u i ,u j ), and use binary cross entropy as the loss function to train the model;

[0081]

[0082] As shown in formula (11), the final loss function is a combination of the prediction loss and the regularization term;

[0083]

[0084] In the formula, λ represents the regularization weight; ||.|| F represents the Frobenius norm; represents a learnable embedding.

[0085] Comparative Example 1:

[0086] This comparative example compares this method with common recommendation methods, and selects five popular datasets from real-world scenarios, namely Software, Appliances, ML-1M, YooChoose, and Diginetica, in order to provide a diverse analytical perspective for the experiment (these datasets have significant differences in data type, capacity, and sparsity); Software and Appliances are from Amazon.com, which are a collection of a large number of product reviews (this comparative example focuses on the top two product categories on the Amazon platform; they have attracted widespread attention due to their highly sparse review data and extensive content, and have also brought certain challenges to our experiments); ML-1M is a popular dataset in the MovieLens series, containing millions of user rating records. It is a recognized benchmark test set in the field of recommendation systems and provides a basis for comparing the performance of different recommendation models; YooChoose and Diginetica are anonymous serialized datasets, which can not only enrich the diversity of experimental data, but also provide a new perspective for exploring the generalization ability of the model.

[0087] First, we extracted elements such as interaction timestamps, project titles, and project descriptions from the original dataset; then, we constructed project interaction sequences based on timestamps (for serialized datasets, users are anonymous, and this comparison uses session IDs as user IDs to construct project interaction sequences); next, in order to ensure the scientific nature of the evaluation process, this comparison uses the latest interaction data as the test set, the immediately following one as the validation set, and the rest as the training set, and the split ratio of the three types of data is 1:1:8; finally, given that some baseline models tend to exclude users and projects with interaction frequencies below a certain threshold during the preprocessing stage, this comparison analyzes two datasets of different sizes; in addition, Table 1 provides a statistical description of the original dataset, and Table 2 provides a statistical description of the dataset after excluding users and projects with interaction frequencies less than 5; this strategy aims to evaluate the performance of the proposed pre-trained model at different sparsity levels to verify its ability to alleviate the data sparsity problem.

[0088] Table 1 Statistical description of the original data set

[0089]

[0090] Table 2 Statistical description of the filtered data set

[0091]

[0092] In this comparative example, we used four popular top-k ranking indicators (i.e., HR@k, NDCG@k, P@k, and MRR@k) as evaluation indicators to measure the recommendation performance; among them, HR@k is used to measure the click-through rate of the top-k recommended items; NDCG@k further considers the ranking score of the position of each recommended item; P@k is used to measure the average ranking value of the correct recommended items; MRR@k and a large MRR value indicates that the correct recommended item is at the top of the recommendation list; in order to verify the performance of the model on recommendation lists of different lengths, this comparative example evaluates the recommendation performance under three different k values.

[0093] During the experiment, for each real interactive user u, this comparison randomly selected 100 negative examples and ranked them together with user u; based on the rankings of these 101 users, this comparison calculated the above performance indicators; in addition, the model parameter settings of this comparison include the dimension of the hidden layer being consistent with the basic model, the sequence length being 120, the learning rate being 0.0015, and the regularization parameter λ being 0.001.

[0094] Regarding the baseline method, this comparative example selected the following three currently popular and representative session-based recommendation models as downstream recommendation models, and applied the proposed pre-trained model to these recommendation models; this comparative example verified the effectiveness of the pre-trained model by comparing the recommendation performance before and after applying the pre-training strategy;

[0095] 1)SASRec: In this method, all items are strictly arranged in the order of interaction time, and a self-attention mechanism is used for sequence modeling to capture the temporal dependencies between items;

[0096] 2) TiSASRec: As an advanced version of SASRec, TiSASRec not only retains the core advantages of the self-attention mechanism, but also integrates the time interval information between sequence items in the attention layer, which can further enhance the model's ability to model time-sensitive user preferences.

[0097] 3) SR-GNN (S.Wu, Y.Tang, Y.Zhu, et al. Session-based recommendation with graph neural networks. in: Proceedings of the AAAI conference on artificial intelligence, 2019, pp. 346-353.): This method introduces a gated convolutional layer, through which the model can flexibly capture the complex transformation relationship between sequence items, thereby showing stronger generalization ability and accuracy in sequence recommendation.

[0098] During the experiment, this comparison innovatively combines the item embedding (Q') learned by the base model with the item embedding (Q) obtained by the pre-trained model to create a new item embedding representation (Q * ), and then used to make next hop recommendation, and the fusion method is shown in formula (12);

[0099]

[0100] Here, ∈ represents the bias term; here, this comparative example uses a simple summation operation to fuse the embeddings from the two sources; in fact, we have tried other fusion methods (such as embedding the items obtained by the pre-trained model as the input of the downstream recommendation model, and continuing to optimize the embedding representation in the downstream recommendation model), but the recommendation performance is not as good as the simple summation method (but the training time of the downstream recommendation model is indeed shortened).

[0101] Table 3 below shows the recommendation performance of different recommendation models before and after applying the pre-trained model. The prefix "pre" in the model name indicates that the pre-trained model has been used.

[0102] Table 3 Comparative experimental results

[0103]

[0104] As can be seen from Table 3 above, in the highly sparse dataset software, combining the pre-trained model with the basic recommendation model achieves the most significant performance improvement in the Top-5 recommendation scenario (this result strongly proves the unique advantage of the pre-trained model in alleviating the data sparsity problem); similarly, in the sparse dataset of Appliances, the application of the pre-trained model also brings significant performance improvement (which further verifies its effectiveness in sparse scenarios), and this is because the pre-trained model introduces multi-source information (text information, sequence order, interaction timestamp) for item embedding, thereby obtaining a universal item embedding rich in semantic information (that is, combining this embedding can effectively improve the performance of the downstream recommendation model); in relatively dense datasets such as ML-1M, the pre-trained model also shows the potential to improve the performance of the basic recommendation model; although the improvement is reduced compared with the sparse dataset, this is enough to prove its wide applicability and flexibility in the sequence recommendation task; in summary, the pre-trained model proposed in the present invention is a powerful preprocessing tool that can provide new ideas for optimizing downstream recommendation models under various data conditions.

[0105] Through the above experimental results, this comparative example finds that in interaction-intensive datasets, the advantages of pre-trained models are relatively limited; this finding is attributed to the fact that in dense datasets, the rich interaction records between users and items are sufficient to provide sufficient information for user modeling and recommendation; therefore, the knowledge gained in the pre-training stage is not significantly helpful for such scenarios; however, when faced with relatively sparse datasets, the advantages of pre-trained models are particularly prominent, and their improvement effect is extremely significant (this is because in data-sparse scenarios, pre-trained models can effectively fill the information gaps in the target dataset caused by insufficient interactions by learning general knowledge of large-scale datasets, such as deep semantic features of items, universal rules of user behavior patterns, etc.); in addition, it is worth noting that compared with pre-SASRec, the excellent performance of pre-TiSASRec on sparse datasets is more prominent; this result reveals the complementary relationship between recommendation model architecture and pre-training methods; in addition, this finding shows that careful selection of model architecture and combining it with a suitable pre-training strategy can significantly improve recommendation performance.

[0106] It is also worth pointing out that the significant difference in performance between the basic recommendation models lies in TiSASRec’s unique data filtering strategy, which removes users and items with less than 5 interactions; this design was originally intended to weaken the interference of noisy data on model training, but in extremely sparse data environments, this mechanism may accidentally reduce the size of training samples, thereby limiting the model’s ability to capture the diversity of user interests, resulting in unsatisfactory performance on sparse datasets; in contrast, SASRec does not apply any data filtering strategy, so it can maintain stable performance when processing sparse or dense datasets.

[0107] In order to further verify the versatility of the pre-trained model, this comparative example also conducted a comparative experiment on the session-based recommendation model SR-GNN; due to the lack of item title and description information in this dataset, only the sequence position and interaction time were used in the hybrid permanent attention layer; the experimental results are shown in Table 4. It can be seen that even under such a limited data source, the pre-trained model can still significantly improve the recommendation performance of SR-GNN, which fully demonstrates the wide applicability and effectiveness of the pre-trained model proposed in this paper in the field of session-based recommendation.

[0108] Table 4 SR-GNN comparative experimental results

[0109]

[0110] The universal pre-trained model proposed in the present invention uses interaction information as the main data source, can extract project interaction sequences according to the interaction time, convert the interaction timestamp into fine-grained time information, comprehensively consider the project name, project description, sequence order and fine-grained interaction time, and learn project embedding based on the sequence modeling method to provide effective input for downstream recommendation tasks; the experiments conducted on five real-world datasets and three basic recommendation models above show that the pre-trained model proposed in the present invention has strong compatibility and effectiveness, and can not only be applied to different recommendation models, but also can effectively alleviate the problem of data sparsity.

[0111] Comparative Example 2:

[0112] In order to analyze the specific effects of the three core components of the pre-trained model in Example 1 (i.e., project title, project description, and timestamp) on project embedding, this paper conducts an ablation experiment; this experiment uses SASRec as the basic recommendation model and selects an unfiltered software dataset as the experimental object; the experimental results are shown in Table 5, and "our" represents the complete pre-trained model, "no_title" represents the pre-training mode after deleting the project title, "no_time" represents the pre-training mode after deleting the timestamp, and "no "description" represents the pre-training model after deleting the item description; it can be seen from Table 5 that the absence of the item title has the most significant impact on the model performance, and its effect is significantly greater than that of the item description as text information (this finding strongly proves the central position of the item title in item embedding); at the same time, the timestamp information also shows undeniable importance (in the performance evaluation of the top 10 and top 20 suggestions, its importance is comparable to that of the item title, highlighting the key role of temporal dynamics in item embedding); in addition, although the use of item description alone may not produce a significant performance improvement effect, combining the item title with the item description can bring more significant performance improvement than using any component alone; in short, each component of the pre-training model proposed in the present invention plays an important role in item embedding and is indispensable.

[0113] Table 5 Ablation experiment results

[0114]

[0115] Comparative Example 3:

[0116] In the parameter sensitivity experiment of this comparative example, the influence of learning rate and item sequence length on item embedding is mainly analyzed; the experiment uses SASRec as the basic recommendation model, selects Software and ML-1M datasets as experimental objects, and the experimental results are as follows: Figure 2 and Figure 3As shown in the figure; on the Software dataset, as the sequence length increases, the recommendation performance gradually improves (this is because the longer the sequence length, the more information can be used for item embedding, which naturally improves the modeling performance); when the item sequence length reaches 120, the pre-trained model shows the best performance; however, further increasing the sequence length leads to a decrease in performance (this is because user interests are dynamic, and long-term interactions do not contribute to item embedding, but instead become noise, thus affecting the accuracy of modeling); when the learning rate is set to 0.0015, the performance is best (neither too large nor too small can achieve the ideal performance, which indicates that 0.0015 is the most appropriate learning rate); the sequence length and learning rate show the same change pattern on the dataset ML-1M; when the item sequence length is 80 and the learning rate is 0.0015, the model achieves the best performance.

[0117] The above embodiments are intended to illustrate the present invention, not to limit the present invention. Any solution that is a simple transformation of the present invention belongs to the protection scope of the present invention.

Claims

1. A general pre-training method for item embedding based on sequence modeling, characterized by: This is achieved through a pre-trained model, which includes a pre-processing layer, an embedding layer, and a hybrid perception attention layer, and the training process includes the following steps: Step 1, using the preprocessing layer to perform data preprocessing in sequence through item sequence extraction, time conversion and text processing, wherein the item sequence extraction is to construct an access sequence about the item based on the interaction time and each element in the access sequence is a user who interacts with the item in chronological order, the time conversion is to convert the timestamp into fine-grained time information, and the text processing is to simplify the text vocabulary; Step 2: Using the embedding layer to convert discrete features and text features into low-dimensional dense vectors; Step 3: Utilize the hybrid-aware attention layer to capture the complex collaborative relationship between sequence users based on multi-source information and through the self-attention mechanism in sequence modeling to obtain project embedding.

2. The general pre-training method for item embedding based on sequence modeling as claimed in claim 1, characterized in that: In step 1, the item sequence extraction converts each sequence into a fixed length n and obtains an item interaction sequence S = {u1, u2, ..., u n } and the interaction time series T = {t1, t2, ..., t n }, the fine-grained time information is the year, month, day, hour, minute and second time information, and the simplified text vocabulary is to delete useless characters in the project name and description and / or perform stemming and lemmatization operations.

3. The general pre-training method for item embedding based on sequence modeling as claimed in claim 2, characterized in that: In step 2, the initialized user embedding matrix E is given first u ∈R |U|×d , position embedding matrix E p ∈R n×d , year embedding matrix E y ∈R 5000×d , monthly embedding matrix E mon ∈R 24×d , daily embedding matrix E d ∈R 62×d , when embedding matrix E h ∈R 48×d , embedding matrix E min ∈R 120×d and the second embedding matrix E sec ∈R 120×d And d is used to represent the dimension of the embedding vector, and then the embedding layer is used to obtain the user embedding m∈R in the item interaction sequence through the embedding search operation n×d , position embedding and time embedding and input the project name and description into the SBERT model to obtain the project name embedding N∈R n×d and item description embedding D∈R n×d , the position embedding and time embedding include value vector embedding and key vector embedding, the value vector embedding includes p V ∈R n× d, y V ∈R n×d ,mon V ∈R n×d 、day V ∈R n×d 、h V ∈R n×d 、min V ∈R n×d and sec V ∈R n×d , the key vector embedding includes p K ∈R n×d ,y K ∈R n ×d ,mon K ∈R n×d 、day K ∈R n×d 、h K ∈R n×d 、min K ∈R n×d and sec K ∈R n×d .

4. The general pre-training method for item embedding based on sequence modeling as claimed in claim 3, characterized in that: In step 3, the hybrid perceptual attention layer implements the hybrid perceptual attention process by first performing self-attention calculation on the interaction sequence, then applying residual connection to retain the key information in the original user embedding, and finally integrating the processing results with the project text information.

5. The general pre-training method for item embedding based on sequence modeling as claimed in claim 4, characterized in that: In step 3, given a project interaction sequence M = (m1, m2, ..., e n ) and use m j represents the embedding of the jth interacting user, and uses formula (2) to generate a new embedding sequence Z = (z1, z2, ..., z n ); Where W V ∈R d×d represents the input projection matrix; embeddings representing where the user interacted; and Respectively represent fine-grained temporal embedding; a ij Represents the weight coefficient.

6. The general pre-training method for item embedding based on sequence modeling as claimed in claim 5, characterized in that: Formula (3) is used to calculate the a according to the Softmax function. ij ; In the formula, e ij Represents the attention weights calculated based on user embedding, location embedding, year embedding, month embedding, day embedding, hour embedding, minute embedding, and second embedding.

7. The general pre-training method for item embedding based on sequence modeling as claimed in claim 6, characterized in that: Using formula (4) to express the e ij ; Where W Q ∈R d×d and W K ∈R d×d denote the query projection matrix and the key vector of the interacting user respectively; and The key vector embeddings representing fine-grained interaction time respectively; is the normalization factor.

8. The general pre-training method for item embedding based on sequence modeling as claimed in claim 7, characterized in that: In step 3, z is calculated using formula (5) j Add linear transformation; FFN(z j )=max(0,z j W1+b1)W2+b2 (5); Where W1, W2∈R d×d Represents the projection matrix of two-layer linear transformation; b1, b2∈R d Represents the deviation term.

9. The general pre-training method for item embedding based on sequence modeling as claimed in claim 8, characterized in that: In step 3, formula (6), formula (7) and formula (8) are used to calculate z j Perform layer normalization, residual connection and dropout regularization; z j =z j +Dropout(FFN(LayerNorm(z j ))) (6); Q = N + D + V (8); In the formula, m i represents the original user embedding; Avg(·) represents the average pooling operation; N represents the item name embedding; D represents the item description embedding; V represents the preliminary item embedding obtained by the self-attention layer.

10. The general pre-training method for item embedding based on sequence modeling as claimed in claim 9, characterized in that: Given an item k, the item interaction sequence S k ∈S, for each expected positive user u_i∈S^k, randomly select a negative user To generate a pair of training samples O = (u i ,u j ) and use binary cross entropy as the loss function, and use formula (9), formula (10) and formula (11) to optimize the pre-trained model; R i，k =q k m i T (9); In the formula, m i represents the embedding of candidate user i; q k represents the embedding of the current item k; represents the learnable embedding; λ represents the regularization weight; ||.|| F represents the Frobenius norm.

Images