Transform-based cross-sequence multi-behavior sequence recommendation method

Abstract

The invention discloses a cross-sequence multi-behavior sequence recommendation method based on a Transform. The cross-sequence multi-behavior sequence recommendation method comprises the following steps of: firstly, carrying out multi-behavior sequence recommendation; the method comprises the following steps: firstly, segmenting a user historical interaction sequence, and then executing cross-sequence heterogeneous relationship propagation; performing short-term interaction mode learning; global representation aggregation is carried out; and finally, generating a recommended content list in combination with recent preferences of the user and project embedding, and performing label data enhancement and model parameter adjustment by utilizing historical interaction of the user in consideration of the content priority. The invention provides a user sequence segmentation method, which ensures that the subsequences keep the relatively concentrated and stable preference of the user, so that a model can more accurately capture the dynamic change preference of the user, and provides a similar subsequence judgment mechanism and a method for executing cross-sequence information propagation between similar subsequences. And the time coding information is reserved in the information spreading process, so that the diversity and accuracy of recommended contents are improved.

Description

Technical Field

[0001] The present invention mainly relates to the technical field related to recommendation systems, and specifically to a Transformer-based cross-sequence multi-behavior sequence recommendation method. Background Art

[0002] In order to alleviate the problem of information overload caused by the rapid development of information technology, the concept of recommendation system with the goal of providing personalized content recommendation was proposed. Since user preferences usually evolve over time, the sequential recommendation system that models the user's dynamic preferences based on the user's historical interaction behavior came into being and achieved remarkable results, gradually becoming a hot research topic.

[0003] In recent years, as deep learning technology has achieved great success in many fields, people have begun to turn their attention to using various neural network architectures to model user-item interaction records and explore the evolution of user interests to solve the problem of sequential recommendation. Existing studies on sequential recommendation mostly focus on modeling a single type of interaction behavior record. In practical applications, the types of interactions between users and items are usually diverse, and different behaviors reflect different degrees of user preferences. For example, a user is more likely to buy an item that has been added to the shopping cart than a product that has only been browsed. There have been related studies that have attempted to distinguish the latent semantics of complex interaction types. Early methods included pre-defined cascade relationships and learning user preferences from multi-behavior data. The current mainstream methods include using recursive neural models and graph neural network models.

[0004] Existing multi-behavior recommendation work has achieved certain results in mining the degree of preference reflected by complex behaviors, but all of them focus on mining the user's own interaction sequence, ignoring the supplementary information provided by the interaction sequences of other users with similar interests, and insufficiently utilizing the information of interaction time, which makes it difficult to capture the dynamic preferences of users that evolve over time. Therefore, how to cut the user's historical interaction sequence so that each subsequence presents the user's relatively stable and concentrated preferences is an unresolved problem. Secondly, how to mine user interaction information from a global perspective and perform information propagation is of great significance to improving the performance of the recommendation system. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a Transformer-based cross-sequence multi-behavior sequential recommendation method CMB (Cross sequence Multi-Behavior Sequential Recommendation based on Transformer), which can accurately capture the long-term and short-term preferences of users and realize accurate service content recommendation.

[0006] To achieve the above object, the technical solution of the present invention is as follows: a Transformer-based cross-sequence multi-behavior sequence recommendation method, characterized in that it includes the following steps:

[0007] Step 1: Segment the user's historical interaction sequence;

[0008] Step 2: Propagation of heterogeneous relations across sequences;

[0009] Step 3: Short-term interactive mode learning;

[0010] Step 4: Global representation aggregation;

[0011] Step 5: Combine the user's recent preferences with the item embedding to generate a list of recommended content, while considering the content priority, and adjust the model parameters by using the user's historical interactions to perform label data enhancement.

[0012] Beneficial effects of the present invention: Compared with the existing multi-behavior recommendation methods, which have the problems of insufficient utilization of temporal context information, inaccurate capture of user dynamic preferences, and neglect of mining user interaction information under the global view, the advantages of the present invention are: (1) A user sequence segmentation method is proposed to ensure that the subsequences maintain the user's relatively concentrated and stable preferences, so that the model can more accurately capture the user's dynamically changing preferences. (2) A similar subsequence determination mechanism and a method for performing cross-sequence information propagation between similar subsequences are proposed, and the time coding information is retained during the information propagation process, thereby improving the diversity and accuracy of the recommended content. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is the overall framework diagram of the Transformer-based cross-sequence multi-behavior sequence recommendation method of the present invention.

[0014] Figure 2 Schematic diagram of user interaction sequence segmentation.

[0015] Figure 3 Schematic diagram of similar subsequences.

[0016] Different colors are used in the figure to distinguish the behavior types and different channel parameter matrices in the multi-channel projection layer. DETAILED DESCRIPTION

[0017] Furthermore, the segmentation of the user historical interaction sequence described in step 1 is specifically as follows:

[0018] The recommendation scenario of the present invention takes the e-commerce platform as an example. Behaviors that reflect user preferences, such as browsing and collecting, are called auxiliary behaviors, while behaviors that the platform hopes users will do to generate revenue for the platform, such as purchasing, are called target behaviors. Based on this scenario, the user set is written as U = {u1,…,u i,…,u I}, the item set is written as V = {v1,…,v j ,…v J}, the various types of interactive behaviors that appear in the scenario are defined as set B (indexed by b) containing all possible behavior types (e.g., browsing, marking as favorite, adding to cart, purchasing).

[0019] where I, J∈Z represent the number of users and items respectively.

[0020] In the context of the present invention, each user's historical interaction is recorded as a multi-behavior interaction record. Specifically, for a user u i ∈U, whose historical interaction information is defined as a triple (v i,x , b i,x , t i,x ), represents user u i At time x, b i,x The xth item of the type behavior interaction) constitutes user u i The multi-behavior interaction record is represented by S i ={(v i,1 , b i,1 , t i,1 ),…,(v i,x , b i,x , t i,x ),…,(v i,X , b i,X , t i,X )}.

[0021] Where X represents the sequence length |S i |.

[0022] In order to model the user's preferences that evolve over time in the short term, we consider segmenting the user's interaction records and encoding them with contextual information. Then, in order to enhance information fluidity and improve the diversity of the recommendation system, this method designs a cross-sequence information transmission component so that the items that the user has interacted with can obtain information from similar interaction records of other users.

[0023] First, the user’s interaction sequence is segmented by the Seg(·) operation to obtain multiple subsequences. i and its interaction sequence S i , the Seg(·) operation can be expressed as formula (1):

[0024]

[0025] Where P (indexed by p) represents the number of user subsequences divided, b target Indicates the target behavior, and Respectively represent u i the lengths of the first and last subsequences, Indicates the maximum length of a subsequence.

[0026] Specifically, the process integrates all user interaction sequences, and divides the user interaction sequence into several subsequences with the item where the user performs the target behavior (such as purchase) as the cutoff point. The reason for this is that before a user purchases an item, the items that the user interacts with through other behaviors have different degrees of similarity or correlation. Before a user purchases a product, the preferences reflected by the user interaction subsequence are relatively concentrated and stable. Obtaining the user's subsequence in this way helps to accurately capture the user's short-term preferences. If the user has not performed the target behavior, in order to analyze the user's short-term preferences, the maximum subsequence length is used. Segment the user's historical interaction sequence.

[0027] In the global view, if the last item in the interaction subsequences segmented by all users is the same item interacted with by the target behavior, then the two subsequences are considered similar. For example, given user u i The kth subsequence of User i′ There are subsequences If yes Then the sequence with sequence Similar, recorded as Each subsequence can have multiple similar subsequences.

[0028] Furthermore, the cross-sequence heterogeneous relationship propagation described in step 2 is specifically as follows:

[0029] This method designs an embedding network to jointly integrate behavior type information and time signals as context information into the feature representation of items. First, following the mainstream recommendation paradigm, the initial embedding of items is j ∈R d The item embedding matrix M∈R generated by embedding the item set V into the same space |I|×d Get, related to the project id.

[0030] where d represents the potential state dimension. Then, e b ∈R d Associated with the user behavior type, it indicates the bth type of behavior embedding.

[0031] In addition, in order to give the model time perception, this method introduces a time encoding strategy in the context information encoding layer, that is, using formula (2) to map the scalar timestamp to the vector space:

[0032]

[0033] where τ(·) represents the function that maps timestamps to 2D vectors, and W t ∈R 2d×d represents the time transformation matrix, and the odd and even indices of the 2D vector are represented by 2l and 2l+1. In addition, in order to avoid the influence of large embedding values, use As a scaling factor, W t The role of the matrix is ​​to introduce an adjustable linear projection mechanism, giving the model the ability to flexibly learn temporal context information.

[0034] After that, the behavior type representation and temporal context information are jointly injected into the item representation by the following operations:

[0035]

[0036] in Represents item embeddings that incorporate contextual information and serves as input to the cross-sequence information transfer component, Represents the corresponding addition operation of vector elements.

[0037] In order to capture the dynamic similarity relationship between different items in the sequence view and the global view, this method designs a cross-sequence information propagation encoder based on the attention mechanism in the multi-head representation space and built on the Transformer architecture.

[0038] Specifically, the embedding of the xth interaction term in the sequence is calculated by the multi-head dot product attention method as shown in formula (4):

[0039]

[0040] in represents the user u calculated after cross-sequence information aggregation i The refined representation of the xth interaction term under the kth subsequence, Y represents the item The set of related items (i.e., the sequence Other items and sequences in ), E y represents the embedding representation of the relevant terms, α x,y Represents the items learned and v y The correlation coefficient between them, σ(·) represents the softmax function, h∈H represents the hth head of the attention layer, Q h , K h , Represent the Query, Key and Value transformation matrices of the h-th head (three important parameter matrices in the attention mechanism), and divide the scores of the query and value vectors by to stabilize the gradient.

[0041] The introduction of the multi-head attention mechanism enables the encoder to perform information aggregation of different semantic dimensions in multiple representation subspaces. The cross-sequence heterogeneous relationship propagation module integrates the context information encoding layer and the cross-sequence information propagation layer, which not only gives the model sensitivity to the user interaction context information, but also captures the dynamic characteristics of multiple behaviors and preserves the transition relationship between items in the sequence view and the global view.

[0042] Furthermore, the short-term interaction mode learning described in step 3 is specifically as follows:

[0043] This method designs a behavior-aware encoder to aggregate specific types of behavior patterns in the short term, giving the model the ability to distinguish the heterogeneity of behaviors within a sequence. i with sequence u i The behavior pattern of type b in the corresponding short term can be formally expressed as formula (5):

[0044]

[0045] in Indicates that the aggregation calculates u i exist The characteristic representation of the behavior of b in the period, ρ(·) is the judgment function defined by this method. If the two input items are equal, the result is 1, otherwise it is 0. is the item-refined embedding representation output by the cross-sequence heterogeneous relation propagation module, W b is a behavior-aware transformation matrix used to enhance the semantics of a specific type of behavior. Its optimization path is based on the multi-channel parameter learning strategy shown in formula (6):

[0046]

[0047] Where M represents the number of channels (indexed by m), P∈R M×d and μ∈R M Represent the conversion matrix and bias parameters in the fully connected layer, respectively. Denotes the transformation matrix corresponding to the mth channel. The purpose of introducing the multi-channel parameter learning strategy is to preserve the heterogeneity of interactive behaviors during message transmission, and use the channel-based aggregation layer to encode the behavior-aware semantics, so that the CMB model can capture the potential semantics of different types of user behaviors.

[0048] After obtaining the embedding representation of a specific behavior type, the short-term behavior type encoding needs to be fed to the aggregation layer to capture the potential interdependencies between different types of behaviors and obtain the short-term feature representation of the user. Different from the static attention relationship aggregation scheme adopted by other recommendation models, this method uses an adaptive attention network to calculate the contribution index of each behavior type to the short-term representation. The aggregation layer operation can be formalized as the following formula:

[0049]

[0050] in Is user u i The characteristic representation of the kth subsequence is u i The short-term preference encoding in the corresponding period, β i,b represents the sum of all behavior embeddings as the query learned u i The attention weight corresponding to action b, W A and μ A are the transformation matrix and bias of the aggregation layer respectively.

[0051] After the aggregation layer, user short-term characterization The time and behavior-aware context information is explicitly preserved in the

[0052] Correspondingly, for the message transmission process from the user to the project side, in order to refine the behavior-aware embedding of the project, the embedding encoding of the project also performs the above-mentioned propagation and aggregation process, as shown in formula (8):

[0053]

[0054] in and E j Respectively represent project v j Specific behavior type embedding and fused behavior-aware embedding. This scheme can distinguish the impact of user behavior on items and incorporate time signals in the message transmission process, because the user subsequence feature encoding is involved in the calculation during the propagation process.

[0055] Furthermore, the global representation aggregation described in step 4 is specifically as follows:

[0056] The user's preference is essentially dynamic over time, so the CMB model further proposes a global graph relation encoder to inject the user's embedding representations of each period into the global feature representation, capturing the user's multi-behavior dynamic patterns from a long-term and short-term perspective. This method defines a global user graph to include all user nodes u i ∈U and their interacting subsequence nodes That is, \(G = \{U, S, \epsilon\}\), where \(S\) and \(\epsilon\) represent the set of all user subsequences and the connection of each user \(u\) i with the edge of his interaction subsequence node (1 < k < K). The global representation aggregation module performs graph structure information aggregation on the global user graph \(G\) to generate the global embedding representation of the user, and incorporates temporal context information during the information aggregation process, enabling the short-term representations at different times to interact in a distinguishable manner. The calculation process is as shown in Equation (9):

[0057]

[0058] where represents the global feature representation of user \(u\), \(\sigma(\cdot)\) represents the softmax activation function, and \(\eta\) i represents the learned attention weight of user \(u\) k with its \(k\)-th subsequence, i and is the temporal encoding returned by the context information encoding layer, corresponding to the interaction occurrence time of the last item in the sequence .

[0059] By doing so, CMB retains the user preference embeddings of each period and incorporates a unified global user feature embedding representation.

[0060] After the above process, the CMB framework performs information aggregation and encodes the short-term interaction behavior patterns and long-term dynamic user embedding representations of users. During the embedding propagation process, CMB uses the user-item interaction graph and the global user graph \(G\) as the computational graph. During this period, the local relationship features from the neighborhood will be aggregated to obtain the context representation. Such a message passing paradigm can be generalized to Equation (10) by incorporating higher-order connectivity (\(l\) represents the \(l\)-th layer of the graph neural network in the graph message propagation process):

[0061]

[0062] where and are the item and user embeddings initialized by the encoding layer respectively. By extending the \(L\)-layer graph relation encoder, CMB is given the ability to capture higher-order neighbor collaboration signals. Finally, the final user item and subsequence feature representations are obtained through the element-wise addition operation of the \(L\) layers, and the process can be formulated as Equation (11):

[0063]

[0064] Furthermore, the recommended content list is generated by combining the user's recent preferences and item embeddings as described in step 5, while taking content priority into consideration, and adjusting model parameters by using user historical interactions to enhance label data, as follows:

[0065] In the prediction layer of the model, in order to reflect the dynamic characteristics of user preferences, this method jointly uses the short-term feature representation of the user in the last stage With the project feature representation E j Predict the user's possible future interaction items. i and project v j The probability of an interaction occurring can be quantified as in is the parameter vector, represents the element-by-element multiplication operation. During the training phase, the parameter adjustment process is optimized by using the user's historical interactions to enhance the label data, so u i and v j exist The prediction score for the corresponding period can be calculated as: In this case, the objective function of model training and the marginal pairwise loss can be formalized as follows:

[0066]

[0067] Where I and K i Represent the number of users and user u respectively i The number of interaction subsequences, C represents the number of samples (indexed by c), p c and n c Represent the corresponding positive samples and negative samples respectively. This method further introduces a weight decay regularization term To alleviate the overfitting phenomenon.

[0068] Experimental process:

[0069] a) Dataset

[0070] In order to study the performance of the CMB model in real scenarios, the present invention collected two public data sets from two real-world e-commerce platforms for experimental evaluation (see Table 1). Both data sets contain multiple types of user behaviors.

[0071] Table 1

[0072] Dataset details

[0073] Dataset name Number of users Number of projects Number of interactions Number of behavior types Taobao 147894 99037 7658926 4 IJCAI 423423 874328 36203512 4

[0074] Taobao: Taobao is the largest e-commerce platform in Asia, providing services such as guaranteed transactions. This dataset contains user-product interaction records collected from Taobao's retail system over a period of time, covering four types of interactions. Specifically, multiple types of behaviors include browsing pages, adding to carts, marking as liked, and purchasing.

[0075] IJCAI-Contest: This dataset records user interactions on online e-commerce websites. It also contains four types of interactive behaviors: click, add to cart, mark as favorite, and purchase. The source of the dataset is the IJCAI competition.

[0076] The experiment first organizes each record in the data set into the format of user id, project id, behavior type, and timestamp, and generates an interaction sequence S according to the order of the user's interaction timestamps. i ,Considering the profitability of the e-commerce platform, “purchase” is set as the target behavior, and other types of user behaviors are regarded as auxiliary behaviors that provide supplementary information for the target behavior.

[0077] b) Evaluation indicators

[0078] Following the mainstream recommendation system evaluation method, two widely adopted indicators are used to evaluate the performance of the CMB model and the baseline model: hit rate (HR@N) and normalized discounted cumulative gain (NDCG@N), so as to measure the accuracy of the top N recommendation results provided by these methods for different users. The performance of the model is proportional to the NDCG and HR scores. HR@N represents the percentage of items in the top-N list recommended to the user in the test set, reflecting the accuracy of the recommendation results. NDCG@N calculates the relevance of the recommendation results at each position in the recommendation list. It not only considers the relevance of the recommended items, but also the position of the recommended items. This is because the ranking position of the items in the recommendation system has a great impact on the user experience. The NDCG score can reflect the ranking quality of the recommendation results. For the training and test data samples generated based on the temporal information of the user interaction data, the leave-one-out method is used for evaluation, that is, for each user in the test set, the last interaction item in the data set is used as the test instance for recommendation.

[0079] c) Comparison method

[0080] [1] Neural network combined with collaborative filtering model:

[0081] NCF: A neural network is introduced to learn the interaction information between users and items. Unlike the traditional matrix decomposition model that directly operates on the user-item interaction matrix, NCF is an explicit model that predicts ratings by learning the embedding vectors of users and items. It can learn nonlinear user-item interaction relationships.

[0082] [2] Sequence recommendation model based on RNN / CNN:

[0083] GRU4Rec: This method utilizes the Gated Recurrent Unit (GRU) structure to capture long-term dependencies in user behavior sequences.

[0084] Caser: Combining the advantages of convolutional neural networks and recurrent neural networks, using one-dimensional convolutional layers to capture local temporal patterns and local dependencies in user behavior sequences.

[0085] [3] Sequence recommendation model based on attention mechanism / Transformer:

[0086] SASRec: Uses a multi-head self-attention mechanism to capture the sequential patterns of user historical behaviors without repeated operations on the input sequence.

[0087] BERT4Rec: A Transformer-based model that leverages the bidirectional encoder features of BERT to simultaneously consider the forward and backward dependencies of user behavior sequences and capture richer contextual information.

[0088] MATN: This method designs a Transformer-based memory-augmented network to model the interdependencies of multiple behaviors.

[0089] [4] Sequential recommendation model based on graph neural network:

[0090] SR-GNN: This model designs a gated graph neural network to process user session data.

[0091] HyperRec: Uses hypergraphs to represent user behavior sequences and predict the user's next interactive item. The hypergraph structure allows HyperRec to capture multi-dimensional relationships in user behavior sequences (co-occurrence relationships between items, similarities between users, and user behavior timing information, etc.)

[0092] [5] Multi-behavior recommendation model:

[0093] MAGNN: Use memory-augmented graph neural networks to encode long-term and short-term dependencies of items and aggregate the two through a fusion layer.

[0094] MBGCN: This model integrates various user behavior data into a unified graph to alleviate the data sparsity problem, and then uses a graph convolutional neural network to learn node representation.

[0095] TGT: A temporal graph encoder is designed to jointly encode node representations by combining temporal and behavioral context information.

[0096] MBSSL: A multi-behavior self-supervised learning framework and an adaptive optimization method are designed to integrate time and behavior information for node self-identification at both the inter-behavior and intra-behavior levels.

[0097] KEMB-Rec: Uses user behavior hypermetagraph to mine user behavior patterns and mines semantic relationships based on single behavior interaction graph.

[0098] Tables 2 and 3 show the performance of these models and the CMB model on all indicators on the two datasets. As can be seen from the table, the performance of the patented model is better than other methods in the HR@N and NDCG@N indicators on the two datasets, and it also has at least 5% performance improvement compared to the advanced multi-behavior recommendation method.

[0099] Table 2

[0100] HR@N and NDCG@N performance of all models on Taobao dataset (N=1, 5, 10)

[0101]

[0102] Table 3

[0103] HR@N and NDCG@N performance of all models on the IJCAI dataset (N=1, 5, 10)

[0104]

[0105] The above content describes the implementation process and advantages of the present invention. Those skilled in the art should understand that without departing from the principles of the present invention, the present invention may be subject to various changes and improvements, and these changes and improvements are all within the scope of the present invention claimed for protection.

Claims

1. A Transformer-based cross-sequence multi-behavior sequence recommendation method, characterized in that: The following steps are involved: Step 1: Segment the user's historical interaction sequence; Step 2: Propagation of heterogeneous relations across sequences; Step 3: Short-term interactive mode learning; Step 4: Global representation aggregation; Step 5: Combine the user's recent preferences with the item embedding to generate a list of recommended content, while considering the content priority, and adjust the model parameters by using the user's historical interactions to perform label data enhancement.

2. The Transformer-based cross-sequence multi-behavior sequence recommendation method according to claim 1, characterized in that: The segmentation of the user historical interaction sequence described in step 1 is as follows: The recommendation scenario of the present invention takes the e-commerce platform as an example. Behaviors that reflect user preferences, such as browsing and collecting, are called auxiliary behaviors, while behaviors that the platform hopes users will do to generate revenue for the platform, such as purchasing, are called target behaviors. Based on this scenario, the user set is written as U = {u1,…,u i ,…,u I }, the item set is written as V = {v1,…,v j ,…v J }, the various types of interactive behaviors that appear in the scenario are defined as set B (indexed by b) containing all possible behavior types (e.g., browsing, marking as favorite, adding to cart, purchasing). where I, J∈Z represent the number of users and items respectively. In the context of the present invention, each user's historical interaction is recorded as a multi-behavior interaction record. Specifically, for a user u i ∈U, whose historical interaction information is defined as a triple (v i,x , b i,x , t i,x ), represents user u i At time x, b i,x The xth item of the type behavior interaction) constitutes user u i The multi-behavior interaction record is represented by S i ={(v i,1 , b i,1 , t i,1 ),…,(v i,x , b i,x , t i,x ),…,(v i,X , b i,X , t i,X )}. Where X represents the sequence length |S i |. In order to model the user's preferences that evolve over time in the short term, we consider segmenting the user's interaction records and encoding them with contextual information. Then, in order to enhance information fluidity and improve the diversity of the recommendation system, this method designs a cross-sequence information transmission component so that the items that the user has interacted with can obtain information from similar interaction records of other users. First, the user’s interaction sequence is segmented by the Seg(·) operation to obtain multiple subsequences. i and its interaction sequence S i , the Seg(·) operation can be expressed as formula (1): Where P (indexed by p) represents the number of user subsequences divided, b target Indicates the target behavior, and Respectively represent u i the lengths of the first and last subsequences, Indicates the maximum length of a subsequence. Specifically, the process integrates all user interaction sequences, and divides the user interaction sequence into several subsequences with the item where the user performs the target behavior (such as purchase) as the cutoff point. The reason for this is that before a user purchases an item, the items that the user interacts with through other behaviors have different degrees of similarity or correlation. Before a user purchases a product, the preferences reflected by the user interaction subsequence are relatively concentrated and stable. Obtaining the user's subsequence in this way helps to accurately capture the user's short-term preferences. If the user has not performed the target behavior, in order to analyze the user's short-term preferences, the maximum subsequence length is used. Segment the user's historical interaction sequence. In the global view, if the last item in the interaction subsequences segmented by all users is the same item interacted with by the target behavior, then the two subsequences are considered similar. For example, given user u i The kth subsequence of User i′ There are subsequences If yes Then the sequence with sequence Similar, recorded as Each subsequence can have multiple similar subsequences. Furthermore, the cross-sequence heterogeneous relationship propagation described in step 2 is specifically as follows: This method designs an embedding network to jointly integrate behavior type information and time signals as context information into the feature representation of items. First, following the mainstream recommendation paradigm, the initial embedding of items is j ∈R d The item embedding matrix M∈R generated by embedding the item set V into the same space |I|×d Get, related to the project id. where d represents the potential state dimension. Then, e b ∈R d Associated with the user behavior type, it indicates the bth type of behavior embedding. In addition, in order to give the model time perception, this method introduces a time encoding strategy in the context information encoding layer, that is, using formula (2) to map the scalar timestamp to the vector space: where τ(·) represents the function that maps timestamps to 2D vectors, and W t ∈R 2d×d represents the time transformation matrix, and the odd and even indices of the 2D vector are represented by 2l and 2l+1. In addition, in order to avoid the influence of large embedding values, use As a scaling factor, W t The role of the matrix is ​​to introduce an adjustable linear projection mechanism, giving the model the ability to flexibly learn temporal context information. After that, the behavior type representation and temporal context information are jointly injected into the item representation by the following operations: in Represents item embeddings that incorporate contextual information and serves as input to the cross-sequence information transfer component, Represents the addition operation of vector elements. In order to capture the dynamic similarity relationship between different items in the sequence view and the global view, this method designs a cross-sequence information propagation encoder based on the attention mechanism in the multi-head representation space and built on the Transformer architecture. Specifically, the embedding of the xth interaction term in the sequence is calculated by the multi-head dot product attention method as shown in formula (4): in represents the user u calculated after cross-sequence information aggregation i The refined representation of the xth interaction term under the kth subsequence, Y represents the item The set of related items (i.e., the sequence Other items and sequences in ), E y represents the embedding representation of the relevant terms, α x,y Represents the items learned and v y The correlation coefficient between them, σ(·) represents the softmax function, h∈H represents the hth head of the attention layer, Q h , K h , Represent the Query, Key and Value transformation matrices of the h-th head (three important parameter matrices in the attention mechanism), and divide the scores of the query and value vectors by to stabilize the gradient. The introduction of the multi-head attention mechanism enables the encoder to perform information aggregation of different semantic dimensions in multiple representation subspaces. The cross-sequence heterogeneous relationship propagation module integrates the context information encoding layer and the cross-sequence information propagation layer, which not only gives the model sensitivity to the user interaction context information, but also captures the dynamic characteristics of multiple behaviors and preserves the transition relationship between items in the sequence view and the global view. Furthermore, the short-term interaction mode learning described in step 3 is specifically as follows: This method designs a behavior-aware encoder to aggregate specific types of behavior patterns in the short term, giving the model the ability to distinguish the heterogeneity of behaviors within a sequence. i with sequence u i The behavior pattern of type b in the corresponding short term can be formally expressed as formula (5): in Indicates that the aggregation calculates u i exist The characteristic representation of the behavior of b in the period, ρ(·) is the judgment function defined by this method. If the two input items are equal, the result is 1, otherwise it is 0. is the item-refined embedding representation output by the cross-sequence heterogeneous relation propagation module, W b is a behavior-aware transformation matrix used to enhance the semantics of a specific type of behavior. Its optimization path is based on the multi-channel parameter learning strategy shown in formula (6): Where M represents the number of channels (indexed by m), P∈R M×d and μ∈R M Represent the conversion matrix and bias parameters in the fully connected layer, respectively. Represents the transformation matrix corresponding to the mth channel. The purpose of introducing the multi-channel parameter learning strategy is to preserve the heterogeneity of interactive behaviors during message transmission, and use the channel-based aggregation layer to encode the behavior-aware semantics, so that the CMB model can capture the potential semantics of different types of user behaviors. After obtaining the embedding representation of a specific behavior type, the short-term behavior type encoding needs to be fed to the aggregation layer to capture the potential interdependencies between different types of behaviors and obtain the short-term feature representation of the user. Different from the static attention relationship aggregation scheme adopted by other recommendation models, this method uses an adaptive attention network to calculate the contribution index of each behavior type to the short-term representation. The aggregation layer operation can be formalized as the following formula: in Is user u i The characteristic representation of the kth subsequence is u i The short-term preference encoding in the corresponding period, β i,b represents the sum of all behavior embeddings as the query learned u i The attention weight corresponding to action b, W A and μ A are the transformation matrix and bias of the aggregation layer respectively. After the aggregation layer, user short-term characterization The time and behavior-aware context information is explicitly preserved in the Correspondingly, for the message transmission process from the user to the project side, in order to refine the behavior-aware embedding of the project, the embedding encoding of the project also performs the above-mentioned propagation and aggregation process, as shown in formula (8): in and E j Respectively represent project v j Specific behavior type embedding and fused behavior-aware embedding. This scheme can distinguish the impact of user behavior on items and incorporate time signals in the message transmission process, because the user subsequence feature encoding is involved in the calculation during the propagation process. Furthermore, the global representation aggregation described in step 4 is specifically as follows: The user's preference is essentially dynamic over time, so the CMB model further proposes a global graph relation encoder to inject the user's embedding representations of each period into the global feature representation, capturing the user's multi-behavior dynamic patterns from a long-term and short-term perspective. This method defines a global user graph to include all user nodes u i ∈U and their interacting subsequence nodes That is, G = {U, S, ε}, where S and ε represent the set of all user subsequences and the sequence connecting each user u, respectively. i Interaction subsequence node with him The global representation aggregation module performs graph structure information aggregation on the global user graph G to generate the global embedding representation of the user, and incorporates the temporal context information in the information aggregation process, so that the short-term representations of different periods interact with each other in a distinguishable way. The calculation process is shown in formula (9): in Represents user u i The global feature representation, σ(·) represents the softmax activation function, η k Represents the learned user u i With the attention weight of its k-th subsequence, is the time code returned by the context information encoding layer, corresponding to the sequence The time when the last interaction of the item in occurred. By doing so, CMB preserves user preference embeddings across epochs and incorporates them into a unified global user feature embedding representation. After the above process, the CMB framework performs information aggregation and encodes the user's short-term interaction behavior pattern and long-term dynamic user embedding representation. In the embedding propagation process, CMB uses the user-item interaction graph and the global user graph G as the calculation graph. During this period, the local relationship features from the neighborhood will be aggregated to obtain the contextual representation. Such a message passing paradigm can be summarized as formula (10) by incorporating high-order connectivity (l represents the l-th layer of the graph neural network in the graph-based message propagation process): in and They are the project and user embeddings initialized in the encoding layer. By extending the L-layer graph relation encoder, CMB is given the ability to capture high-order neighbor collaboration signals. Finally, the final user is obtained by adding the corresponding elements of the L-layer embedding. project and subsequences Feature representation, the process can be formulated as (11): Furthermore, the recommended content list is generated by combining the user's recent preferences and item embeddings as described in step 5, while taking content priority into consideration, and adjusting model parameters by using user historical interactions to enhance label data, as follows: In the prediction layer of the model, in order to reflect the dynamic characteristics of user preferences, this method jointly uses the short-term feature representation of the user in the last stage With the project feature representation E j Predict the user's possible future interaction items. i and project v j The probability of an interaction occurring can be quantified as in is the parameter vector, represents the element-by-element multiplication operation. During the training phase, the parameter adjustment process is optimized by using the user's historical interactions to enhance the label data, so u i and v j exist The prediction score for the corresponding period can be calculated as: In this case, the objective function of model training and the marginal pairwise loss can be formalized as follows: Where I and K i Represent the number of users and user u respectively i The number of interaction subsequences, C represents the number of samples (indexed by c), p c and n c Represent the corresponding positive samples and negative samples respectively. This method further introduces a weight decay regularization term To alleviate the overfitting phenomenon.

Images