Session recommendation method based on cross-dependence attention enhancement

Abstract

The invention discloses a session recommendation method based on cross-dependence attention enhancement, and belongs to the technical field of session recommendation. The method mainly solves the problem of insufficient article association extraction in a traditional session recommendation system, especially the problem of interest drift easily occurring during long sequence processing. The method comprises the following steps: firstly, initializing high-dimensional semantic representation and high-dimensional position representation of each article in a user session sequence, and splicing to obtain a high-dimensional representation matrix of the articles; secondly, through the self-attention network and the GRU, the object high-dimensional representation based on long-term dependence and the object high-dimensional representation based on short-term dependence are obtained respectively; and then, fusing the two article representations through a cross attention network combining long and short term dependencies to obtain a bidirectionally enhanced cross-dependent article high-dimensional representation. And finally, intercepting a vector at the last moment as user interest representation at the next moment, and predicting the probability that all articles in the article dictionary are clicked at the next moment.

Description

Technical Field

[0001] The present invention belongs to the technical field of conversation recommendation, and in particular relates to a conversation recommendation method based on cross-dependency attention enhancement. Background Art

[0002] Recommender system technology is one of the fastest growing artificial intelligence applications in recent years. It has become an important tool for various user management or e-commerce systems, which can greatly improve the user experience of the system. However, in order to protect the privacy of users, in many cases, most of the information on the user side and the item side is not available, which makes the research of recommendation systems face challenges. To solve this problem, the research task of conversational recommendation system came into being. This task only uses implicit feedback such as user clicks to predict the next click of anonymous users.

[0003] In recent years, models based on recurrent neural networks (RNNs) and self-attention mechanisms have achieved good results in improving recommendation performance in the field of conversational recommendation by exploring short-term and long-term dependencies between items respectively. However, this paper believes that the above two methods have the problem of insufficient item association extraction when dealing with user behavior sequences, especially long sequences that are more prone to interest drift, which leads to suboptimal solutions for understanding user behavior preferences. Specifically, in longer user-item click sequences, RNN-based conversational recommendation methods are prone to long-distance forgetting, while methods based on self-attention mechanisms can handle this limitation well by calculating the similarity between item pairs. At the same time, methods based on self-attention mechanisms explicitly consider long-term dependencies centered on each item, and introduce those weakly related items that exist in long sequences due to interest drift, such as items triggered by curiosity or unintentional triggering, which in turn damages the accuracy of user preference understanding, while RNN-based conversational recommendation methods can alleviate this problem by focusing on short-term interests. In order to solve the above problems, this paper proposes a conversational recommendation method based on cross-dependency attention enhancement by giving full play to the advantages of self-attention and RNN. Summary of the invention

[0004] The present invention aims to more accurately capture the short-term and long-term dependencies in user behavior sequences by combining the advantages of self-attention and recurrent neural networks (GRU), and to achieve simultaneous modeling of short-term and long-term dependencies between items in the sequence.

[0005] This application uses a self-attention mechanism with position embedding and a gated recurrent unit (GRU) to capture short-term and long-term dependencies, respectively. Then, this application designs a dual cross-dependency attention network, including a cross-dependency attention network with short-term and long-term priority, to enrich the conversion information between items and improve the representation of items. The short-term priority cross-dependency attention network extracts short-term dependencies from a global perspective to enhance the local dependencies between items. The long-term priority cross-dependency attention network extracts long-term dependencies from a local perspective to enhance the global dependencies between items. Finally, this application uses linear combinations to represent the sequence as a weighted combination of items after the short-term and long-term priority cross-dependency attention networks, thereby realizing the prediction of the user's next item of interest.

[0006] The present invention provides a conversation recommendation method based on cross-dependency attention enhancement, comprising the following steps:

[0007] Step 1. Initialize the high-dimensional semantic representation and high-dimensional position representation of the items in the user-item click sequence, and obtain the high-dimensional representation of the items and the user-item click sequence representation matrix through concatenation operation;

[0008] Step 2. Input the item representation matrix of the user-item click sequence obtained in the previous step into the self-attention mechanism network, the feedforward network, and the temporal network GRU respectively, and generate an item representation matrix containing long-term dependency context information and an item representation matrix containing short-term dependency respectively;

[0009] Step 3. A cross-dependency attention model is designed. The long-term dependency item representation matrix and the short-term dependency item representation matrix obtained in step 2 are used as the Qurey, key, and value of the cross-dependency attention for representation learning. Finally, the output enhanced by the cross-dependency attention model is used to obtain the user item click sequence representation;

[0010] Step 4. Based on the user item click sequence representation obtained in step 3, do a dot product operation with the item dictionary to form the probability value of each item becoming the next item. Then, in descending order, recommend the top ten or top twenty items with the highest probability value to the user as the recommendation list recommended to the user at the next moment.

[0011] In the above scheme, the specific process of step 1 is:

[0012] Step 1-1. The item dictionary is represented as V = {v1, v2, ..., v |v|}, where v j represents the jth item in the item dictionary, 1≤j≤|V|, |V| is the size of the item dictionary. The high-dimensional semantic representation of the item dictionary is X={x1, x2, ..., x |v|}, where x jrepresents the high-dimensional semantic representation of the jth item in the item dictionary, represents a d-dimensional vector, where d represents the dimension of the hidden layer; the high-dimensional position of each position sequence item in the user item click sequence is represented as P = {p1, p2, ..., p |maxlen|}, maxlen indicates the maximum length of the sequence, Represents the high-dimensional position representation of the item clicked by the user at the i-th moment in the sequence;

[0013] User item click sequence S = {s1, s2, ..., s t}, where s i It represents the sequence number of the item clicked by the user at the i-th moment in the item dictionary V, 1≤i≤t, t<maxlen.

[0014] Step 1-2. Sequence the session into i The high-dimensional semantic representation and high-dimensional position representation of the corresponding items are denoted as x i ∈X,p i ∈P. Represent the high-dimensional semantics x i and the high-dimensional position representation p i Perform splicing operation to get s i The corresponding high-dimensional representation c i And the user session sequence item representation matrix C. Use concat concatenation operation:

[0015] c i =Concat(x i , p i )

[0016] C = {c1, c2, ..., c t}

[0017] In the above scheme, the specific process of step 2 is:

[0018] Step 2-1. Input the item representation matrix C of the user-item click sequence obtained in step 1 into the self-attention network to obtain the item representation matrix based on long-term dependency:

[0019]

[0020] where C′=[C′1,C′2,...,C′ t ], are all learnable weight parameters, softmax is the activation function, is the scale factor.

[0021] Then two linear transformations and a ReLU activation function are applied to enhance the nonlinear capability of the model, and a residual connection is introduced after the feedforward network to facilitate the model's utilization of underlying information.

[0022]

[0023] in, is the learnable weight parameter, is the learnable bias vector.

[0024] The dimension is kept consistent with the hidden layer state of GRU through linear transformation, where is a learnable variable.

[0025]

[0026] That is, the item representation matrix based on long-term dependencies learned through the self-attention network.

[0027] Step 2-2. Pass the item representation matrix of the user item click sequence obtained in step 1 through the GRU model, combining the current and past time sequence information to obtain a high-dimensional representation of the item. Specifically, the item representation h at time step i is i , is the activation h at the previous moment i-1 and candidate activation A linear combination of .

[0028]

[0029] z i =σ(W z x i +U z h i-1 )

[0030]

[0031] r i =σ(W r x i +U r h i-1 ) Among them, the item representation matrix in the user item click sequence is H = [h1, h2, ..., h t ], are all learnable weight parameters, z i and r i They represent the update gate and the forget gate respectively. H is the item representation matrix based on short-term dependencies learned after GRU.

[0032] In the above scheme, the specific process of step 3 is:

[0033] Step 3-1. Based on the item representation based on long-term dependency and the item representation based on short-term dependency obtained in step 2, this paper designs a new cross-dependency attention network to improve the understanding of user behavior preferences, such as Figure 2 As shown. The representation of the cross-dependency attention network is as follows:

[0034]

[0035] M=ReLU(M′W3+b3)W4+b4+Q

[0036] in is the learnable weight parameter is the learnable bias vector;

[0037] This application simply abbreviates the cross-dependency attention network unit as M=CrossAttention(Q, K, V).

[0038] The enhanced criss-cross attention model based on long-term dependency can be expressed as:

[0039]

[0040] The enhanced criss-cross attention model based on short-term dependency can be expressed as:

[0041]

[0042] Step 3-2. The M obtained above ce and M se Fusion, we get a cross-dependency attention model based on bidirectional complementary enhancement, which is expressed as:

[0043]

[0044] in, is a learnable weight parameter. This application only uses the last vector as the user item click sequence to represent Z = M de t .

[0045] In the above scheme, the specific process of step 4 is:

[0046] Based on the sequence representation Z = M de t , predict all items in the item dictionary and get the probability value of each candidate item:

[0047]

[0048] in, Represents the probability value of the i-th item in the item dictionary;

[0049] w k is the normalized weight, and L2Norm is the L2 normalization function.

[0050] The items with the top 20 or top 10 probability values ​​are taken as the recommendation results for the next moment in the current user session.

[0051] This paper proposes a conversational recommendation method based on cross-dependency attention enhancement, which mainly solves the problem of insufficient item association extraction in traditional conversational recommendation systems, especially the problem of interest drift that is prone to occur when processing long sequences. This method combines the advantages of self-attention and recurrent neural network (GRU) to more accurately capture the short-term and long-term dependencies in user behavior sequences, and achieves simultaneous modeling of short-term and long-term dependencies between items in the sequence. The specific advantages are as follows:

[0052] 1. This application initializes the high-dimensional semantic representation and high-dimensional position representation of items in the user-item click sequence. This application: obtains the high-dimensional representation of the item and the user-item click sequence representation matrix through splicing operations, which can better capture the semantic and position information of the item.

[0053] 2. Application of the self-attention mechanism network and GRU model of this application This application: Input the item representation matrix of the user item click sequence into the self-attention mechanism network and GRU respectively, and generate an item representation matrix containing long-term dependent context information and an item representation matrix containing short-term dependencies respectively. The self-attention mechanism network handles the interest drift problem in long sequences by calculating the similarity between item pairs, while GRU focuses on short-term interests. The combination of the two can more comprehensively understand user behavior.

[0054] 3. Design of the cross-dependency attention model of this application This application: designs a new cross-dependency attention network, uses the long-term dependency item representation matrix and the short-term dependency item representation matrix obtained in step 2 as the query, key, and value of the cross-dependency attention for representation learning, and finally obtains the user item click sequence representation by the output enhanced by the cross-dependency attention model. This network can consider the long-term and short-term dependencies of items at the same time, and improve the understanding of user behavior preferences.

[0055] 4. Generation of recommendation results of this application This application: Based on the user-item click sequence representation obtained in step 3, a dot product operation is performed with the item dictionary to form the probability value of each item becoming the next item, and then the items with the top ten or top twenty probability values ​​are recommended to the user in descending order as the recommendation list recommended to the user at the next moment.

[0056] Through these steps, the present invention can more accurately capture the dependencies in the user behavior sequence and improve the performance of the recommendation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the process flow of the present invention;

[0058] Figure 2 It is the overall model of the network;

[0059] Figure 3 There are three types of cross-attention networks. DETAILED DESCRIPTION

[0060] The following is a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, modifications or equivalent substitutions made to the present invention should all be included in the scope of the claims of the present invention.

[0061] In addition, in order to better illustrate the present invention, numerous specific details are given in the following specific embodiments. It will be understood by those skilled in the art that the present invention can also be implemented without these specific details.

[0062] The present invention provides a conversation recommendation method based on cross-dependency attention enhancement, comprising the following steps:

[0063] Step 1. Initialize the high-dimensional semantic representation and high-dimensional position representation of the items in the user-item click sequence, and obtain the high-dimensional representation of the items and the user-item click sequence representation matrix through concatenation operation;

[0064] Step 2. Input the item representation matrix of the user-item click sequence obtained in the previous step into the self-attention mechanism network, the feedforward network, and the temporal network GRU respectively, and generate an item representation matrix containing long-term dependency context information and an item representation matrix containing short-term dependency respectively;

[0065] Step 3. A cross-dependency attention model is designed. The long-term dependency item representation matrix and the short-term dependency item representation matrix obtained in step 2 are used as the Qurey, key, and value of the cross-dependency attention for representation learning. Finally, the output enhanced by the cross-dependency attention model is used to obtain the user item click sequence representation;

[0066] Step 4. Based on the user item click sequence representation obtained in step 3, do a dot product operation with the item dictionary to form the probability value of each item becoming the next item. Then, in descending order, recommend the top ten or top twenty items with the highest probability value to the user as the recommendation list recommended to the user at the next moment.

[0067] In the above scheme, the specific process of step 1 is:

[0068] Step 1-1. The item dictionary is represented as V = {v1, v2, ..., v |v|}, where v j represents the jth item in the item dictionary, 1≤j≤|V|, |V| is the size of the item dictionary. The high-dimensional semantic representation of the item dictionary is X={x1, x2, ..., x |v|}, where x j represents the high-dimensional semantic representation of the jth item in the item dictionary, represents a d-dimensional vector, where d represents the dimension of the hidden layer; the high-dimensional position of each position sequence item in the user item click sequence is represented as P = {p1, p2, ..., p |maxlen|}, maxlen indicates the maximum length of the sequence, Represents the high-dimensional position representation of the item clicked by the user at the i-th moment in the sequence;

[0069] User item click sequence S = {s1, s2, ..., s t}, where s i represents the sequence number of the item clicked by the user at the i-th moment in the item dictionary V, 1≤i≤t,t <maxlen。

[0070] Step 1-2. Sequence the session into i The high-dimensional semantic representation and high-dimensional position representation of the corresponding items are denoted as x i ∈X,p i ∈P. Represent the high-dimensional semantics x i and the high-dimensional position representation p i Perform splicing operation to get s i The corresponding high-dimensional representation c i And the user session sequence item representation matrix C. Use concat concatenation operation:

[0071] c i =Concat(x i , p i )

[0072] C = {c1, c2, ..., c t}

[0073] In the above scheme, the specific process of step 2 is:

[0074] Step 2-1. Input the item representation matrix C of the user-item click sequence obtained in step 1 into the self-attention network to obtain the item representation matrix based on long-term dependency:

[0075]

[0076] where C′=[C′1,C′2,...,C′ t ], are all learnable weight parameters, softmax is the activation function, is the scale factor.

[0077] Then two linear transformations and a ReLU activation function are applied to enhance the nonlinear capability of the model, and a residual connection is introduced after the feedforward network to facilitate the model's utilization of underlying information.

[0078]

[0079] in, is the learnable weight parameter, is the learnable bias vector.

[0080] The dimension is kept consistent with the hidden layer state of GRU through linear transformation, where is a learnable variable.

[0081]

[0082] That is, the item representation matrix based on long-term dependencies learned through the self-attention network.

[0083] Step 2-2. Pass the item representation matrix of the user item click sequence obtained in step 1 through the GRU model, combining the current and past time sequence information to obtain a high-dimensional representation of the item. Specifically, the item representation h at time step i is i , is the activation h at the previous moment i-1 and candidate activation A linear combination of .

[0084]

[0085] z i =σ(W z x i +U z h i-1 )

[0086]

[0087] r i =σ(W r x i +U r h i-1 )

[0088] Among them, the high-dimensional representation of items in the user item click sequence is H = [h1, h2, ..., h t ], are all learnable weight parameters, z i and r i They represent the update gate and the forget gate respectively. H is the item representation matrix based on short-term dependencies learned after GRU.

[0089] In the above scheme, the specific process of step 3 is:

[0090] Step 3-1. Based on the item representation based on long-term dependency and the item representation based on short-term dependency obtained in step 2, this paper designs a new cross-dependency attention network to improve the understanding of user behavior preferences, such as Figure 2 As shown. The representation of the cross-dependency attention network is as follows:

[0091]

[0092] M=ReLU(M′W3+b3)W4+b4+Q

[0093] in is the learnable weight parameter is the learnable bias vector;

[0094] This application simply abbreviates the cross-dependency attention network unit as M=CrossAttention(Q, K, V).

[0095] The enhanced criss-cross attention model based on long-term dependency can be expressed as:

[0096]

[0097] The enhanced criss-cross attention model based on short-term dependency can be expressed as:

[0098]

[0099] Step 3-2. The M obtained above ce and M se Fusion, we get a cross-dependency attention model based on bidirectional complementary enhancement, which is expressed as:

[0100]

[0101] in, is a learnable weight parameter. This application only uses the last vector as the user item click sequence to represent Z = M de t .

[0102] In the above scheme, the specific process of step 4 is:

[0103] Based on the sequence representation Z = M de t , predict all items in the item dictionary and get the probability value of each candidate item:

[0104]

[0105] in, Represents the probability value of the i-th item in the item dictionary;

[0106] w k is the normalized weight, and L2Norm is the L2 normalization function.

[0107] The items with the top 20 or top 10 probability values ​​are taken as the recommendation results for the next moment in the current user session.

Claims

1. A conversation recommendation method based on cross-dependency attention enhancement, characterized in that: The following steps are involved: Step 1. Initialize the high-dimensional semantic representation and high-dimensional position representation of the items in the user-item click sequence, and obtain the high-dimensional representation of the items and the user-item click sequence representation matrix through concatenation operation; Step 2. Input the item representation matrix of the user-item click sequence obtained in the previous step into the self-attention mechanism network, the feedforward network, and the temporal network GRU respectively, and generate an item representation matrix containing long-term dependency context information and an item representation matrix containing short-term dependency respectively; Step 3. A cross-dependency attention model is designed. The long-term dependency item representation matrix and the short-term dependency item representation matrix obtained in step 2 are used as the Qurey, key, and value of the cross-dependency attention for representation learning. Finally, the output enhanced by the cross-dependency attention model is used to obtain the user item click sequence representation; Step 4. Based on the user item click sequence representation obtained in step 3, do a dot product operation with the item dictionary to form the probability value of each item becoming the next item. Then, in descending order, recommend the top ten or top twenty items with the highest probability value to the user as the recommendation list recommended to the user at the next moment.

2. According to the conversation recommendation method based on cross-dependency attention enhancement according to claim 1, it is characterized in that: The specific process of step 1 is: Step 1-1. The item dictionary is represented as V = {v1, v2, ..., v|v | }, where v j represents the jth item in the item dictionary, 1≤j≤|V|, |V| is the size of the item dictionary, and the high-dimensional semantic representation of the item dictionary is X={x1, x2, .., x |v| }, where x j represents the high-dimensional semantic representation of the jth item in the item dictionary, represents a d-dimensional vector, where d represents the dimension of the hidden layer; the high-dimensional position of each position sequence item in the user item click sequence is represented as P = {p1, p2, ..., p |maxlen| }, maxlen indicates the maximum length of the sequence, Represents the high-dimensional position representation of the item clicked by the user at the i-th moment in the sequence; User item click sequence S = {s1, s2, ..., s t }, where s i represents the sequence number of the item clicked by the user at the i-th moment in the item dictionary V, 1≤i≤t,t <maxlen; Step 1-2. Add the user item click sequence to s i The high-dimensional semantic representation and high-dimensional position representation of the corresponding items are denoted as x i ∈X,p i ∈P, represents the high-dimensional semantics x i and the high-dimensional position representation p i Perform splicing operation to get s i The corresponding high-dimensional representation c i And the item representation matrix C of the user-item click sequence, using the concat operation: c i =Concat(x i ,p i ) C={c1,c2,...,c t }。 3. The conversation recommendation method based on cross-dependency attention enhancement according to claim 1, characterized in that: The specific process of step 2 is: Step 2-1. Input the item representation matrix C of the user-item click sequence obtained in step 1 into the self-attention network to obtain the item representation matrix C based on long-term dependency: where C′=[C′1,C′2,...,C′ t ], are all learnable weight parameters, softmax is the activation function, is the scale factor; Two linear transformations and a ReLU activation function are applied to enhance the nonlinear ability of the model, and a residual connection is introduced after the feedforward network to facilitate the model's use of underlying information: in, is the learnable weight parameter, is the learnable bias vector; The dimension is kept consistent with the hidden layer state of GRU through linear transformation, where is a learnable variable; That is, the item representation matrix based on long-term dependency learned through the self-attention network; Step 2-2. Pass the item representation matrix of the user item click sequence obtained in step 1 through the GRU model, combining the current and past time sequence information to obtain a high-dimensional representation of the item. Specifically, the item representation h at time step i i , is the activation h at the previous moment i-1 and candidate activation A linear combination of: z i =σ(W z x i +U z h i-1 ) r i =σ(W r x i +U r h i-1 ) Among them, the item representation matrix in the user item click sequence is H = [h1, h2, ..., h t ],W z , U z , W, U, W r , are all learnable weight parameters, z i and r i They represent the update gate and the forget gate respectively, and H is the item representation matrix based on short-term dependencies learned after GRU.

4. The conversation recommendation method based on cross-dependency attention enhancement according to claim 1, characterized in that: The specific process of step 3 is: Step 3-1. Based on the item representation based on long-term dependency and the item representation based on short-term dependency obtained in step 2, a new cross-dependency attention network is used to improve the understanding of user behavior preferences. The representation of the cross-dependency attention network is as follows: M=ReLU(M′W3+b3)W4+b4+Q Where W QC , W KC , W VC 、W3、 is the learnable weight parameter is the learnable bias vector; The cross-dependency attention network unit is abbreviated as M = CrossAttention (Q, K, V), where The enhanced criss-cross attention model based on long-term dependency is expressed as: The enhanced criss-cross attention model based on short-term dependency is expressed as: Step 3-2. The M obtained above ce and M se Fusion, we get a cross-dependency attention model based on bidirectional complementary enhancement, which is expressed as: in, is a learnable weight parameter, and the last vector is represented as the user-item click sequence Z = M de t .

5. According to the conversation recommendation method based on cross-dependency attention enhancement as described in claim 1, it is characterized in that: The specific process of step 4 is: Based on the sequence representation Z = M de t , predict all items in the item dictionary and get the probability value of each candidate item: in, Represents the probability value of the i-th item in the item dictionary; w k is the normalized weight, L2Norm is the L2 normalization function; The items with the top 20 or top 10 probability values ​​are taken as the recommendation results for the next moment in the current user session.

Images