A conversation recommendation method based on dynamic attention mechanism

Abstract

This invention discloses a conversation recommendation method based on a dynamic attention mechanism, comprising the following steps: S1: inputting a conversation sequence; S2: extracting an interest sequence from continuous user behavior using a dynamic attention mechanism to model the user's deep intent; S3: evaluating the probability of each candidate item based on the user's deep intent and making a recommendation. This invention proposes a conversation recommendation method based on a dynamic attention mechanism to alleviate the problem of item irrelevance. Through the dynamic attention mechanism, the hidden representation of each item is modeled based on all items preceding it, i.e., using the current item and its preceding items to model the current hidden interest, representing the next preferred option at the time of the current click. In this way, dynamic interests can be tracked with each click, thereby improving recommendation accuracy and achieving higher accuracy compared to existing technologies.

Description

Technical Field

[0001] This invention relates to the field of computer systems and related technologies, and in particular to a session recommendation method based on a dynamic attention mechanism. Background Technology

[0002] Recommending the next item based on a user's current interests is crucial for online platforms. Most existing attention-based methods aim to model a user's primary purpose using the entire session, often in combination with other neural network architectures such as recurrent neural networks (RNNs) and graph neural networks (GNNs). However, most of them, to some extent, ignore the fact that items are not strictly ordered and that using the entire session to model user interests neglects capturing intent at each click.

[0003] SR: Session-based recommendation

[0004] SR (Surveyor Detection) aims to predict what a user will click next based on their current session. The SR task is formally defined as follows: Let I = {i1, i2, ..., i...} N} represents a set of items, where N is the total number of items. Each session is represented as a list sorted by timestamp. Where i s，k ∈I (1≤k≤m) represents a session The user's clicked item.

[0005] SR mission is prediction The next click i s，m+1 For the conversation The model calculates the probability of all possible items. The project's recommendation score is a vector. The corresponding element will be recommended and The items corresponding to the top-K scores in the data.

[0006] Most existing SR research treats a session as an ordered sequence, where items clicked in chronological order are elements within that sequence. Graph Neural Networks (GNNs), summarized in many recurrent neural and attention-based approaches, have demonstrated excellent performance. RNN-based methods model items in a session as a time-dependent sequence. This is considered key to their success. However, they neglect the coherence of items, as items are not strictly ordered sequences in real-world scenarios. For example, a user might play albums sequentially or randomly, resulting in two distinct records. Attention-based models aim to differentiate between items to capture the main purpose of the session, but they focus on the entire session while ignoring the local relationships between items. That is, existing attention-based SR techniques model user intent using the entire session, neglecting to capture the intent of each click. While Convolutional Neural Networks (CNNs) can capture the local relationships between items, they still model the session as a time-dependent sequence, impacting prediction accuracy. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing technologies that model conversations as time-dependent sequences, which affects prediction accuracy, and to provide a conversation recommendation method based on a dynamic attention mechanism.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A session recommendation method based on dynamic attention mechanism includes the following steps:

[0010] S1: Input session sequence;

[0011] S2: Extract interest sequences from continuous user behavior through dynamic attention mechanism to model the user's deep-seated intentions;

[0012] S3: Evaluate the probability of each candidate item based on the user's deep intent and make recommendations.

[0013] In some embodiments, step S2 includes the following steps:

[0014] S21: The input session sequence is transformed into a high-dimensional vector through the embedding layer to obtain the embedding vector of each item;

[0015] S22: Extract hidden user interest sequences from continuous user behavior through the interest tracking layer;

[0016] S23: The interest enhancement layer models the user's deeper intent based on the extracted interest sequence.

[0017] In some embodiments, step S22 includes the following steps:

[0018] S221: Arrange the embedded vector X = {x1, x2, ..., x...} in chronological order. m Cut into m slices;

[0019] S222: For each slice X′ t ={x1, x2, ..., x t The first multi-head attention network is used to extract hidden user interest sequences.

[0020] Where X is a high-dimensional vector sequence, and t≤m represents the sequence number.

[0021] In some embodiments, the expression formula for extracting the user's potential interests using a first multi-head attention network in step S222 is as follows:

[0022] Q x =f(W q x t +b q ),

[0023] l′ t =MultiHead(Q x , X′ t , X′ t ),

[0024] in It is a weight matrix. is the bias vector, and f(·) is the activation function ReLU.

[0025] In some embodiments, step S222 ensures that the extraction of hidden user interests for the t-th item can only depend on its previous items by using a mask matrix in the first multi-head attention.

[0026] In some embodiments, step S22 further includes the following steps:

[0027] S223: Applying a positional feedforward neural network gives the model more nonlinearity, expressed in the following formula:

[0028] FFN(L′)=max(0,L′W3+b1)W4+b2,

[0029] Among them, W3, It is a weight matrix, b1, It is the deviation vector.

[0030] In some embodiments, the interest enhancement layer in step S23 employs a second multi-head attention network to learn global information hidden behind the evolution of user interests, expressed as follows:

[0031] O global =MulitiHead(Q l ,L,L)

[0032] Q l It is the last element l in L m O global It represents global information based on user interest sequences.

[0033] In some embodiments, the first multi-head attention network and the second multi-head attention network are the same multi-head attention network.

[0034] In some embodiments, step S3 includes the following steps:

[0035] S31: Combine the embedding vector x of the last term with the last latent interest embedding l output by the interest tracking layer. m By connecting these connections, a representation of the local information is obtained at the end of the session using the first feedforward neural network.

[0036] The formula for step S31 is as follows:

[0037] O local =f(W local ([l m ||x m ])+b local ),

[0038] in, It is a weight matrix. It is the bias vector, and f(·) represents the nonlinear activation function SELU. Represents local information;

[0039] S32: O local and the output O of the interest enhancement layer global By connecting these connections, a second feedforward neural network is applied to obtain a representation of the user's potential interests.

[0040] The formula for step S32 is as follows:

[0041] O = f(W) out ([O global ||O local ])+b out ),

[0042] in, It is a weight matrix. It is the bias vector, and f(·) represents the nonlinear activation function SELU;

[0043] S33: Calculate the probability of each candidate item and make a recommendation.

[0044] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the steps of the above-described method.

[0045] The present invention has the following beneficial effects:

[0046] This invention proposes a session recommendation method based on a dynamic attention mechanism to alleviate the problem of item irrelevance. By using a dynamic attention mechanism, the hidden representation of each item is modeled based on all previous items. In other words, the dynamic attention mechanism extracts interest sequences from continuous user behavior, modeling the user's deep-seated intent. In this way, dynamic interests can be tracked with each click, resulting in better recommendations and higher accuracy compared to existing technologies. Attached Figure Description

[0047] Figure 1 This is a flowchart of the conversation recommendation method based on dynamic attention mechanism in an embodiment of the present invention;

[0048] Figure 2 This is a modeling diagram of the conversation recommendation method based on dynamic attention mechanism in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of a conversation recommendation method based on a dynamic attention mechanism in an embodiment of the present invention. Detailed Implementation

[0050] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0051] refer to Figure 1 , Figure 2 To address the issue that items are not strictly arranged in order, this invention proposes a session recommendation method based on a dynamic attention mechanism, such as... Figure 3 As shown, the recommendation method in this embodiment of the invention uses a novel Dynamic Attention Enhanced Interest Network (DAIN) model to simulate user interests and thus capture the user's intent for each click. The basic idea of ​​this embodiment is to first utilize the decoder's attention mechanism to mine the user's hidden interest states to track changes in user interests, and then integrate all mined hidden interest states to obtain a deeper level of user intent. Finally, a representation of session preferences is obtained based on this information of user interest.

[0052] Specifically, the following steps are included:

[0053] S1: Input session sequence;

[0054] S2: Extract interest sequences from continuous user behavior through a dynamic attention mechanism to model the user's deeper intent.

[0055] S21: Transform the input session sequence into a high-dimensional vector using an embedding layer to obtain the embedding vector for each item: Construct an embedding layer that takes the input session s = [i s，1 i s，2 , ..., i s，m The vector is converted to a high-dimensional vector to obtain the embedding vector for each item. The model then uses a location embedding module to capture the temporal impact of the session. For each i-th item in the input session, the hidden representation is as shown in the equation:

[0056] x i =([e i ||p i ])

[0057] in It is the project embedding vector. It is the position embedding vector, d is the embedding size, and || denotes the concatenation operation. This represents the vector representation of item i.

[0058] Unlike some recommendation systems that have access to long-term user history, many e-commerce platforms lack sufficient information to understand user intent. To address this issue, this embodiment first utilizes an interest tracking layer to extract and track a series of hidden user interests from continuous user behavior, and then uses an interest enhancement layer to model the user's deeper intent based on the extracted interest sequence.

[0059] S22: Extracting Hidden User Interest Sequences from Continuous User Behavior via an Interest Tracking Layer: In SR, user behavior sequences are the carriers of latent interests, which typically change after a user takes an action. To better understand users' deeper intentions and utilize user behavior sequences, DAIN employs an interest tracking layer to track users' dynamic interests by extracting the interest state of each user action. Figure 2 The graphical model of the interest tracking layer is shown. This embodiment employs a multi-head attention structure using a Transformer decoder to mine hidden user interest states, balancing efficiency and performance. In terms of efficiency, this structure can generate hidden interest states in parallel using a special mask matrix, rather than computationally computing them sequentially like an RNN. In terms of performance, the attention mechanism allows for modeling dependencies regardless of their distance in the input or output sequences, and can selectively collect information from each node based on different weights. Furthermore, multi-head attention allows the model to collectively focus on information from different representation subspaces at different locations. The formula for the multi-head attention network used in this embodiment is as follows:

[0060] head i =Attention(QW i Q KW i K VW i V )

[0061] MultiHead(Q,K,V)=Concat(head1,...,head h )

[0062] in and It is the weight matrix, d k , is head i The dimension of the attention network is h, where h is the number of heads. The calculation method for the attention network is shown in the following equation:

[0063] a′ ij =W0f(QW1+KW2+b0)

[0064]

[0065] in W1, It is a weighted matrix, and max-len is the maximum length of the session. is the bias matrix, and f(·) is the activation function ReLU.

[0066] The present invention then describes how the interest tracking layer extracts the user's potential interests with each click and captures the changes in the user's interests in the behavioral sequence.

[0067] S221: Obtain the embedding vector sequence X = {x1, x2, ..., x...} from the embedding layer. m After representing the project, first slice it into m slices in chronological order:

[0068] X'1={x1}, X'2={x1, x2},..., X'm={x1, x2,...,x m}, for X′ t ={x1, x2, ..., x t For each slice like this, t≤m represents the sequence number, where X is a high-dimensional vector sequence.

[0069] S222: For each slice X′ t ={x1, x2, ..., x t The first multi-head attention network is used to extract hidden user interest sequences, as shown in the equation:

[0070] Qx =f(W q x t +b q ),

[0071] l′ t =MultiHead(Q x , X′ t , X′ t )

[0072] in It is a weight matrix. It is the bias vector, and f(·) is the activation function ReLU.

[0073] By performing the above process in parallel using a mask matrix in the first multi-head attention, it is ensured that the extraction of the t-th item depends only on its preceding items. Through the above process, the interest tracking layer outputs a series of hidden interest states for the session:

[0074] L′={l′1, l′2,..., l′ m}

[0075] S223: Applying a positional feedforward neural network (FFN) imparts more nonlinearity to the model, as shown in the equation:

[0076] FFN(L′)=max(0,L′W3+b1)W4+b2

[0077] Among them, W3, Both are weight matrices, b1, These are all bias vectors, and all sessions will share the same parameters. Residual connections and layer normalization are then added to the results to mitigate model training instability. The model further incorporates a Dropout mechanism to alleviate overfitting.

[0078] The interest tracking layer is represented as ITL, where L = ITL(X), L = {l1, l2, ..., l...} m It's about interest and...

[0079] The final output of the trace layer. Specifically, [l1, l2, ..., l...] m} are the potential interest states of m users extracted from a session containing m items, each of which collects information from all previous items during the generation process.

[0080] S23: An interest enhancement layer models the user's deeper intent based on the extracted interest sequence: As the user's session progresses on the e-commerce platform, their potential interests change, and this evolution directly influences their choice of the next item. Therefore, this embodiment designs an interest enhancement layer to deeply mine and analyze the evolution of user interests. The interest enhancement layer provides more relevant historical information to enhance the representation of the final interest and predicts changes based on user interest trends.

[0081] The output of the interest tracking layer is L = {l1, l2, ..., l...} m This can be viewed as a representation of the user's potential interests after each action taken during the session. To obtain the user's final intention for the next item, the same second multi-head attention network is directly used to learn the global information hidden behind the evolution of user interests, as shown in the equation:

[0082] O global =MulitiHead(Q l , L, L)

[0083] Q l It is the last element l in L m And O global It represents global information based on user interest sequences.

[0084] In summary, the query uses the user's last potential interest, and the entire list of potential interests serves as both the keyword and value. Therefore, this attention weight, based on known information about all user potential interests, captures the correlation between potential interests and the user's current interest preferences during a session.

[0085] After going through the above layers, we obtain the project's embedding representation X = {x1, x2, ..., x}. m The latent user interests are represented as L = {l1, l2, ..., l}. m} and user deep intent representation O global Calculate the probability of the next clicked item.

[0086] S3: Evaluate the probability of each candidate item based on the user's deep intent and make recommendations:

[0087] S31: As Figure 3 As shown, first, the last term x m Embedding and the last potential interest embedding l m Connect them together, and then apply the first feedforward neural network to obtain a representation of the local information at the end of the session, as shown in the following equation: O local =f(W local ([l m ||x m])+b local ),

[0088] in It is a weight matrix. It is the bias vector, and f(·) represents the nonlinear activation function SELU. Represents local information;

[0089] S32: O local and the output O of the interest enhancement layer global Connect them together and apply a second feedforward neural network to obtain the final output:

[0090] O = f(W) out ([O global ||O local ])+b out )

[0091] It is a weight matrix. It is the bias vector, and f(·) represents the nonlinear activation function SELU. This represents the final output.

[0092] S33: Calculate the probability of each candidate item and make a recommendation. For each item i∈I, the corresponding score is calculated as shown in the following formula:

[0093]

[0094]

[0095]

[0096] Where x i Let L2Norm be the embedding vector of item i, and L2Norm be the L2 regularization function. Let i represent the final probability of term i.

[0097] This embodiment models item consistency by capturing the user's interest at each click and uses dynamic interest to enhance the representation of session preferences. An attention mechanism is employed to simulate dynamic interest at each click, and then this attention mechanism is applied again to enhance the representation of session interests using dynamic interest. Therefore, this model is named the Dynamic Attention Enhanced Interest Network. Specifically, since treating the session as a time series may lead to inconsistent interest acquisition, and modeling the entire session for the primary purpose would lack local item transformations, an attention mechanism is first used to capture the interest of the prefix subsequence of the entire sequence. In this way, dynamic interest at each click can be obtained.

[0098] Table 1 shows the performance of all comparison methods on three SR datasets. The overall best results are shown in bold, and the underlined results represent the best results across all baselines. Compared to existing SR models, the method of this embodiment significantly outperforms NARM, STAMP, SR-GNN, and FGNN, demonstrating that the accuracy of the method in this embodiment is higher than that of existing technologies.

[0099] Table 1

[0100]

[0101] in:

[0102] FPMC (Rendle, Freudenthaler, and Schmidt-Thieme 2010) is a traditional sequential method based on Markov chains;

[0103] GRU4REC (Hidasi et al. 2015) is a representative sequence model based on RNN;

[0104] NARM (Li et al. 2017) is an RNN-based model that uses an attention mechanism to capture the main purpose of a conversation;

[0105] STAMP (Liu et al. 2018) uses a self-attention mechanism to represent the intent of a conversation;

[0106] SR-GNN (Wu et al. 2019a) uses a gated graph convolutional neural network to model project transformations;

[0107] FGNN (Qiu et al. 2019) describes SR as a graph classification problem;

[0108] S2-DHCN (Xia et al. 2021) is a state-of-the-art SR model. It is the first model to integrate self-supervised learning into SR and hypergraph modeling.

[0109] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] The above description provides a further detailed explanation of the present invention in conjunction with specific / preferred embodiments, and it should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of the present invention, and all such substitutions or modifications should be considered within the scope of protection of the present invention. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., indicates that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. A conversation recommendation method based on a dynamic attention mechanism, characterized in that, Includes the following steps: S1: Input session sequence; S2: Extract interest sequences from continuous user behavior through dynamic attention mechanism to model the user's deep-seated intentions; S3: Evaluate the probability of each candidate item based on the user's deep intent and make recommendations; Step S2 includes the following steps: S21: The input session sequence is transformed into a high-dimensional vector through the embedding layer to obtain the embedding vector of each item; S22: Extract hidden user interest sequences from continuous user behavior through the interest tracking layer; S23: The interest enhancement layer models the user's deeper intent based on the extracted interest sequence; Step S22 includes the following steps: S221: Arrange the embedded vector sequence in chronological order Cut into One slice; S222: For each slice The first multi-head attention network is used to extract hidden user interest sequences. in It is a high-dimensional vector sequence. Indicates the serial number; Step S222 ensures that the extraction of hidden user interests for the t-th item can only depend on its previous items by using a mask matrix in the first multi-head attention. The interest enhancement layer described in step S23 uses a second multi-head attention network to learn global information hidden behind the evolution of user interests, expressed as follows: ， in It is the output sequence of the interest tracking layer. The last element in , It represents global information based on user interest sequences.

2. The conversation recommendation method based on dynamic attention mechanism as described in claim 1, characterized in that, The formula for extracting the user's latent interests using the first multi-head attention network described in step S222 is as follows: ， ， in It is a weight matrix. It is the deviation vector. It is the activation function ReLU. It is the extracted sequence of hidden user interests.

3. The conversation recommendation method based on dynamic attention mechanism as described in claim 1, characterized in that, Step S22 further includes the following steps: S223: Applying a positional feedforward neural network gives the model more nonlinearity, expressed in the following formula: ， in, , It is a weight matrix. , It is the deviation vector. It is a hidden interest state.

4. The conversation recommendation method based on dynamic attention mechanism as described in claim 1, characterized in that, The first multi-head attention network and the second multi-head attention network are the same multi-head attention network.

5. The conversation recommendation method based on dynamic attention mechanism as described in claim 1, characterized in that, Step S3 includes the following steps: S31: Embed the last term into the vector The last latent interest embedding output by the interest tracking layer By connecting these connections, a representation of the local information is obtained at the end of the session using the first feedforward neural network. The formula for step S31 is as follows: ， in, It is a weight matrix. It is the deviation vector. SELU represents the nonlinear activation function. Represents local information; S32: will and the output of the interest enhancement layer By connecting these connections, a second feedforward neural network is applied to obtain a representation of the user's potential interests. The formula for step S32 is as follows: ， in, It is a weight matrix. It is the deviation vector. SELU represents the nonlinear activation function; S33: Calculate the probability of each candidate item and make a recommendation.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program can perform the steps of the method described in any one of claims 1-5.

Images