A sequence recommendation method and device based on multi-scale interest dynamic hierarchical Transformer
Through the multi-scale interest dynamic level Transformer method, the problem of insufficient user interest capture in the existing technology is solved, the recommendation accuracy is improved, the user's personalized needs is met, and the personalized recommendation performance is improved.
Patent Information
- Application Number
- CN202111219530.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-20
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-10-20
AI Technical Summary
The existing sequence recommendation methods lack multi-scale modeling when capturing user interests, resulting in low recommendation accuracy.
Using a sequence recommendation method based on a multi-scale dynamic hierarchy Transformer, by learning the distributed representation of users and items, using the neighbor attention mechanism to merge neighbor blocks layer by layer from bottom to top, and combining the Transformer layer self-attention mechanism to model user interests Multi-scale hierarchical representation, and interactive models are performed to obtain a unified representation of multi-scale interest between users and items.
It improves the prediction accuracy of the recommendation system, meets users' personalized needs for items of interest, and provides solutions with excellent personalized recommendation performance.
Smart Images

Figure CN113987336B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of e-commerce platform recommendation systems, and in particular to a sequence recommendation method and device based on a multi-scale interest dynamic hierarchical Transformer. Background Art
[0002] The main goal of the sequential recommendation task is to predict the items that users will be interested in in the near future based on the user's historical behavior sequence. Unlike traditional recommendation tasks, sequential recommendation considers the time sequence of user-item interactions and treats the user's historical behavior data as a sequence, aiming to capture the dynamic changes of user interests. Unlike time-sensitive recommendation tasks, sequential recommendation emphasizes the potential change pattern in the time sequence rather than the timestamp itself. Unlike content-based recommendation tasks, sequential recommendation only focuses on the interest changes reflected by the sequence itself, and does not introduce additional information outside the sequence, such as the attributes of the item. Multi-scale phenomena are prevalent in people's daily lives, such as time, location, and matter. From a microscopic perspective, the composition of matter is composed of molecules, atoms, and ions, and atoms are composed of atomic nuclei and extranuclear electrons. It is these multi-level structures that can explain the microscopic state of matter. In daily life, the user shopping behavior sequence naturally has multi-time scale structures, such as years, months, and days, but what is really useful for predicting future interests is the structure that reflects multiple interest scales.
[0003] Many scholars and research institutions around the world have conducted research on sequential recommendation and proposed multi-scale dependency algorithms, most of which focus on heuristic design of multi-scale implicit space representation methods. The two most representative methods are: based on the self-attention mechanism, significant improvements have been made in NLP tasks, and a new direction has been provided for applying the self-attention mechanism to sequential recommendation problems. Kang et al. developed a self-attention model SASRec for sequential recommendation. It adaptively assigns different weights to historical items at each time step. At the same time, Wang et al. and Hu et al. also learned hierarchical representations of users based on the self-attention mechanism to combine their long-term and short-term preferences.
[0004] Early work in recommendation methods aimed to extract sequential rules based on statistical co-occurrence patterns, and sequential recommendation mainly relied on the Markov chain (MC) assumption. Academia and industry jointly studied methods that combined matrix factorization (MF) and first-order MC to capture long-term user preferences and short-term user interests, respectively. All these MC-based methods have a common flaw, that is, these models only model local sequential patterns between adjacent items. One of the main challenges facing sequential recommendation is to extract and represent the historical interaction sequence of user preferences.
[0005] The MEANTIME (Mixture of AtteNTIon mechanisms with Multi-temporalEmbeddings) method combines temporal information and multiple encoding functions to provide a unique position embedding for the self-attention module. At the same time, it uses the diversity of the self-attention structure to capture various sequential patterns from the user's behavior sequence. Although MEANTIME uses multiple temporal embeddings to capture different sequential patterns in user behavior sequences, it does not perform multi-scale modeling of interests, resulting in low recommendation accuracy. Summary of the invention
[0006] The present invention proposes a sequence recommendation method and device based on a multi-scale interest dynamic hierarchical Transformer, which are used to solve or at least partially solve the technical problem of low recommendation accuracy in the prior art.
[0007] In order to solve the above technical problems, the first aspect of the present invention discloses a sequence recommendation method based on a multi-scale interest dynamic hierarchical Transformer, comprising:
[0008] S1: Learn the distributed representation of users and items based on the input user historical behavior sequence used to represent the interaction between users and items;
[0009] S2: Based on the distributed representation of users and items, the neighbor blocks are merged layer by layer from bottom to top through the neighbor attention mechanism to obtain a hierarchical representation of multi-scale user interests in the user's historical behavior sequence;
[0010] S3: The relationship between the user interest at the next moment and the hierarchical representation of the multi-scale user interest of the user's historical behavior sequence is modeled through the Transformer layer self-attention mechanism to obtain the user interest representation at the next moment;
[0011] S4: The user interest representation at the next moment is concatenated with the hierarchical representation of the multi-scale user interest of the user's historical behavior sequence. The interaction between the hierarchical representation of the multi-scale user interest and the distributed representation of the item is modeled through the residual structure of the Transformer layer to obtain a unified representation of the multi-scale interest of the user-item interaction.
[0012] S5: The unified representation of the multi-scale interests of the user and the item interaction is integrated to obtain a dynamic representation of the user's multi-scale interests, wherein the dynamic representation of the user's multi-scale interests is used to predict the user's multi-scale interest preferences and perform sequential recommendations.
[0013] In one embodiment, step S2 includes:
[0014] S2.1: Adopt the neighbor attention mechanism to determine whether adjacent items in the user's historical behavior sequence need to be merged;
[0015] S2.2: dynamically generate a block mask matrix according to the result of determining whether merging is required;
[0016] S2.3: Calculate the implicit representation of each layer based on the dynamically generated block mask matrix;
[0017] S2.4: Based on the implicit representation of each layer and the self-attention mechanism, the implicit representation of each layer at the corresponding scale is calculated;
[0018] S2.5: Update the representation of the object according to the dynamically generated block mask matrix;
[0019] S2.6: Based on the implicit representation at each scale and the updated item representation, a hierarchical representation of multi-scale user interests in the historical behavior sequence is obtained.
[0020] In one embodiment, S2.1 includes:
[0021] Adopting the neighbor attention mechanism, we calculate the similarity between the item representations of the current block and the left and right neighbor blocks respectively;
[0022] Based on the similarity of the item representation between the current block and the left adjacent block, it is determined whether the current block and the left adjacent block need to be merged. Based on the similarity of the item representation between the current block and the right adjacent block, it is determined whether the current block and the right adjacent block need to be merged.
[0023] In one embodiment, S2.5 includes:
[0024] According to the dynamic block mask generated by each layer of the dynamic hierarchical Transformer module, the object representation is obtained, and the AGG aggregation operation is performed on the object representation in each block to generate the final set of node representations in each scale as the updated object representation.
[0025] In one embodiment, step S3 uses Transformer to model the next time t and multi-scale user interests The dependence relationship between them, the calculation method of output O is as follows:
[0026]
[0027] Among them, PE i Indicates the sine and cosine position code of position i, PE 0,…,t-1 It represents the position encoding vector from position 0 to t-1, represents the implicit representation of item interaction of user u at the first layer, represents the implicit representation of item interaction of user u at layer D, Eu is the implicit representation of all user u’s item interactions, and the unified representation of user interests is recorded as
[0028] In one embodiment, step S5 uses a fully connected network to model the interaction relationship between items and users, and the connection weight is W0∈R n×p , the activation function of this fully connected network is softmax, which is used to predict the next item that the user is about to buy or browse The formula is as follows:
[0029]
[0030] is a unified representation of the multi-scale interests of users and items in interaction, and b is a bias.
[0031] Based on the same inventive concept, the second aspect of the present invention provides a sequence recommendation device based on a multi-scale interest dynamic hierarchical Transformer, comprising:
[0032] The embedding module is used to learn the distributed representation of users and items based on the input user historical behavior sequence representing the interaction between users and items;
[0033] Dynamic hierarchical Transformer module, which is used for distributed representation based on users and items. It merges neighbor blocks layer by layer from bottom to top through the neighbor attention mechanism to obtain a hierarchical representation of multi-scale user interests in the user's historical behavior sequence;
[0034] The Transformer module is used to model the relationship between the user interest at the next moment and the hierarchical representation of the multi-scale user interest of the user's historical behavior sequence through the Transformer layer self-attention mechanism to obtain the user interest representation at the next moment;
[0035] It is also used to concatenate the user interest representation at the next moment with the hierarchical representation of the multi-scale user interest of the user's historical behavior sequence. The interaction between the hierarchical representation of the multi-scale user interest and the distributed representation of the item is modeled through the residual structure of the Transformer layer, and a unified representation of the multi-scale interest of the user-item interaction is obtained.
[0036] The output module is used to fuse the unified representation of the multi-scale interests of the user and the item to obtain the dynamic representation of the user's multi-scale interests, wherein the dynamic representation of the user's multi-scale interests is used to predict the user's multi-scale interest preferences.
[0037] In one embodiment, the dynamic hierarchical Transformer module is composed of D layers, where D is the upper limit of the number of scales. Each layer is composed of a Transformer layer with a neighbor attention mechanism and a dynamic mask mechanism. The neighbor attention mechanism can generate a dynamic block mask matrix of the Transformer of this layer. The dynamic block mask matrix is used to update the object representation. The updated representation and the dynamic block mask matrix are used as the input of the lower layer Transformer.
[0038] The above one or more technical solutions in the embodiments of the present application have at least one or more of the following technical effects:
[0039] The present invention provides a sequence recommendation method based on a multi-scale interest dynamic hierarchical Transformer. Based on the distributed representation of users and items, the neighbor blocks can be merged layer by layer from bottom to top through the neighbor attention mechanism to obtain a multi-scale representation of the user's historical behavior sequence; and the relationship between the user's interest at the next moment and the hierarchical representation of the multi-scale user interest is modeled through the Transformer layer self-attention mechanism to obtain the user's interest representation at the next moment; and then the user interest representation at the next moment is spliced again with the hierarchical representation of the multi-scale user interest, and the interaction between the hierarchical representation of the multi-scale user interest and the distributed representation of the item is modeled to obtain a unified representation of the multi-scale interest of the user's interaction with the item, and then the unified representation of the multi-scale interest of the user's interaction with the item is fused. Therefore, the present invention can obtain the best user preference representation in the presence of a user's historical behavior sequence to meet the user's personalized needs for items of interest, thereby improving the accuracy of recommendations. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0041] Figure 1 The present invention is a flowchart of a sequence recommendation method based on a multi-scale interest dynamic hierarchical Transformer in the implementation of the present invention.
[0042] Figure 2 This is a structural diagram of a sequence recommendation device (MIDHT) based on a multi-scale interest dynamic hierarchical Transformer in the implementation of the present invention.
[0043] Figure 3 for Figure 2Flowchart of the implementation of the dynamic hierarchical Transformer module of the MIDHT model in .
[0044] Figure 4 Schematic diagram of the module of the dynamic hierarchical Transformer of the MIDHT model in the implementation of the present invention. DETAILED DESCRIPTION
[0045] The inventors of this application have found through extensive research and practice that:
[0046] The MEANTIME (Mixture of AtteNTIon mechanisms with Multi-temporalEmbeddings) method combines temporal information and multiple encoding functions to provide a unique position embedding for the self-attention module. At the same time, it uses the diversity of the self-attention structure to capture various sequential patterns from the user's behavior sequence. Although MEANTIME uses multiple temporal embeddings to capture different sequence patterns in user behavior sequences, it does not perform multi-scale modeling of interests. Based on this, the present invention provides a sequence recommendation method and device based on a multi-scale interest dynamic hierarchical Transformer. The hierarchical clustering method is used to study the hierarchical characteristics in the user behavior sequence data, and analyze the hierarchical dependencies of user interests, so as to better provide users with personalized recommendation services.
[0047] The main inventive concepts of the present invention are as follows:
[0048] This paper provides a sequential recommendation method based on a multi-scale interest dynamic layer Transformer. The multi-scale interest dynamic layer is used to obtain a multi-scale representation of the user's historical behavior sequence. Then, the Transformer is used to model the dependency between the next moment t and the multi-scale user interest. The unified representation of the multi-scale interest of the user and the item interaction is then fused to obtain a dynamic representation of the user's multi-scale interest. Therefore, the best user preference representation can be obtained in the presence of a user's historical behavior sequence to meet the user's personalized needs for items of interest.
[0049] Provided is a device corresponding to the sequence recommendation method, which is the sequence recommendation model in the present invention, namely MIDHT (Multi-scale Interest Dynamic Hierarchical Transformer), which performs fine-grained multi-scale modeling of user interests. The dynamic hierarchical Transformer module in the sequence recommendation model first uses the neighbor attention mechanism to determine whether two adjacent items need to be merged; then dynamically generates a block mask matrix using the judgment result of the previous step; then calculates the implicit representation of the current layer based on the dynamic block mask matrix and the self-attention mechanism; and then the corresponding hierarchical structure can be inferred from the dynamic block mask matrices of all layers. The present invention has the characteristics of high computational efficiency and significant personalization effect, and can well predict the user's final interest preference. Therefore, the personalized recommendation performance of the present invention is excellent.
[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0051] Embodiment 1
[0052] The embodiment of the present invention provides a sequence recommendation method based on a multi-scale interest dynamic hierarchical Transformer, including:
[0053] S1: Learn the distributed representation of users and items based on the input user historical behavior sequence used to represent the interaction between users and items;
[0054] S2: Based on the distributed representation of users and items, the neighbor blocks are merged layer by layer from bottom to top through the neighbor attention mechanism to obtain a hierarchical representation of multi-scale user interests in the user's historical behavior sequence;
[0055] S3: The relationship between the user interest at the next moment and the hierarchical representation of the multi-scale user interest of the user's historical behavior sequence is modeled through the Transformer layer self-attention mechanism to obtain the user interest representation at the next moment;
[0056] S4: The user interest representation at the next moment is concatenated with the hierarchical representation of the multi-scale user interest of the user's historical behavior sequence. The interaction between the hierarchical representation of the multi-scale user interest and the distributed representation of the item is modeled through the residual structure of the Transformer layer to obtain a unified representation of the multi-scale interest of the user-item interaction.
[0057] S5: The unified representation of the multi-scale interests of the user and the item interaction is integrated to obtain a dynamic representation of the user's multi-scale interests, wherein the dynamic representation of the user's multi-scale interests is used to predict the user's multi-scale interest preferences and perform sequential recommendations.
[0058] Specifically, the user's historical behavior sequence can be obtained by cleaning and calculating the acquired raw data. S1 can use embedding technology to learn the distributed representation of users and items.
[0059] In the specific implementation process, the input is the user's historical behavior sequence S1, S2, ..., S t , where S t represents the interaction sequence between the user and the item at time t; the user set U = {u1,u2,…,u m}, where |U|=m, u m represents the mth user u; the item set I = {i1,i2,…,i n}, where |I|=n,i n represents the nth item i. The item sequence that user u interacted with before time t is recorded as in represents a subset of the item set, that is, the sequence of user u's interactions with items at time t-1. Given the set of all user interaction sequences The goal of the sequential recommendation task is to predict the items that user u will interact with at the next time t.
[0060] It should be noted that the formal definition parameters and object types specified herein are only used to illustrate the implementation process of the present invention and are not intended to limit the present invention.
[0061] The distributed representations of users and items are learned through the Embedding layer, and the learned distributed representations of all users and items are recorded as U∈R m×r and V∈R n×r , where r is the size of the distributed representation dimension, m is the number of users, and n is the number of items. User behavior sequence and user number u, through the Embedding layer, get implicit representations respectively and E u =U[u,:]∈R r ,in For sequence Length, is the implicit representation of item interaction of user u at the 0th layer, E u is the implicit representation of all user u’s item interactions.
[0062] It should be noted that the representation parameters, user behavior sequence representation and equation expression specified herein are only for illustrating the specific implementation steps of the present invention and are not intended to limit the present invention.
[0063] Step S2 updates the distributed representation of users and items obtained in step S1. Specifically, through the neighbor attention mechanism, neighbor blocks are merged layer by layer from bottom to top to form a multi-scale user interest tree, and a hierarchical expression of user interests in the user historical behavior sequence is obtained.
[0064] Step S3 uses Transformer to model the dependency between the next time t and multi-scale user interests for the updated item representation obtained in step S2 (the hierarchical expression of user interests in the user historical behavior sequence);
[0065] Step S4 models the interaction between the unified multi-scale user interest representation and the implicit representation of the item (the updated item representation) for the unified expression of the user's interest at the next moment obtained in step S3. (That is, S1 is to obtain the separate embedding representation of the item and the user respectively, and S4 is to obtain the embedding representation of the interaction between the item and the user)
[0066] Step S5 combines the user interest representation at the next moment obtained in step S3 and the multi-scale interest representation of the user's interaction with the item obtained in step S4 into a dynamic representation of the user's multi-scale interest.
[0067] See also Figure 1 , which is a flowchart of a sequence recommendation method based on multi-scale interest dynamic hierarchical Transformer in the implementation of the present invention.
[0068] In one embodiment, step S2 includes:
[0069] S2.1: Adopt the neighbor attention mechanism to determine whether adjacent items in the user's historical behavior sequence need to be merged;
[0070] S2.2: dynamically generate a block mask matrix according to the result of determining whether merging is required;
[0071] S2.3: Calculate the implicit representation of each layer based on the dynamically generated block mask matrix;
[0072] S2.4: Based on the implicit representation of each layer and the self-attention mechanism, the implicit representation of each layer at the corresponding scale is calculated;
[0073] S2.5: Update the representation of the object according to the dynamically generated block mask matrix;
[0074] S2.6: Based on the implicit representation at each scale and the updated item representation, a hierarchical representation of multi-scale user interests in the historical behavior sequence is obtained.
[0075] See also Figure 3 , which is the implementation flow chart of the dynamic hierarchical Transformer module of the MIDHT model.
[0076] In the specific implementation process, a judgment result is given on whether adjacent items in the user's historical behavior sequence need to be merged, that is, the neighbor attention mechanism is used to judge whether they need to be merged; the above steps are repeated until the relationship between all blocks is judged and each maximum mergeable area is found;
[0077] Step S2.2 re-merges each of the largest mergeable regions obtained in step S2.1, and dynamically generates a block mask matrix (the essence of the block mask matrix is a block diagonal matrix).
[0078] Generate a new block mask matrix representation as follows:
[0079]
[0080] in It should be noted that the method for generating a new block mask matrix representation is only used to illustrate the specific implementation steps of the present invention and is not intended to limit the present invention.
[0081] Step S2.3 uses the data-adaptive hierarchy generated by each layer of the dynamic hierarchical Transformer module to calculate the implicit representation of the current layer.
[0082] Step S2.4 fuses the dynamic block mask with the self-attention mechanism to obtain the implicit representation at this scale.
[0083] Step S2.5 uses the dynamically generated block mask matrix in step S2.2 and the implicit representation of the hierarchy in step S2.3. Transformer updates the item representation according to the block mask. S2.6 can infer the multi-scale hierarchical structure of the historical sequence based on the D block mask matrices obtained from the dynamic hierarchical Transformer module.
[0084] An adaptive decoupled multi-scale sequence recommendation method is used to heuristically design the implicit representation space so that it can explicitly infer hierarchical problems. At the same time, multi-scale implicit representations and explicit user interest hierarchy trees are learned to predict users' multi-scale interest preferences and better provide assistance for users' personalized service needs.
[0085] In one embodiment, S2.1 includes:
[0086] Adopting the neighbor attention mechanism, we calculate the similarity between the item representations of the current block and the left and right neighbor blocks respectively;
[0087] Based on the similarity of the item representation between the current block and the left adjacent block, it is determined whether the current block and the left adjacent block need to be merged. Based on the similarity of the item representation between the current block and the right adjacent block, it is determined whether the current block and the right adjacent block need to be merged.
[0088] In the specific implementation process, the similarity between the current block and the left and right neighboring blocks is reflected by the probability value. The larger the probability value, the more similar they are.
[0089] According to the current block With left neighbor The combined probability value P i,i-1 The relationship between the left neighboring block and the threshold, and the relationship between the probability of merging the left neighboring block and the current block and the threshold, determine whether the current block and the left neighboring block need to be merged; at the same time, according to the current block With the right neighbor The combined probability value P i,i+1 Relationship with the threshold, the probability value P of merging the right neighbor block with the current block i+1,i The relationship with the threshold determines whether the current block needs to be merged with the right neighboring block. The threshold can be selected according to the actual situation. In the following example, the threshold is 0.5. The formula for calculating the probability is as follows:
[0090]
[0091] in, represents the adjacent block attention matrix between the i-th block and the i-1-th block in the k-th layer. When P i,i+1 >0.5 and P i+1,i >0.5, indicating and Two blocks need to be merged, and the F(i) value is used to mark whether the current block needs to be merged with the left and right neighboring blocks, that is, when F(i) = 1, it needs to be merged; otherwise, when F(i) = 0, it does not need to be merged. Continue to judge Can it be merged with its right neighbor block? Repeat the above steps until the relationship between all blocks is determined and recorded in F. According to F, find each largest mergeable area and record it as [i, j], satisfying F(i-1) = 0 or does not exist and F(j+1) = 0 or does not exist and for F(c)=1.
[0092] It should be noted that the representation method of determining the left neighbor block and the right neighbor block and the representation of the determination method are only used to illustrate the specific implementation steps of the present invention and are not used to limit the present invention.
[0093] In one embodiment, S2.5 includes:
[0094] According to the dynamic block mask generated by each layer of the dynamic hierarchical Transformer module, the object representation is obtained, and the AGG aggregation operation is performed on the object representation in each block to generate the final set of node representations in each scale as the updated object representation.
[0095] The dynamic block mask is The set of items represented is
[0096] In the specific implementation process, mean aggregation is used to calculate the node representation in each scale. The calculation equation is expressed as follows:
[0097]
[0098] It should be noted that the node representation method in each scale is only used to illustrate the specific implementation steps of the present invention and is not intended to limit the present invention.
[0099] In one embodiment, step S3 uses Transformer to model the next time t and multi-scale user interests The dependence relationship between them, the calculation method of output O is as follows:
[0100]
[0101] Among them, PE i Indicates the sine and cosine position code of position i, PE 0,…,t-1 It represents the position encoding vector from position 0 to t-1, express, The unified representation of user interests is denoted as
[0102] In one embodiment, step S5 uses a fully connected network to model the interaction relationship between items and users, and the connection weight is W0∈R n×p , the activation function of this fully connected network is softmax, which is used to predict the next item that the user is about to buy or browse The formula is as follows:
[0103]
[0104] is a unified representation of the multi-scale interests of users and items in interaction, and b is a bias.
[0105] Compared with the prior art, the beneficial effects of the present invention are:
[0106] The present invention uses a multi-scale interest dynamic hierarchical transformer to fine-grainedly model the multi-scale interests of users, and improves the prediction accuracy of the recommendation system when users have dynamic interests. It meets the personalized needs of users for e-commerce websites and various multimedia information websites that provide personalized services, helps users discover information that is valuable to them, and allows the information to be displayed to users who are interested in it.
[0107] Embodiment 2
[0108] Based on the same inventive concept, this embodiment provides a sequence recommendation device based on a multi-scale interest dynamic hierarchical Transformer, including:
[0109] The embedding module is used to learn the distributed representation of users and items based on the input user historical behavior sequence representing the interaction between users and items;
[0110] Dynamic hierarchical Transformer module, which is used for distributed representation based on users and items. It merges neighbor blocks layer by layer from bottom to top through the attention mechanism of neighbor blocks to obtain a hierarchical representation of multi-scale user interests in the user's historical behavior sequence;
[0111] The Transformer module is used to model the relationship between the user interest at the next moment and the hierarchical representation of the multi-scale user interest of the user's historical behavior sequence through the Transformer layer self-attention mechanism to obtain the user interest representation at the next moment;
[0112] It is also used to concatenate the user interest representation at the next moment with the hierarchical representation of the multi-scale user interest of the user's historical behavior sequence. The interaction between the hierarchical representation of the multi-scale user interest and the distributed representation of the item is modeled through the residual structure of the Transformer layer, and a unified representation of the multi-scale interest of the user-item interaction is obtained.
[0113] The output module is used to fuse the unified representation of the multi-scale interests of the user and the item to obtain the dynamic representation of the user's multi-scale interests, wherein the dynamic representation of the user's multi-scale interests is used to predict the user's multi-scale interest preferences.
[0114] Specifically, this embodiment provides a sequence recommendation device based on a multi-scale interest dynamic hierarchical Transformer, that is, a sequence recommendation model: a MIDHT model, and the output module is a fully connected network (fully connected layer).
[0115] See also Figure 2 , which is a structural diagram of a sequence recommendation device (MIDHT) based on multi-scale interest dynamic hierarchical Transformer.
[0116] In one embodiment, the dynamic hierarchical Transformer module is composed of D layers, where D is the upper limit of the number of scales. Each layer is composed of a Transformer layer with neighbor attention and a dynamic mask mechanism. The neighbor attention mechanism can generate a dynamic block mask matrix of the Transformer of this layer. The dynamic block mask matrix is used to update the object representation. The updated representation and the dynamic block mask matrix are used as the input of the lower layer Transformer.
[0117] See also Figure 4 , which is a module diagram of the dynamic hierarchical Transformer.
[0118] The later item representation formula is as follows:
[0119]
[0120] in represents the updated item representation of user u at the k+1th level, represents the multi-head attention representation of user u at the k+1th layer, where W H is a hyperparameter, express Input representation to the fully connected layer.
[0121] It should be noted that the value of D and the updated representation of the item are specified here only to illustrate the specific implementation process of the present invention and are not intended to limit the present invention.
[0122] Since the device introduced in the second embodiment of the present invention is a device used to implement the method for associated user identification based on the multi-layer self-attention mechanism in the first embodiment of the present invention, a person skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not repeated here. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.
[0123] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A sequence recommendation method based on multi-scale interest dynamic hierarchical Transformer, characterized in that: include: S1: Learn the distributed representation of users and items based on the input user historical behavior sequence used to represent the interaction between users and items; S2: Based on the distributed representation of users and items, the neighbor blocks are merged layer by layer from bottom to top through the neighbor attention mechanism to obtain a hierarchical representation of multi-scale user interests in the user's historical behavior sequence; S3: The relationship between the user interest at the next moment and the hierarchical representation of the multi-scale user interest of the user's historical behavior sequence is modeled through the Transformer layer self-attention mechanism to obtain the user interest representation at the next moment; S4: The user interest representation at the next moment is concatenated with the hierarchical representation of the multi-scale user interest of the user's historical behavior sequence. The interaction between the hierarchical representation of the multi-scale user interest and the distributed representation of the item is modeled through the residual structure of the Transformer layer to obtain a unified representation of the multi-scale interest of the user-item interaction. S5: The unified representation of multi-scale interests of users and items is integrated to obtain a dynamic representation of users' multi-scale interests. The dynamic representation of users' multi-scale interests is used to predict users' multi-scale interest preferences and perform sequential recommendations. Wherein, step S2 comprises: S2.1: Adopt the neighbor attention mechanism to determine whether adjacent items in the user's historical behavior sequence need to be merged; S2.2: dynamically generate a block mask matrix according to the result of determining whether merging is required; S2.3: Calculate the implicit representation of each layer based on the dynamically generated block mask matrix; S2.4: Based on the implicit representation of each layer and the self-attention mechanism, the implicit representation of each layer at the corresponding scale is calculated; S2.5: Update the representation of the object according to the dynamically generated block mask matrix; S2.6: Based on the implicit representation at each scale and the updated item representation, a hierarchical representation of multi-scale user interests in the historical behavior sequence is obtained; S2.1 includes: Adopting the neighbor attention mechanism, we calculate the similarity between the item representations of the current block and the left and right neighbor blocks respectively; Based on the similarity of the item representation between the current block and the left adjacent block, determine whether the current block and the left adjacent block need to be merged; based on the similarity of the item representation between the current block and the right adjacent block, determine whether the current block and the right adjacent block need to be merged; S2.5 includes: According to the dynamic block mask generated by each layer of the dynamic hierarchical Transformer module, the object representation is obtained, and the AGG aggregation operation is performed on the object representation in each block to generate the final set of node representations in each scale as the updated object representation.
2. The sequence recommendation method based on multi-scale interest dynamic hierarchical Transformer according to claim 1, characterized in that: Step S3 uses Transformer to model the next time t and multi-scale user interests The dependence relationship between them, the calculation method of output O is as follows: Among them, PE i Indicates the sine and cosine position code of position i, PE 0,…,t-1 It represents the position encoding vector from position 0 to t-1, represents the implicit representation of item interaction of user u at the first layer, represents the implicit representation of item interaction of user u at layer D, E u is the implicit representation of all user u’s item interactions, and the unified representation of user interests is recorded as 3. The sequence recommendation method based on multi-scale interest dynamic hierarchical Transformer according to claim 1, characterized in that: Step S5 uses a fully connected network to model the interaction between items and users, with a connection weight of W0∈R n×p , the activation function of this fully connected network is softmax, which is used to predict the next item that the user is about to buy or browse The formula is as follows: is a unified representation of the multi-scale interests of users and items in interaction, and b is a bias.
4. A sequence recommendation device based on multi-scale interest dynamic hierarchical Transformer, characterized in that: The device is a MIDHT model, including: The embedding module is used to learn the distributed representation of users and items based on the input user historical behavior sequence representing the interaction between users and items; Dynamic hierarchical Transformer module, which is used for distributed representation based on users and items. It merges neighbor blocks layer by layer from bottom to top through the neighbor attention mechanism to obtain a hierarchical representation of multi-scale user interests in the user's historical behavior sequence; The Transformer module is used to model the relationship between the user interest at the next moment and the hierarchical representation of the multi-scale user interest of the user's historical behavior sequence through the Transformer layer self-attention mechanism to obtain the user interest representation at the next moment; It is also used to concatenate the user interest representation at the next moment with the hierarchical representation of the multi-scale user interest of the user's historical behavior sequence. The interaction between the hierarchical representation of the multi-scale user interest and the distributed representation of the item is modeled through the residual structure of the Transformer layer, and a unified representation of the multi-scale interest of the user-item interaction is obtained. An output module is used to fuse the unified representation of the multi-scale interests of the user and the item to obtain a dynamic representation of the user's multi-scale interests, wherein the dynamic representation of the user's multi-scale interests is used to predict the user's multi-scale interest preferences; The dynamic hierarchical Transformer module is specifically used for: Adopt the neighbor attention mechanism to determine whether adjacent items in the user's historical behavior sequence need to be merged; Dynamically generate a block mask matrix based on the judgment result of whether merging is required; Compute the implicit representation of each layer based on the dynamically generated block mask matrix; According to the implicit representation of each layer, based on the self-attention mechanism, the implicit representation of each layer at the corresponding scale is calculated; Update the representation of the object according to the dynamically generated block mask matrix; Based on the implicit representation at each scale and the updated item representation, a hierarchical representation of multi-scale user interests in the historical behavior sequence is obtained; The neighbor attention mechanism is used to determine whether adjacent items in the user's historical behavior sequence need to be merged, including: Adopting the neighbor attention mechanism, we calculate the similarity between the item representations of the current block and the left and right neighbor blocks respectively; Based on the similarity of the item representation between the current block and the left adjacent block, determine whether the current block and the left adjacent block need to be merged; based on the similarity of the item representation between the current block and the right adjacent block, determine whether the current block and the right adjacent block need to be merged; The neighbor attention mechanism is used to determine whether adjacent items in the user's historical behavior sequence need to be merged, including: According to the dynamic block mask generated by each layer of the dynamic hierarchical Transformer module, the object representation is obtained, and the AGG aggregation operation is performed on the object representation in each block to generate the final set of node representations in each scale as the updated object representation.
5. The multi-scale interest dynamic hierarchical Transformer sequence recommendation device according to claim 4, characterized in that: The dynamic hierarchical Transformer module consists of D layers, where D is the upper limit of the number of scales. Each layer consists of a Transformer layer with a neighbor attention mechanism and a dynamic mask mechanism. The neighbor attention mechanism can generate a dynamic block mask matrix for the Transformer of this layer. The dynamic block mask matrix is used to update the object representation. The updated representation and the dynamic block mask matrix are used as the input of the lower Transformer.
