Sequence recommendation method based on diffusion model and attention mechanism

Abstract

The invention discloses a sequence recommendation method based on a diffusion model and a self-attention mechanism, and the method mainly comprises the steps: firstly obtaining the embedded representation of an input user historical behavior sequence and candidate recommendation items through an embedded layer, then carrying out the forward noise addition, and training an information extraction network based on the attention mechanism in the process; noise samples are randomly sampled from Gaussian distribution, a user historical behavior sequence is input into an embedding layer to obtain a user historical behavior embedding representation, and the embedding representation is input into an information extraction network to obtain a user interest feature representation. And then inputting the noise sample, the user historical behavior embedded representation and the user interest feature representation into a diffusion model for reverse process denoising, obtaining the most matched recommendation sequence embedded representation after a denoising process with specified steps, and decoding through a decoding layer to obtain a recommendation item sequence with the highest score, thereby completing a sequence recommendation task. According to the method, the user interest features in the user historical behavior sequence are extracted by using the diffusion model and through the improved information extraction network and the self-attention mechanism, and the accuracy of personalized sequence recommendation is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of information retrieval, and in particular to a sequence recommendation method based on a diffusion model. Background Art

[0002] With the rapid development of the Internet, a large amount of data is generated every day. This data contains rich information, but due to its unstructured characteristics, it is difficult to obtain information. Recommendation systems are widely used in the network field to alleviate the problem of information overload. Sequential recommendation understands and predicts changes in user interests by considering the temporal and personalized characteristics of user behavior, and intelligently provides users with more personalized and accurate recommendation services. With the advent of the AI ​​era, many types of networks such as CNN, RNN, and Transformer have successfully demonstrated excellent performance in sequential recommendation systems, and neural network recommendation systems have gradually become the mainstream recommendation systems. These methods achieve recommendations by representing items as a set of fixed vectors and then inputting them into a predetermined set of neural networks.

[0003] At present, common sequence recommendation methods are divided into traditional content-based or collaborative filtering recommendation methods and deep learning-based sequence recommendation methods. Traditional sequence recommendation methods mainly make recommendations based on similar user behaviors. These methods include user-based collaborative filtering, item-based coordinated filtering, matrix decomposition, latent semantic models, feature matching, classification algorithms and other algorithms. They rely on the interactive information between users and items or the content features of items, and recommend through similar users or similar items or content similarity. The excellent generalization ability and ability to handle nonlinear problems of deep neural networks make them an ideal choice for sequence recommendation, because neural network models can better capture the temporal dependencies of user behaviors, model users' short-term and long-term interests, and effectively handle the sparsity of data. However, such methods cannot fit the uncertainty of users in the real world well during the reasoning process, and negative sampling must be used in the recommendation process to increase the recommendation range of the model. Summary of the invention

[0004] In order to solve the problems existing in the above-mentioned prior art, the present invention provides a sequential recommendation method based on a diffusion model, which uses a diffusion model to replace the traditional recurrent neural network, and incorporates natural noise into the network training process to enable the network to adapt to invalid interactions in user behavior data. It also introduces an information extraction network to extract user preference information hidden in the user history sequence, and uses this information for guided diffusion in the reverse process. This effectively improves the accuracy of the model's personalized recommendation.

[0005] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0006] The present invention provides a personalized sequence recommendation method based on a diffusion model and an attention mechanism, comprising the following steps:

[0007] Step S1: Map the user historical behavior sequence and target item of the training set into corresponding embedding representations through the embedding layer;

[0008] Step S2: The target object embedding representation is denoised for a specified number of steps through the forward process of the diffusion model to obtain the corresponding noise embedding representation after denoising;

[0009] Step S3: embedding the user's historical behavior sequence into the Transformer-based information extraction network Guidance to obtain the user's interest feature representation, and training a high-precision information extraction network Guidance through gradient descent;

[0010] Step S4: Sample a random noise sequence from the Gaussian distribution by random sampling, and map it into a noise embedding representation through an embedding layer;

[0011] Step S5: Map the input user history behavior sequence into the corresponding embedding representation through the embedding layer and input the information extraction network Guidance to obtain the user interest feature representation;

[0012] Step S6: The user interest feature representation and the current diffusion step number are subjected to a reverse denoising process of the diffusion model. After the denoising process of the specified diffusion step number, a recommended item sequence representation is obtained;

[0013] Step S7: Finally, the recommended item sequence representation is mapped into a specific item sequence through the decoding layer to complete the sequence recommendation task;

[0014] Furthermore, in step S1, for the input behavior sequence Select the first n-1 user history As a user's historical behavior sequence, As the target to be predicted, u represents the item that user i interacted with for the nth time. First, S 1:n―1 and Input embedding layer to get their respective embedding representation v 1:n―1 and v n ;

[0015] Furthermore, in step S2, the target object is embedded into the representation v n Perform the forward noise addition process of the diffusion model with the specified diffusion step number t, and obtain the noise embedding representation after t steps of noise addition

[0016] Secondly, the diffusion model forward noise addition process uses the following formula:

[0017]

[0018] Among them, t represents the number of diffusion steps of the input, and the remaining parameters are determined by the hyperparameter α t It is calculated that the hyperparameter α t It is a value that changes linearly with the number of diffusion steps.

[0019] The specific calculation process is as follows:

[0020] β t =1―α t

[0021]

[0022] Specifically, calculation It can be expressed by the following formula:

[0023]

[0024] Furthermore, in step S3, the user historical behavior sequence is embedded into the representation v 1:n―1 Input the information extraction network Guidance to obtain the user interest feature hinfo, and embed the obtained user interest feature hinfo and noise into a representation The diffusion step number t is predicted by denoising through the reverse denoising process of the diffusion model A high-precision Guide network is trained using gradient descent using the specified loss function.

[0025] The specific calculation formula is as follows:

[0026] Denoising process:

[0027]

[0028] Loss function:

[0029] L t―1 =D KL (q(v n t―1 |v n t―1 ,v n―1 0 )||p θ (v n t―1 |v n t ,h info )

[0030] L vlb =L 0 +L 1 +…+L T―1 +L T

[0031] Further, in step S4, random noise ∈ is sampled from the Gaussian distribution by random sampling and mapped into a noise embedding representation μ through an embedding layer;

[0032] The specific sampling formula is:

[0033]

[0034] in is a standard Gaussian distribution.

[0035] Furthermore, in step S5, the input user historical behavior sequence to be predicted is mapped into the corresponding embedding representation v through the embedding layer. 1:n―1 , the embedded representation is input into the information extraction network Guidance to obtain the user interest feature representation hinfo;

[0036] Furthermore, in step S6, the random noise embedding representation μ is subjected to a reverse denoising process of the diffusion model. After the denoising process with a specified number of diffusion steps t, the recommended item sequence representation is obtained.

[0037] The specific process is as follows:

[0038] The denoising formula from step t to step t-1 is as follows:

[0039]

[0040] in, Represents a normal distribution, which is calculated by specifying the mean variance A sample is taken from a Gaussian distribution.

[0041] Mean calculation formula:

[0042]

[0043] Variance calculation formula:

[0044]

[0045] where v 0 It is predicted by a Transformer-based noise prediction network, which calculates and predicts the input noise embedding representation μ, the user interest feature representation hinfo, and the current specific denoising step number t1.

[0046] The specific prediction process is as follows:

[0047]

[0048] The prediction result consists of two parts. The first part is the prediction result of denoising the specified interest features of a specific user in the diffusion model denoising process through the user interest feature representation hinfo; the second part is the prediction result of denoising through random noise. The current prediction is obtained by fusion representation of the two parts of the prediction results.

[0049] Iterate the above denoising process, repeatedly updating the user interest feature representation hinfo and the current prediction result in each prediction Finally, the recommended item sequence representation is obtained

[0050] Further, in step S7, the recommended item sequence is represented as After passing through the fully connected layer, it is sent to a standard conditional random field CRF to perform the result mapping of the item sequence. The probability of the item sequence p(y|s; θ) is defined as:

[0051]

[0052] where y v Represents the recommended item sequence representation All possible tag sequences.

[0053] Filter out the recommended item sequence representation The item sequence y with the highest conditional probability * As the final output:

[0054]

[0055] Compared with the prior art, the present invention has the following advantages:

[0056] (1) The present invention proposes a sequence recommendation method based on a diffusion model, which can add rich random Gaussian noise in the sequence recommendation process, better fit the uncertainty in actual situations, and effectively improve the accuracy of sequence recommendation.

[0057] (2) The present invention proposes to use a Transformer-based information extraction network to generate user interest features. The information extraction network can fully mine the user interest information hidden in the user's historical behavior sequence and fully extract the context information of the sequence, thereby generating an efficient interest feature representation.

[0058] (3) The present invention proposes to use the forward process and reverse process of the diffusion model to effectively simulate the generation process of user behavior and enhance the richness of recommendations.

[0059] (4) The present invention proposes the use of information fusion technology to effectively reduce model time and memory consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flowchart of the steps of a sequence recommendation method based on a diffusion model and an attention mechanism according to the technical solution of the present invention;

[0061] Figure 2 It is an overall framework diagram of a sequence recommendation method based on a diffusion model and an attention mechanism according to the technical solution of the present invention;

[0062] Figure 3 It is an information extraction model diagram of a sequence recommendation method based on a diffusion model and an attention mechanism according to the technical solution of the present invention; DETAILED DESCRIPTION

[0063] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0064] like Figure 1 As shown, a sequence recommendation method based on a diffusion model and an attention mechanism includes the following steps:

[0065] Step S1: Map the user historical behavior sequence and target item of the training set into corresponding embedding representations through the embedding layer;

[0066] For the input behavior sequence Select the first n-1 user history As a user's historical behavior sequence, As the target to be predicted, u represents the item that user i interacted with for the nth time. First, S 1:n―1 and Input embedding layer to get their respective embedding representation v 1:n―1 and v n ;

[0067] Step S2: Embed the target item into representation v n Perform the forward noise addition process of the diffusion model with the specified diffusion step number t, and obtain the noise embedding representation after t steps of noise addition

[0068] Secondly, the diffusion model forward noise addition process uses the following formula:

[0069]

[0070] Among them, t represents the number of diffusion steps of the input, and the remaining parameters are determined by the hyperparameter α t It is calculated that the hyperparameter α t It is a value that changes linearly with the number of diffusion steps.

[0071] The specific calculation process is as follows:

[0072] β t =1―α t

[0073]

[0074] Specifically, calculation It can be expressed by the following formula:

[0075]

[0076] Step S3: Embed the user's historical behavior sequence into representation v 1:n―1 Input the information extraction network Guidance to obtain the user interest feature hinfo, and embed the obtained user interest feature hinfo and noise into a representation The diffusion step number t is predicted by denoising through the reverse denoising process of the diffusion model A high-precision Guide network is trained using gradient descent using the specified loss function.

[0077] The specific calculation formula is as follows:

[0078] Denoising process:

[0079]

[0080] Loss function:

[0081] L t―1 =D KL (q(v n t―1 |v n t―1 ,v n―1 0 )||p θ (v n t―1 |v n t ,h info )

[0082] L vlb =L 0 +L 1 +…+L T―1 +L T

[0083] In step S4: random noise ∈ is sampled from the Gaussian distribution by random sampling and mapped into a noise embedding representation μ through the embedding layer;

[0084] The specific sampling formula is:

[0085]

[0086] in is a standard Gaussian distribution.

[0087] Step S5: Map the input user historical behavior sequence to be predicted into the corresponding embedding representation v through the embedding layer 1:n―1 , the embedded representation is input into the information extraction network Guidance to obtain the user interest feature representation hinfo;

[0088] Step S6: The random noise embedding representation μ is subjected to the reverse denoising process of the diffusion model. After the denoising process with the specified diffusion step number t, the recommended item sequence representation is obtained.

[0089] The specific process is as follows:

[0090] The denoising formula from step t to step t-1 is as follows:

[0091]

[0092] in, Represents a normal distribution, which is calculated by specifying the mean variance A sample is taken from a Gaussian distribution.

[0093] Mean calculation formula:

[0094]

[0095] Variance calculation formula:

[0096]

[0097] where v 0 It is predicted by a Transformer-based noise prediction network, which calculates and predicts the input noise embedding representation μ, the user interest feature representation hinfo, and the current specific denoising step number t1.

[0098] The specific prediction process is as follows:

[0099]

[0100] The prediction result consists of two parts. The first part is the prediction result of denoising the specified interest features of a specific user in the diffusion model denoising process through the user interest feature representation hinfo; the second part is the prediction result of denoising through random noise. The current prediction is obtained by fusion representation of the two parts of the prediction results.

[0101] Iterate the above denoising process, repeatedly updating the user interest feature representation hinfo and the current prediction result in each prediction Finally, the recommended item sequence representation is obtained

[0102] Step S7: Represent the recommended item sequence After passing through the fully connected layer, it is sent to a standard conditional random field CRF to perform the result mapping of the item sequence. The probability of the item sequence p(y|s; θ) is defined as:

[0103]

[0104] where y v Represents the recommended item sequence representation All possible tag sequences.

[0105] Filter out the recommended item sequence representation The item sequence y with the highest conditional probability * As the final output:

[0106]

[0107] Finally, we get the input user behavior sequence The corresponding item sequence y * , so far the sequence recommendation task is completed.

[0108] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A Chinese named entity recognition method based on a pre-trained model and dictionary statistical information, comprising the following steps: Step S1: Map the user historical behavior sequence and target item of the training set into corresponding embedding representations through the embedding layer; Step S2: The target object embedding representation is denoised for a specified number of steps through the forward process of the diffusion model to obtain the corresponding noise embedding representation after denoising; Step S3: embedding the user's historical behavior sequence into the Transformer-based information extraction network Guidance to obtain the user's interest feature representation, and training a high-precision information extraction network Guidance through gradient descent; Step S4: Sample a random noise sequence from the Gaussian distribution by random sampling, and map it into a noise embedding representation through an embedding layer; Step S5: Map the input user history behavior sequence into the corresponding embedding representation through the embedding layer and input the information extraction network Guidance to obtain the user interest feature representation; Step S6: The user interest feature representation and the current diffusion step number are subjected to a reverse denoising process of the diffusion model. After the denoising process of the specified diffusion step number, a recommended item sequence representation is obtained; Step S7: Finally, the recommended item sequence representation is mapped into a specific item sequence through the decoding layer to complete the sequence recommendation task.

2. The sequence recommendation method based on diffusion model and attention mechanism according to claim 1, characterized in that: The user historical behavior sequence and target item of the training set are mapped into corresponding embedding representations through the embedding layer. Select the first n-1 user history As a user's historical behavior sequence, As the target to be predicted, u represents the item that user i interacted with for the nth time. First, S 1:n―1 and Input embedding layer to get their respective embedding representation v 1:n―1 and v n。 3. The sequence recommendation method based on diffusion model and attention mechanism according to claim 1, characterized in that: The step S2 includes embedding the target object into a representation v n Perform the forward noise addition process of the diffusion model with the specified diffusion step number t, and obtain the noise embedding representation after t steps of noise addition Secondly, the diffusion model forward noise addition process uses the following formula: Among them, t represents the number of diffusion steps of the input, and the remaining parameters are determined by the hyperparameter α t It is calculated that the hyperparameter α t It is a value that changes linearly with the number of diffusion steps.

4. The sequence recommendation method based on diffusion model and attention mechanism according to claim 1, characterized in that: The step S3 includes embedding the user historical behavior sequence into a representation v 1:n―1 Input the information extraction network Guidance to obtain the user interest feature hinfo, and embed the obtained user interest feature hinfo and noise into a representation The diffusion step number t is predicted by denoising through the reverse denoising process of the diffusion model A high-precision Guide network is trained using gradient descent using the specified loss function.

5. The sequence recommendation method based on diffusion model and attention mechanism according to claim 1, characterized in that: The step S4 samples random noise ∈ from the Gaussian distribution by random sampling and maps it into a noise embedding representation μ through an embedding layer.

6. The sequence recommendation method based on diffusion model and attention mechanism according to claim 1, characterized in that: The step S6 further comprises: The random noise is embedded into the representation μ and the noise is reversed through the diffusion model. After the noise reduction process with the specified diffusion step number t, the recommended item sequence representation is obtained. The specific process is as follows: The denoising formula from step t to step t-1 is as follows: in, Represents a normal distribution, which is calculated by specifying the mean variance A sample is taken from a Gaussian distribution. Mean calculation formula: Variance calculation formula: where v 0 It is predicted by a Transformer-based noise prediction network, which calculates and predicts the input noise embedding representation μ, the user interest feature representation hinfo, and the current specific denoising step number t1. The specific prediction process is as follows: The prediction result consists of two parts. The first part is the prediction result of denoising the specified interest features of a specific user in the diffusion model denoising process through the user interest feature representation hinfo; the second part is the prediction result of denoising through random noise. The current prediction is obtained by fusion representation of the two parts of the prediction results. Iterate the above denoising process, repeatedly updating the user interest feature representation hinfo and the current prediction result in each prediction Finally, the recommended item sequence representation is obtained .

7. The sequence recommendation method based on diffusion model and attention mechanism according to claim 1, characterized in that: The step S7 further comprises: The recommended item sequence is represented as After passing through the fully connected layer, it is sent to a standard conditional random field CRF to perform the result mapping of the item sequence. The probability of the item sequence p(y|s; θ) is defined as: where y v Represents the recommended item sequence representation All possible tag sequences. Filter out the recommended item sequence representation The item sequence y with the highest conditional probability * As the final output: 。

Images