Sequence recommendation methods, systems, devices, and media based on attention mechanisms and persistent memory.

Abstract

This invention discloses a sequence recommendation method, system, device, and medium based on attention mechanisms and persistent memory, comprising: defining item categories within the system; collecting and processing sequence data within the system to obtain processed sequence data; constructing a sequence recommendation model, wherein the constructed sequence recommendation model includes an embedding generation module, a location information generation module, a self-attention module based on persistent memory, and a prediction module; training the sequence recommendation model using the processed sequence data as samples; and model deployment and prediction, wherein the model deployment and prediction involves inputting the target sequence data into the trained sequence recommendation model and predicting the next item based on the model output. This invention, by introducing a persistent memory mechanism, enables the model to focus on global cross-sample knowledge throughout the task and learn general knowledge of the task, which plays a significant role in improving recommendation performance.

Description

Technical Field

[0001] This invention belongs to the technical field of supervised deep learning and recommendation systems, and specifically designs a sequence recommendation method, system, device and medium based on attention mechanism and persistent memory. Background Technology

[0002] In recent years, artificial intelligence and related industries have been rapidly developing and expanding, becoming a focus of attention for academia, industry, and governments worldwide. In the field of sequence recommendation, methods based on self-attention mechanisms have made significant progress; however, most of these methods only focus on the information of the current input sequence, ignoring background knowledge across samples throughout the task. Therefore, how to combine attention mechanisms with general knowledge of the task in the sequence recommendation process remains a challenging problem to be solved. Summary of the Invention

[0003] The main objective of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide a sequence recommendation method based on attention mechanisms and persistent memory. By introducing a persistent memory mechanism, the model can focus on global cross-sample knowledge throughout the task and learn general knowledge of the task, which plays a significant role in improving recommendation performance.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] This invention provides a sequence recommendation method based on attention mechanisms and persistent memory, comprising the following steps:

[0006] Define the categories of recommended items in the system; collect and process the sequence data in the system to obtain the processed sequence data;

[0007] A sequence recommendation model is constructed, which includes an embedding generation module, a location information generation module, a self-attention module based on persistent memory, and a prediction module.

[0008] The processed sequence data is used as a sample to train the sequence recommendation model; the training sequence recommendation model specifically involves:

[0009] The sample is input into the embedding generation module, and the output of the embedding generation module is concatenated with the position information generated by the position information generation module to obtain the embedding matrix of the current input sample.

[0010] The embedding matrix of the sample is input into a self-attention module based on persistent memory to obtain the feature vector of the sample;

[0011] The feature vector of the sample is input into the prediction module to obtain the recommended prediction result. The cross-entropy between the prediction result and the real result is calculated as the objective function of the training model. The stochastic gradient descent method is used to train the sequence recommendation model, and finally the trained sequence recommendation model is obtained.

[0012] The trained sequence recommendation model is deployed and predicted by inputting the target sequence data into the trained sequence recommendation model and predicting the next item based on the model's output.

[0013] Preferably, the categories include movies, music, and advertising.

[0014] Preferably, in the sample training sequence recommendation model, the i-th training sample is x. i .

[0015] Preferably, the formula for calculating the embedding matrix of the current input sample is:

[0016]

[0017] in, Let x represent the embedding matrix of the current input sample. i1 x i2 ... x in Let P1, P2, ..., Pn be the 1st, 2nd, and nth items in the current sample, and M be the learnable embedding transformation matrix. n This represents the learnable position transformation matrix.

[0018] Preferably, the embedding matrix of the sample is input into a self-attention module based on persistent memory to obtain the feature vector of the sample; specifically:

[0019] First, the embedding matrix is ​​subjected to several different linear transformations to obtain the query matrix Q, the key matrix K, and the value matrix V, as shown in the following formulas:

[0020]

[0021] Q = qW Q

[0022] K = kW K

[0023] V = vW V

[0024] Among them, W Q W K W V Represents a linear operation matrix;

[0025] Then, the K and V matrices are respectively compared with the persistent memory matrix M. K M V K is obtained by performing a chain operation. * V * The matrix is ​​calculated using the following formula:

[0026]

[0027]

[0028] Finally, regarding Q and K * V * Attention weighting, normalization, and fully connected linear operations are performed to obtain the feature vector, as shown in the following formula:

[0029]

[0030]

[0031]

[0032]

[0033] Where S is the attention-weighted result, The result is the normalized result, where z is the result after the fully connected linear operation. For feature vectors, W1, W2, b1, and b2 are scaling factors, and they are all adaptively learnable parameters; LN() represents the layer normalization operation.

[0034] Furthermore, recursive skip connections are employed after attention computation and feedforward computation in the persistent memory-based self-attention module.

[0035] Preferably, the recommended prediction result is a feature vector. In the prediction module, a single fully connected layer and a Softmax operation are used to obtain the score of each item. The item with the highest score is taken as the prediction result of the next item in the current sequence.

[0036] Preferably, the training process also includes iterative optimization using gradient descent.

[0037] In another aspect, the present invention provides a sequence recommendation system based on attention mechanism and persistent memory, which is applied to the sequence recommendation method based on attention mechanism and persistent memory, including a preprocessing module, a model building module, a model training module and a model prediction module;

[0038] The preprocessing module is used to define the categories of items within the system; collect and process sequence data in the system to obtain processed sequence data;

[0039] The model building module is used to build a sequence recommendation model, which includes an embedding generation module, a location information generation module, a self-attention module based on persistent memory, and a prediction module.

[0040] The model training module is used to train a sequence recommendation model using the processed sequence data as samples; the trained sequence recommendation model specifically includes:

[0041] The sample is input into the embedding generation module, and the output of the embedding generation module is concatenated with the position information generated by the position information generation module to obtain the embedding matrix of the current input sample.

[0042] The embedding matrix of the sample is input into a self-attention module based on persistent memory to obtain the feature vector of the sample;

[0043] The feature vector of the sample is input into the prediction module to obtain the recommended prediction result. The cross-entropy between the prediction result and the real result is calculated as the objective function of the training model. The stochastic gradient descent method is used to train the sequence recommendation model, and finally the trained sequence recommendation model is obtained.

[0044] The model prediction module is used to deploy and predict the trained sequence recommendation model. Deploying and predicting the sequence recommendation model involves inputting the target sequence data into the trained sequence recommendation model and predicting the next item based on the model's output.

[0045] In another aspect, the present invention provides an electronic device, characterized in that the electronic device comprises:

[0046] At least one processor; and,

[0047] A memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform the sequence recommendation method based on attention mechanisms and persistent memory.

[0049] In another aspect, the present invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the sequence recommendation method based on attention mechanism and persistent memory.

[0050] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0051] 1. The sequence recommendation method based on attention mechanism and persistent memory proposed in this invention utilizes the self-attention mechanism to process sequence data, which has a strong ability to model data. By introducing a persistent memory mechanism, this sequence recommendation method enables the model to pay attention to global cross-sample knowledge in the entire task and learn general knowledge of the task, which plays a significant role in improving the recommendation effect.

[0052] 2. The sequence recommendation method proposed in this invention is based on autoregressive training, which has low training cost, strong robustness, and is simple to implement. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of the overall process of the sequence recommendation method based on attention mechanism and persistent memory as described in the embodiments of the present invention;

[0055] Figure 2 This is a schematic diagram of the training process of the recommendation model as described in an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the sequence recommendation system based on attention mechanism and persistent memory as described in an embodiment of the present invention;

[0057] Figure 4 This is a structural diagram of the electronic device according to an embodiment of the present invention;

[0058] Figure 5 This is a schematic diagram of the structure of the storage medium described in an embodiment of the present invention. Detailed Implementation

[0059] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0060] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0061] Sequence recommendation: Sequence recommendation recommends information or products to users based on their interests and behavioral characteristics, helping users quickly discover the products they truly need from a sea of ​​information, increasing user stickiness, and promoting information clicks and product sales.

[0062] Please see Figure 1 , Figure 2 One embodiment of this application provides a sequence recommendation method based on attention mechanisms and persistent memory, comprising the following steps:

[0063] S1. Define the categories of items within the system; collect and process sequence data in the system to obtain processed sequence data;

[0064] Furthermore, the categories include movies, music, and advertising.

[0065] S2. Construct a sequence recommendation model, which includes an embedding generation module, a location information generation module, a self-attention module based on persistent memory, and a prediction module;

[0066] S3. Use the processed sequence data as samples to train the sequence recommendation model; the training sequence recommendation model specifically includes:

[0067] The sample is input into the embedding generation module, and the output of the embedding generation module is concatenated with the position information generated by the position information generation module to obtain the embedding matrix of the current input sample.

[0068] The embedding matrix of the sample is input into a self-attention module based on persistent memory to obtain the feature vector of the sample;

[0069] The feature vector of the sample is input into the prediction module to obtain the recommended prediction result. The cross-entropy between the prediction result and the real result is calculated as the objective function of the training model. The stochastic gradient descent method is used to train the sequence recommendation model, and finally the trained sequence recommendation model is obtained.

[0070] S31. Let the i-th training sample be x. i The formula for calculating the embedding matrix of the current input sample is as follows:

[0071]

[0072] in, Let x represent the embedding matrix of the current input sample. i1 x i2 x in Let P1, P2, ..., Pn be the 1st, 2nd, and nth items in the current sample, and M be the learnable embedding transformation matrix. n This represents the learnable position transformation matrix.

[0073] S32. The embedding matrix of the sample is input into a self-attention module based on persistent memory to obtain the feature vector of the sample, specifically:

[0074] First, the embedding matrix is ​​subjected to several different linear transformations to obtain the query matrix Q, the key matrix K, and the value matrix V, as shown in the following formulas:

[0075]

[0076] Q = qW Q

[0077] K = kW K

[0078] V = vW V

[0079] Among them, W Q W K W V Represents a linear operation matrix;

[0080] Then, the K and V matrices are respectively compared with the persistent memory matrix M. K M V K is obtained by performing a chain operation. * V * The matrix is ​​calculated using the following formula:

[0081]

[0082]

[0083] Finally, regarding Q and K * V * Attention weighting, normalization, and fully connected linear operations are performed to obtain the feature vector, as shown in the following formula:

[0084]

[0085]

[0086]

[0087]

[0088] Where S is the attention-weighted result, The result is the normalized result, where z is the result after the fully connected linear operation. For feature vectors, W1, W2, b1, and b2 are scaling factors, and they are all adaptively learnable parameters. LN() represents the layer normalization operation.

[0089] In addition, in order to better facilitate the training of deep network models and improve performance, recursive skip connections are used in the self-attention module after attention calculation and feedforward calculation.

[0090] S4. Deploy and predict the trained sequence recommendation model. Deploying and predicting the sequence recommendation model involves inputting the target sequence data into the trained sequence recommendation model and predicting the next item based on the model's output.

[0091] S41, The recommended prediction result is a feature vector. In the prediction module, a single fully connected layer and a Softmax operation are used to obtain the score of each item. The item with the highest score is taken as the prediction result of the next item in the current sequence.

[0092] Specifically, the training process also includes iterative optimization using gradient descent.

[0093] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0094] Based on the same ideas as the sequence recommendation method based on attention mechanism and persistent memory in the above embodiments, the present invention also provides a sequence recommendation system based on attention mechanism and persistent memory, which can be used to execute the above-described sequence recommendation method based on attention mechanism and persistent memory. For ease of explanation, the structural diagrams of the sequence recommendation system embodiments based on attention mechanism and persistent memory only show the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structures do not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0095] Please see Figure 3In another embodiment of this application, a sequence recommendation system 100 based on attention mechanism and persistent memory is provided. The system includes a preprocessing module 101, a model building module 102, a model training module 103, and a model prediction module 104.

[0096] The preprocessing module 101 is used to define the categories of items in the system; collect and process the sequence data in the system to obtain the processed sequence data;

[0097] The model building module 102 is used to build a sequence recommendation model, which includes an embedding generation module, a location information generation module, a self-attention module based on persistent memory, and a prediction module.

[0098] The model training module 103 is used to train a sequence recommendation model using the processed sequence data as samples; the trained sequence recommendation model specifically includes:

[0099] The sample is input into the embedding generation module, and the output of the embedding generation module is concatenated with the position information generated by the position information generation module to obtain the embedding matrix of the current input sample.

[0100] The embedding matrix of the sample is input into a self-attention module based on persistent memory to obtain the feature vector of the sample;

[0101] The feature vector of the sample is input into the prediction module to obtain the recommended prediction result. The cross-entropy between the prediction result and the real result is calculated as the objective function of the training model. The stochastic gradient descent method is used to train the sequence recommendation model, and finally the trained sequence recommendation model is obtained.

[0102] The model prediction module 104 is used to deploy and predict the trained sequence recommendation model. Deploying and predicting the sequence recommendation model involves inputting the target sequence data into the trained sequence recommendation model and predicting the next item based on the model output.

[0103] It should be noted that the sequence recommendation system based on attention mechanism and persistent memory of the present invention corresponds one-to-one with the sequence recommendation method based on attention mechanism and persistent memory of the present invention. The technical features and beneficial effects described in the embodiments of the sequence recommendation method based on attention mechanism and persistent memory described above are applicable to the embodiments of the sequence recommendation system based on attention mechanism and persistent memory. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0104] Furthermore, in the above embodiments of the sequence recommendation system based on attention mechanism and persistent memory, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or for the convenience of software implementation. That is, the internal structure of the sequence recommendation system based on attention mechanism and persistent memory is divided into different program modules to complete all or part of the functions described above.

[0105] Please see Figure 4 In one embodiment, an electronic device is provided that implements a sequence recommendation method based on attention mechanism and persistent memory. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a sequence recommendation program 203 based on attention mechanism and persistent memory.

[0106] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as the portable hard drive of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 200. Furthermore, the first memory 202 can include both internal and external storage units of the electronic device 200. The first memory 202 can be used not only to store application software and various types of data installed on the electronic device 200, such as the code of the sequence recommendation program 203 based on attention mechanisms and persistent memory, but also to temporarily store data that has been output or will be output.

[0107] In some embodiments, the first processor 201 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions of the electronic device 200 and process data.

[0108] Figure 4 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 4 The structure shown does not constitute a limitation on the electronic device 200, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0109] The sequence recommendation program 203 based on attention mechanisms and persistent memory stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions, which, when run in the first processor 201, can achieve the following:

[0110] Define the categories of items within the system; collect and process sequence data within the system to obtain processed sequence data;

[0111] A sequence recommendation model is constructed, which includes an embedding generation module, a location information generation module, a self-attention module based on persistent memory, and a prediction module.

[0112] The processed sequence data is used as a sample to train the sequence recommendation model; the training sequence recommendation model specifically involves:

[0113] The sample is input into the embedding generation module, and the output of the embedding generation module is concatenated with the position information generated by the position information generation module to obtain the embedding matrix of the current input sample.

[0114] The embedding matrix of the sample is input into a self-attention module based on persistent memory to obtain the feature vector of the sample;

[0115] The feature vector of the sample is input into the prediction module to obtain the recommended prediction result. The cross-entropy between the prediction result and the real result is calculated as the objective function of the training model. The stochastic gradient descent method is used to train the sequence recommendation model, and finally the trained sequence recommendation model is obtained.

[0116] The trained sequence recommendation model is deployed and predicted by inputting the target sequence data into the trained sequence recommendation model and predicting the next item based on the model's output.

[0117] Please see Figure 5 Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0118] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A sequence recommendation method based on attention mechanism and persistent memory, characterized in that, It comprises the following steps: defining a category of system recommended items; collecting and processing sequence data in the system to obtain processed sequence data; the category comprises movies, music and advertisements; constructing a sequence recommendation model, which comprises an embedding generation module, a position information generation module, a self-attention module based on persistent memory and a prediction module; a recursive jump connection is used after attention calculation and feedforward calculation in the self-attention module based on persistent memory; training the sequence recommendation model with the processed sequence data as samples; the training of the sequence recommendation model specifically comprises: inputting the samples into the embedding generation module, performing vector concatenation operation on the output result of the embedding generation module and the position information generated by the position information generation module to obtain an embedding matrix of the current input sample; inputting the embedding matrix of the sample into the self-attention module based on persistent memory to obtain a feature vector of the sample; The feature vector of the sample is input into a prediction module to obtain a recommended prediction result, cross entropy between the prediction result and a true result is calculated as a target function of a training model, a random gradient descent method is used to train the sequence recommendation model, and finally a trained sequence recommendation model is obtained; the recommended prediction result is a feature vector In the prediction module, a single-layer full connection layer and a Softmax operation are used to obtain a score of each item, and an item with the highest score is taken as a prediction result of the next item of the current sequence. deploying and predicting the trained sequence recommendation model, wherein the deployment and prediction of the sequence recommendation model comprises inputting target sequence data into the trained sequence recommendation model and predicting the next item according to the model output result. 2.The sequence recommendation method based on attention mechanism and persistent memory according to claim 1, characterized in that, The sample training sequence recommendation model takes the first training sample as . 3.The sequence recommendation method based on attention mechanism and persistent memory according to claim 1, characterized in that, The calculation formula of the embedding matrix of the current input sample is: wherein, denotes the embedding matrix of the current input sample, , , , denotes the 1st, 2nd, n-th item in the current sample, denotes the learnable embedding transformation matrix, , , , denotes the learnable position transformation matrix. 4.The method of claim 3, wherein, the embedding matrix of the sample is input into the self-attention module based on persistent memory to obtain a feature vector of the sample; specifically: first, the embedding matrix is subjected to multiple different linear transformations to obtain a query matrix Q, a key matrix K and a value matrix V, and the specific formula is: wherein , , denotes a linear operation matrix; Then, the K, V matrix respectively with the persistent memory matrix , series operation to get matrix, the formula is: Finally, the Q, is attention weighted, normalized and fully connected linearly to obtain the feature vector, and the specific formula is: wherein, is the result of attention weighting, is the result of normalization, is the result of fully connected linear operation, is the feature vector, is the scaling factor, , , , are all parameters that can be self-adaptively learned; denotes the layer normalization operation.

5. The sequence recommendation method based on attention mechanism and persistent memory according to claim 1, characterized in that, In the training process, gradient descent method is also used for iterative optimization.

6. A sequence recommendation system based on attention mechanism and persistent memory, characterized in that, The sequence recommendation method based on attention mechanism and persistent memory is applied to any one of claims 1-5, comprising a preprocessing module, a model construction module, a model training module and a model prediction module; the preprocessing module is used to define the category of items in the system; collect and process sequence data in the system to obtain processed sequence data; the model construction module is used to construct a sequence recommendation model, which comprises an embedding generation module, a position information generation module, a self-attention module based on persistent memory and a prediction module; the model training module is used to train the sequence recommendation model with the processed sequence data as samples; the training of the sequence recommendation model specifically comprises: inputting the samples into the embedding generation module, performing vector concatenation operation on the output result of the embedding generation module and the position information generated by the position information generation module to obtain an embedding matrix of the current input sample; inputting the embedding matrix of the sample into the self-attention module based on persistent memory to obtain a feature vector of the sample; inputting the feature vector of the sample into the prediction module to obtain a predicted result of recommendation, calculating the cross-entropy between the predicted result and the true result as the objective function of the trained model, using the stochastic gradient descent method to train the sequence recommendation model, and finally obtaining the trained sequence recommendation model; The model prediction module is configured to deploy and predict the trained sequence recommendation model by inputting target sequence data into the trained sequence recommendation model and predicting the next item according to a model output result.

7. An electronic device, comprising: The electronic device includes: at least one processor; and a memory connected to the at least one processor in communication; wherein The memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the sequence recommendation method based on the attention mechanism and the persistent memory as claimed in any one of claims 1-5.

8. A computer-readable storage medium storing a program, characterized in that, The program is executed by the processor to implement the sequence recommendation method based on the attention mechanism and the persistent memory as claimed in any one of claims 1-5.

Images