Offline item recommendation method, device, equipment and storage medium

Abstract

The present invention provides an offline item recommendation method, device, equipment and storage medium, which relates to the field of artificial intelligence technology. The method includes: obtaining the user's interactive item sequence at the current moment; using the trained first action value network to calculate the action value of all items in the interactive item sequence; the action value represents the user's preference for the item; the first action value network is obtained by training in an offline environment; and the item corresponding to the largest action value among the action values ​​of all items is determined as the recommended item. The method provided by the present invention has a high recommendation efficiency, and because the first action value network is trained in an offline environment, that is, the first action value network can be trained without interacting with the real recommendation environment, the recommendation results of the trained first action value network are more accurate and the user experience is better.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an offline item recommendation method, device, equipment and storage medium. Background Art

[0002] In Internet services, users access items or projects in chronological order, where the items a user is about to interact with may be closely related to those he has visited before. This property promotes an important recommendation task - Sequential Recommendation (SR), which regards user historical behaviors as a sequence of actions ordered by time.

[0003] In the prior art, a large part of deep reinforcement learning recommendation algorithms use online reinforcement learning. This type of method needs to be trained in an environment, where the purpose of the environment is to simulate user feedback on recommended items. The quality of the environment has a great impact on the performance of the algorithm, and users' changeable behaviors and dynamically changing interests are also difficult to simulate accurately. In addition, this type of method needs to first train a model to be used as the environment, and then train the algorithm in the environment. It is a two-stage method with poor recommendation performance. Therefore, it is usually trained directly in real recommendation scenarios. However, training directly in recommendation scenarios takes a long time, which prolongs the response time for recommending items to users, affecting user experience. Summary of the invention

[0004] The present invention provides an offline item recommendation method, device, equipment and storage medium, which are used to solve the defect of poor user experience in the existing online reinforcement learning technology, optimize the learning strategy without interacting with the real recommendation environment, and improve the user experience.

[0005] In a first aspect, the present invention provides an offline item recommendation method, comprising:

[0006] Get the user's interactive item sequence at the current moment;

[0007] The trained first action value network is used to calculate the action value of all items in the interactive item sequence; the action value represents the user's preference for the item; the first action value network is trained in an offline environment;

[0008] The item corresponding to the largest action value among all the action values ​​of the items is determined as the recommended item.

[0009] Optionally, before calculating the action values ​​of all items in the interactive item sequence using the trained first action value network, the method may further include:

[0010] Input the item interaction sequence at time t into the first action value network to obtain the action value corresponding to the item to be recommended; t is an integer greater than or equal to 0;

[0011] Input the item interaction sequence at time t+1 into the first action value network, and obtain the item corresponding to the largest action value among the action values ​​output by the first action value network;

[0012] Based on the second action value network, determine the target action value of the item corresponding to the maximum action value;

[0013] Determine the target loss function based on the target action value, the action value corresponding to the item to be recommended, the action value corresponding to all items in the item interaction sequence at time t, and the action value of the negative sample;

[0014] Based on the target loss function and the preset learning rate, the first parameters corresponding to the first action value network and the second action value network are updated.

[0015] Optionally, determining the target loss function based on the target action value, the action value corresponding to the item to be recommended, the action value corresponding to all items in the item interaction sequence at time t, and the action value of the negative sample may include:

[0016] Calculate the difference loss value based on the target action value and the action value corresponding to the recommended item;

[0017] Calculate the first loss function based on the action value corresponding to the item to be recommended and the action value corresponding to all items in the item interaction sequence at time t;

[0018] Calculate the second loss function based on the action value corresponding to the item to be recommended and the action value of the negative sample;

[0019] Based on the differential loss value, the first loss function, and the second loss function, a target loss function is determined.

[0020] Optionally, based on the action value corresponding to the item to be recommended and the action value of the negative sample, before calculating the second loss function, the method may further include:

[0021] Collect multiple negative samples based on the action values ​​of all items output by the first action value network; negative samples are items that the user has not interacted with based on the negative samples at time t;

[0022] The second loss function is calculated based on the action value corresponding to the item to be recommended and the action value of the negative sample, specifically including:

[0023] Based on the action value corresponding to the item to be recommended and the action value of the negative sample, the second loss function is calculated in the form of positive and negative sample pairs.

[0024] Optionally, updating the first parameters corresponding to the first action value network and the second action value network may include:

[0025] Updating a first parameter of the first action value network to obtain an updated first parameter of the first action value network;

[0026] Based on the updated first parameter and the second parameter, the first parameter of the second action value network is updated to obtain the updated first parameter of the second action value network; the second parameter is used to control the update amplitude of the first parameter of the second action value network.

[0027] Optionally, the structures of the first action value network and the second action value network include an embedding representation layer, a state representation layer and an output layer, wherein the state representation layer is connected to the embedding representation layer and the output layer respectively; wherein,

[0028] The embedding representation layer is used to assign a corresponding sequence vector to each item in the interactive item sequence;

[0029] The state representation layer is used to convert the sequence vector into a hidden state vector; the hidden state vector and the sequence vector have the same dimension;

[0030] The output layer is used to map the hidden state vector to the item space and output the action value of all items.

[0031] In a second aspect, the present invention provides an offline item recommendation device, comprising:

[0032] The acquisition module is used to obtain the user's interactive item sequence at the current moment;

[0033] A calculation module is used to calculate the action value of all items in the interactive item sequence using the trained first action value network; the action value indicates the user's preference for the item; the first action value network is trained in an offline environment;

[0034] The determination module is used to determine the item corresponding to the largest action value among all the action values ​​of the items as the recommended item.

[0035] In a third aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the offline item recommendation method as described above is implemented.

[0036] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described offline item recommendation methods.

[0037] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned offline item recommendation methods.

[0038] The present invention provides an offline item recommendation method, device, equipment and storage medium, which obtains the user's interactive item sequence at the current moment, and then uses the trained first action value network to calculate the action value of all items in the interactive item sequence, wherein the action value represents the user's preference for the item, and the first action value network is trained in an offline environment. Finally, the item corresponding to the largest action value among the action values ​​of all items is determined as the recommended item. In the method provided by the present invention, based on the first action value network trained in an offline environment, the user's interactive item sequence at the current moment is calculated to obtain the action value of all items in the interactive item sequence, so as to recommend items to the user according to the size of the action value of all items. The recommendation efficiency is high, and because the first action value network is trained in an offline environment, that is, the first action value network can be trained without interacting with the real recommendation environment, the recommendation results of the trained first action value network are more accurate, and the user experience is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 This is one of the flow charts of the offline item recommendation method provided by the present invention;

[0041] Figure 2 This is the second flow chart of the offline item recommendation method provided by the present invention;

[0042] Figure 3 It is a schematic diagram of the structure of the first action value network and the second action value network provided by the present invention;

[0043] Figure 4 is a schematic diagram of the structure of the offline item recommendation device provided by the present invention;

[0044] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0047] First, the relevant concepts involved in the embodiments of the present invention are introduced:

[0048] Markov decision process refers to the decision maker periodically or continuously observing a random dynamic system with Markov properties and making decisions sequentially. That is, according to the state observed at each moment, an action is selected from the available action set to make a decision. The next state of the system (future) is random, and its state transition probability has Markov properties. The decision maker can make a new decision based on the newly observed state, and repeat this process repeatedly. Markov decision process can also be used as Markov type stochastic optimal control, and its decision variable is the control variable.

[0049] Markov property refers to the property that the probability law of the future development of a random process is independent of the history before observation. Markov property can be simply described as the lack of aftereffect of the state transition probability. A random process with Markov property of state transition probability is called Markov process. Markov decision process can also be regarded as a special case of random countermeasures, in which one party of the countermeasures is without will.

[0050] Secondly, the application scenario of the offline item recommendation method provided by the embodiment of the present invention is introduced.

[0051] There are two main types of early sequence recommendation technologies: collaborative filtering-based technology and Markov chain-based technology. Among them, collaborative filtering-based technology is divided into two categories: user-based and item-based. This type of method uses the similarity between users or items to recommend items, and does not consider the correlation between items in the item sequence; Markov chain-based technology models the user-item interaction sequence as a Markov decision process, emphasizing the modeling of user interaction sequences and predicting recommended items by mining the correlation of interactive item sequences.

[0052] With the rapid development of deep learning, sequence recommendation technology based on deep learning has become mainstream. This type of technology usually uses network structures such as convolutional neural networks, recurrent neural networks, transformer networks, and graph neural networks. Deep neural networks have strong fitting capabilities and can learn the characteristics of user-item interaction sequences, and then mine sequence correlations and complete sequence modeling to predict the next recommended item. Furthermore, reinforcement learning is an effective method to solve the Markov decision process, and the sequence recommendation problem can be modeled as a Markov decision process. With the success of deep reinforcement learning, recommendation algorithms based on deep reinforcement learning have once again become a research hotspot.

[0053] The item recommendation method in the prior art has the following shortcomings:

[0054] The sequence recommendation technology based on collaborative filtering cannot mine the correlation between items in the sequence, cannot effectively model the sequence correlation, and ignores the changes in user preferences during the interaction process.

[0055] Sequential recommendation technology based on deep learning technology belongs to supervised learning methods. Such methods usually model the sequence recommendation problem as a multi-classification problem, and predict the next recommended item or project by learning the existing interaction sequence. Such methods have short-sighted problems and usually only focus on users' preferences in the short term, while ignoring users' long-term benefits.

[0056] The sequential recommendation technology based on online reinforcement learning needs to be trained in a real recommendation environment, but training directly in the recommendation scenario is costly and may have a negative impact on the user experience.

[0057] In summary, the problems existing in existing sequence recommendation technologies are as follows:

[0058] (1) Long-term benefits in the sequence recommendation problem are difficult to model. The sequence recommendation problem is actually a sequence decision problem. Recommending items or projects to users is regarded as making decisions for users. The recommendation model decides which items or projects should be recommended based on the current interaction sequence. Existing deep learning sequence recommendation algorithms do not view this process from a decision-making perspective, but usually try to solve it as a multi-classification problem, ignoring long-term benefits;

[0059] (2) The existing reinforcement learning recommendation algorithms suffer from poor recommendation performance and user experience. A large portion of the existing reinforcement learning recommendation algorithms use online reinforcement learning. This type of method relies on the establishment of a simulation environment. The purpose of the environment is to simulate user feedback on recommended items. The quality of the environment has a great impact on the performance of the algorithm. Users’ changing behaviors and changing interests are also difficult to simulate accurately. Moreover, this type of method requires first training a model to be used as an environment, and then training the algorithm in the environment. It is a two-stage method with poor recommendation performance.

[0060] (3) Lack of attention to the implicit feedback problem in item recommendation. In the recommendation scenario, there are two types of feedback: explicit feedback and implicit feedback. User ratings of items belong to explicit feedback. In this type of feedback, the user's interest is relatively clear. A high score means liking, while a low score means the opposite. Implicit feedback includes: buying, adding to shopping carts, clicking, skipping, etc. Users do not express clear likes or dislikes for the items they browsed. In practical applications, implicit feedback in recommendation scenarios is easier to obtain, and there are negative samples in the explicit feedback data, that is, low-scoring samples, while implicit feedback lacks negative samples.

[0061] Based on the above shortcomings, the present invention proposes a technical solution for offline item recommendation that can complete training in offline data sets, optimize long-term benefits while capturing dynamic changes in user preferences, and introduces a dynamic negative sampling strategy to further optimize the action value network of reinforcement learning to achieve better recommendation performance.

[0062] In this invention, the sequence recommendation problem is converted into a Markov decision process, and then the offline reinforcement learning technology is used to solve the sequence decision problem. The user is regarded as the environment, and the sequence recommendation algorithm can be regarded as an intelligent agent. The action performed by the intelligent agent is the recommended items for the user. First, the modeling of the Markov decision process is introduced, which is described as follows:

[0063] State Space Status at time t The latent vector representation is defined as the user-item interaction sequence at time t obtained after passing through the state representation layer 320 of the first action value network and / or the second action value network. The first action value network and the second action value network can be constructed based on the strategy-value Actor-Critic reinforcement learning algorithm.

[0064] Item Space That is, the action space, the recommended items at time t Refers to the items recommended to the user at time t.

[0065] award The reward the agent receives after outputting a recommended item

[0066] Transition probability The probability that the agent will transition to state s′ after taking action a in state s.

[0067] Discount rate γ: γ∈[0,1], discounts future rewards. When γ=0, the agent is short-sighted and only considers the next reward. When γ=1, the agent treats rewards at all times equally.

[0068] Given The goal of the agent is to find a policy π to maximize the expected cumulative reward:

[0069]

[0070] Among them, T is the maximum moment of the entire decision-making process.

[0071] Combine the following Figure 1-Figure 5 The technical solution for offline item recommendation of the present invention is described in detail. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0072] Figure 1 One of the flowcharts of the offline item recommendation method provided by the embodiment of the present invention is as follows: Figure 1 As shown, the method may include:

[0073] Step 101: Obtain the user's interactive item sequence at the current moment;

[0074] Step 102: Calculate the action value of all items in the interactive item sequence using the trained first action value network; the action value represents the user's preference for the item; the first action value network is trained in an offline environment;

[0075] Step 103: Determine the item corresponding to the largest action value among all the action values ​​of the items as the recommended item.

[0076] Specifically, the interactive item sequence obtained in step 101 may be the interactive item sequence of the user at the current moment, for example, the interactive item sequence at moment t:

[0077] i 1:t ={i1,i2…,i t},

[0078] Where i represents an item, and the subscript represents the interaction between the user and the item at the time when the subscript is used as the moment. Assuming that there are N items in total, integers from 1 to N are assigned to all items, and each item is represented by a unique number. It can be understood that the interactive item sequence can include one or more interactive sequences of items, which is not limited in this embodiment.

[0079] Optionally, the lengths of the interaction item sequences of the above-mentioned items can be unified, for example, unified to a length of l. Specifically, the interaction item sequence with insufficient length can be filled with element 0, and the sequence with a length exceeding l can intercept the l most recent items and input them into the trained first action value network.

[0080] Furthermore, in step 102, the first action value network trained is used to calculate the interactive item sequence at the current moment, and the action value of each item in the interactive item sequence at the current moment can be obtained, wherein the action value represents the user's preference for the item, for example, the action value can be the user's rating of the item, a high rating means that the user likes the item, and conversely, a low rating means that the user does not like the item. The first action value network is obtained by training and reinforcement learning in an offline environment.

[0081] Further, the action value of each item in the interactive item sequence obtained in step 102 at the current moment is sorted to obtain a plurality of action values ​​arranged in order from high to low, and then the item corresponding to the largest action value among the action values ​​of all items can be determined as the recommended item;

[0082] Optionally, based on actual needs, items corresponding to multiple action values ​​ranked high in the action value ranking may be used as recommended items. For example, if the user needs to obtain three recommended items, the items corresponding to the top three action values ​​among all the items may be used as recommended items.

[0083] In the method provided in this embodiment, the interactive item sequence of the user at the current moment is obtained, and then the action value of all items in the interactive item sequence is calculated using the trained first action value network, wherein the action value represents the user's preference for the item, and the first action value network is trained in an offline environment. Finally, the item corresponding to the largest action value among the action values ​​of all items is determined as the recommended item. In the method provided by the present invention, based on the first action value network trained in an offline environment, the interactive item sequence of the user at the current moment is calculated to obtain the action value of all items in the interactive item sequence, thereby recommending items to the user in order of the size of the action value of all items, with high recommendation efficiency. Moreover, since the first action value network is trained in an offline environment, that is, the first action value network can be trained without interacting with the real recommendation environment, the recommendation results of the trained first action value network are more accurate, and the user experience is better.

[0084] Optionally, before calculating the action values ​​of all items in the interactive item sequence using the trained first action value network, the method may further include:

[0085] Input the item interaction sequence at time t into the first action value network to obtain the action value corresponding to the item to be recommended;

[0086] Input the item interaction sequence at time t+1 into the first action value network, and obtain the item corresponding to the largest action value among the action values ​​output by the first action value network;

[0087] Based on the second action value network, determine the target action value of the item corresponding to the maximum action value;

[0088] Determine the target loss function based on the target action value, the action value corresponding to the item to be recommended, the action value corresponding to all items in the item interaction sequence at time t, and the action value of the negative sample;

[0089] Based on the target loss function and the preset learning rate, the first parameters corresponding to the first action value network and the second action value network are updated.

[0090] Specifically, before step 102, the method may first train the first action value network, and the training process is as follows:

[0091] First, the item interaction sequence at time t is input into the first action value network to obtain the action value corresponding to the item to be recommended, where t is an integer greater than or equal to 0; for example, the item interaction sequence at time t, such as i 1:t ={i1,i2…,i t} is input into the first action value network to obtain the recommended item a t (i.e., the real i in the data set t+1 ) corresponds to the action value Q θ (s t ,a t ), it can be understood that the data set is a data set used for training;

[0092] Secondly, the item interaction sequence at time t+1 is input into the first action value network to obtain the item corresponding to the largest action value among the action values ​​output by the first action value network. For example, the interaction sequence i at time t+1 is 1:t+1 ={i1,i2…,i t+1} is input into the first action value network, and the item corresponding to the largest action value in the output of the first action value network is selected as a t+1 ,in,

[0093] a t+1 = argmaxQ θ (s t+1 ,a) (2)

[0094] Then, the item corresponding to the maximum action value is input into the second action value network to determine the target action value of the item corresponding to the maximum action value, for example, Q θ′ (s t+1 ,a t+1 );

[0095] Furthermore, we can use the target action value, such as Q θ′ (s t+1 ,a t+1 ), the action value corresponding to the recommended item, such as Q θ (s t ,a t ), the action values ​​corresponding to all items in the item interaction sequence at time t, for example And the action value of negative samples such as Determine the target loss function.

[0096] Furthermore, based on the determined target loss function, a learning rate can be preset in advance, for example, α is the learning rate, and a stochastic gradient descent method is used to update the first parameters corresponding to the first action value network and the second action value network, wherein the first parameter corresponding to the first action value network is θ, and the first parameter corresponding to the second action value network is θ′, until the network converges.

[0097] Optionally, after the first action value network training is completed, the network performance can also be evaluated offline. The offline evaluation process is as follows:

[0098] a. The interactive item sequence of user u at time t Input into the first action value network;

[0099] b. Use the first action value network to calculate the action values ​​corresponding to all items in the interactive item sequence;

[0100] c. Determine the item corresponding to the maximum action value as the recommended item, for example:

[0101]

[0102] d. Calculate network evaluation indicators to detect the recommendation performance of the network.

[0103] Specifically, examples of network evaluation indicators are as follows:

[0104] Metric 1: Click-through rate (HR):

[0105]

[0106] Metric 2: Normalized Discounted Cumulative Gain (NDCG):

[0107]

[0108] Metric 3: Mean Reciprocal Rank (MRR):

[0109]

[0110] Among them, j represents the real item number, S represents the recommended item list generated by the first action value network, and k represents the ranking of the test data in the recommendation list.

[0111] The above three evaluation indicators can be calculated for each piece of data, and then the average value of all test data is taken to evaluate the performance of the network.

[0112] In the method provided in this embodiment, the action value corresponding to the item to be recommended is obtained by inputting the item interaction sequence at time t into the first action value network, and the item interaction sequence at time t+1 is input into the first action value network to obtain the item corresponding to the largest action value among the action values ​​output by the first action value network, and based on the second action value network, the target action value of the item corresponding to the largest action value is determined, and then the target loss function is determined based on the target action value, the action value corresponding to the item to be recommended, the action value corresponding to all items in the item interaction sequence at time t, and the action value of the negative sample, and then, based on the target loss function and the preset learning rate, the first parameters corresponding to each of the first action value network and the second action value network are updated, so that the first action value network and the second action value network converge quickly, optimize long-term benefits while capturing dynamic changes in user preferences, optimize the action value network of the reinforcement learning agent, and further use the trained first action value network for offline item recommendation to achieve better recommendation performance.

[0113] Optionally, determining the target loss function based on the target action value, the action value corresponding to the item to be recommended, the action value corresponding to all items in the item interaction sequence at time t, and the action value of the negative sample may include:

[0114] Calculate the difference loss value based on the target action value and the action value corresponding to the recommended item;

[0115] Calculate the first loss function based on the action value corresponding to the item to be recommended and the action value corresponding to all items in the item interaction sequence at time t;

[0116] Calculate the second loss function based on the action value corresponding to the item to be recommended and the action value of the negative sample;

[0117] Based on the differential loss value, the first loss function, and the second loss function, a target loss function is determined.

[0118] Specifically, the process of calculating the target loss function can first calculate the differential loss value, the first loss function, and the second loss function, and then accumulate the differential loss value, the first loss function, and the second loss function to obtain the target loss function. An example of the loss function calculation process is as follows:

[0119] First, based on the target action value and the action value corresponding to the recommended item, the difference loss value is calculated. For example, the difference loss value can be calculated according to the following formula (7):

[0120]

[0121] Among them, Q θ (s t ,a t ) represents the action value corresponding to the item to be recommended, Q θ′ (s t+1 ,a t+1 ) represents the target action value, γ represents the discount rate, r(s t ,a t ) represents the reward received by the agent after outputting the recommended item;

[0122] Secondly, based on the action value corresponding to the item to be recommended and the action value corresponding to all items in the item interaction sequence at time t, the first loss function is calculated, for example, the loss function of conservative action value learning, which is calculated according to the following formula (8):

[0123]

[0124] in, represents the expected action value of all items in the item interaction sequence at time t, Q θ (s t ,a t ) represents the action value corresponding to the item to be recommended;

[0125] Furthermore, the second loss function can be calculated based on the action value corresponding to the item to be recommended and the action value of the negative sample. It can be understood that the positive sample is the item to be recommended in the data set. t , i.e. i t+1 , combined with the negative samples obtained by negative sampling (Negative samples are items that users have not interacted with based on negative samples at time t), the second loss function can be based on positive samples, i.e., items to be recommended at The action value Q θ (s t ,a t ) and the negative samples obtained by dynamic negative sampling Action Value The second loss function is calculated by using positive and negative sample pairs, such as bpr loss;

[0126] Furthermore, the above differential loss value, the first loss function, and the second loss function may be added together to jointly determine the target loss function. For example, the target loss function may be determined by referring to the following formula (9):

[0127] Loss=TD error+cql loss+bpr loss (9)

[0128] Among them, Loss represents the target loss function, TD error represents the differential loss value, cql loss represents the first loss function, and bpr loss represents the second loss function.

[0129] In the method provided in this embodiment, during the training of the first action value network, the target loss function is jointly determined by the differential loss value, the first loss function, and the second loss function, wherein the differential loss value is calculated based on the target action value and the action value corresponding to the item to be recommended, the first loss function is calculated based on the action value corresponding to the item to be recommended and the action value corresponding to all items in the item interaction sequence at time t, and the second loss function is calculated based on the action value corresponding to the item to be recommended and the action value of the negative sample. The first action value network is updated using the target loss function, so that the sorting performance of the trained first action value network is improved and the recommendation performance is better.

[0130] Optionally, based on the action value corresponding to the item to be recommended and the action value of the negative sample, before calculating the second loss function, the method may further include:

[0131] Collect multiple negative samples based on the action values ​​of all items output by the first action value network; negative samples are items that the user has not interacted with based on the negative samples at time t;

[0132] The second loss function is calculated based on the action value corresponding to the item to be recommended and the action value of the negative sample, specifically including:

[0133] Based on the action value corresponding to the item to be recommended and the action value of the negative sample, the second loss function is calculated in the form of positive and negative sample pairs.

[0134] Specifically, it can be based on the output Q of the first action value network θ(s t ,·), that is, according to the state s t When , M negative samples are sampled from the action values ​​of all items. For example, a certain amount of items are sampled from items that the user has not interacted with (that is, items with negative action values) as negative samples. After sampling, positive samples, negative samples and scores are output simultaneously in the first action value network, and the score difference is optimized.

[0135] Furthermore, the final set of negative samples collected can be expressed as I represents the item space, i.e., the action space, and u represents the user who generated the current sequence. For the state-action pair (s t ,a t ), the probability of an item being collected as a negative sample is as follows:

[0136]

[0137] The above formula means: if the item belongs to i 1:t+1 , then it will not be selected as a negative sample; on the contrary, if it does not belong to i 1:t+1 , then the probability of the item being selected as a negative sample is an exponential function with e as the base and its action value as the exponent.

[0138] The negative samples i collected above m Recorded as

[0139]

[0140] in, represents the collected negative samples, and M represents the number of negative samples;

[0141] Furthermore, the second loss function can be based on the positive sample, i.e., the item to be recommended a t The action value Q θ (s t ,a t ) and the negative samples obtained by dynamic negative sampling Action Value It is calculated in the form of positive and negative sample pairs, for example, by referring to the following formula (12):

[0142]

[0143] Among them, Q θ (s t ,a t ) represents the action value of the positive sample, represents the action value of negative samples, and M represents the number of negative samples.

[0144] In the method provided in this embodiment, by introducing dynamic negative sampling technology, samples with too high action value functions obtained by offline reinforcement learning are collected as negative samples, and a second loss function such as Bayesian personalized sorting is introduced to construct a sorting sampling loss function at the level of positive and negative samples, so that the action value gap between positive and negative samples is increased, the action value of positive samples is larger, and the action value of negative samples is smaller, which solves the implicit feedback problem in item recommendation, can optimize the sorting performance of the model, and achieve better recommendation performance.

[0145] Optionally, updating the first parameters corresponding to the first action value network and the second action value network may include:

[0146] Updating a first parameter of the first action value network to obtain an updated first parameter of the first action value network;

[0147] Based on the updated first parameter and the second parameter, the first parameter of the second action value network is updated to obtain the updated first parameter of the second action value network; the second parameter is used to control the update amplitude of the first parameter of the second action value network.

[0148] Specifically, the first parameter of the first action value network, such as θ, and the first parameter of the second action value network, such as θ′, can be updated by using, for example, stochastic gradient descent (SGD). Specifically, the first parameter θ of the first action value network can be updated by using stochastic gradient descent. We can preset a learning rate in advance, such as α. The process of updating the first parameter of the first action value network is as follows:

[0149]

[0150] Among them, Loss represents the target loss function, α is the learning rate, and θ represents the first parameter of the first action value network.

[0151] Furthermore, the first parameter of the second action value network can be updated based on the updated first parameter and the second parameter to obtain the updated first parameter of the second action value network, wherein the second parameter, such as τ, is used to control the update amplitude of the first parameter of the second action value network, so that the update speed of the first parameter corresponding to the second target action value network can lag behind the update of the first parameter corresponding to the first action value network.

[0152] Exemplarily, the process of updating the first parameter of the second action-value network is as follows:

[0153] θ′=τθ+(1-τ)θ′ (14)

[0154] Among them, 0<τ<1; θ′ represents the first parameter of the second action value network, θ represents the first parameter of the first action value network, and τ represents a value between 0 and 1.

[0155] In the method provided in this embodiment, the first parameter of the first action value network is updated first, and then the first parameter of the second action value network is updated using the updated first parameter of the first action value network, so that the update speed of the first parameter of the second action value network lags behind the update of the first parameter of the first action value network, and finally the second action value network fits the strategy of reinforcement learning. By introducing the second action value target network, the action value of the item can be obtained from the second action value network, and then the action value function is updated. After a period of time, the new action value function is used to update the second action value network, which can improve the stability of reinforcement learning and achieve improved recommendation performance.

[0156] Figure 2 The second flowchart of the offline item recommendation method provided by the embodiment of the present invention is as follows: Figure 2 As shown, the method may include:

[0157] Step 201: Input the interactive item sequence at time t into the first action value network to obtain the action values ​​corresponding to all items in the item interactive sequence at time t;

[0158] Step 202: Input the interactive item sequence at time t+1 into the first action value network, and obtain the item corresponding to the maximum action value at time t+1 output by the first action value network;

[0159] Step 203: input the item corresponding to the maximum action value into the second action value network to determine the target action value of the item corresponding to the maximum action value;

[0160] Step 204: Calculate the difference loss value based on the target action value and the action value corresponding to the item to be recommended, in combination with the reward received after outputting the recommended item;

[0161] Step 205: Input the item interaction sequence at time t into the first action value network to obtain the action value corresponding to the item to be recommended; t is an integer greater than or equal to 0;

[0162] Step 206: Calculate a first loss function (conservative action value learning loss function) based on the action value corresponding to the item to be recommended and the action values ​​corresponding to all items in the item interaction sequence at time t.

[0163] Step 207: Collect multiple negative samples based on the action values ​​of all items output by the first action value network;

[0164] Step 208: Calculate a second loss function (Bayesian personalized ranking loss function) based on the action value corresponding to the item to be recommended and the action value of the negative sample;

[0165] Step 209: Obtain a target loss function based on the differential loss value, the first loss function, and the second loss function.

[0166] Specifically, this embodiment can be understood as a network training process before offline item recommendation. First, the interactive item sequence at time t is input into the first action value network to obtain the action values ​​corresponding to all items in the item interaction sequence at time t; then, the interactive item sequence at time t+1 is input into the first action value network to obtain the item corresponding to the maximum action value at time t+1 output by the first action value network; the item corresponding to the maximum action value is input into the second action value network to determine the target action value of the item corresponding to the maximum action value;

[0167] Furthermore, combined with the reward received after outputting the recommended item, the differential loss value is calculated based on the target action value and the action value corresponding to the recommended item, which can also be understood as a kind of loss function, as follows:

[0168]

[0169] Among them, Q θ (s t ,a t ) represents the action value corresponding to the item to be recommended, Q θ′ (s t+1 ,a t+1 ) represents the target action value, γ represents the discount rate, r(s t ,a t ) represents the reward received by the agent after outputting the recommended item;

[0170] Furthermore, the item interaction sequence at time t is input into the first action value network to obtain the action value corresponding to the item to be recommended, where t is an integer greater than or equal to 0; based on the action value corresponding to the item to be recommended and the action value corresponding to all items in the item interaction sequence at time t, a first loss function is calculated, for example, a conservative action value learning loss function:

[0171]

[0172] in, represents the expected action value of all items in the item interaction sequence at time t, Q θ (s t ,a t ) represents the action value corresponding to the item to be recommended;

[0173] Furthermore, multiple negative samples can be collected based on the action values ​​of all items output by the first action value network, and then a second loss function, such as a Bayesian personalized ranking loss function, is calculated based on the action values ​​corresponding to the items to be recommended and the action values ​​of the negative samples:

[0174]

[0175] Among them, Q θ (s t ,a t ) represents the action value of the positive sample, represents the action value of negative samples, and M represents the number of negative samples;

[0176] Finally, based on the above differential loss value, the first loss function, and the second loss function, the target loss function is obtained, for example:

[0177] For example, the target loss function is determined by referring to the following formula (9):

[0178] Loss=TD error+cql loss+bpr loss (9)

[0179] Among them, loss represents the target loss function, TD error represents the differential loss value, cql loss represents the first loss function, and bpr loss represents the second loss function.

[0180] Optionally, the first action value network and the second action value network may be updated using the first parameter using a stochastic gradient descent method until the network converges. At this point, the model training process ends.

[0181] In the method provided in this embodiment, offline reinforcement learning technology is applied to the item recommendation problem, so that the reinforcement learning agent can complete training only in the offline data set without interacting with the online recommendation scenario, optimize long-term benefits while capturing the dynamic changes of user preferences, and introduce a dynamic negative sampling strategy to further optimize the action value network of the reinforcement learning agent to achieve better recommendation performance.

[0182] Figure 3 The schematic diagram of the structure of the first action value network and the second action value network provided by the present invention is as follows: Figure 3 As shown, the structures of the first action value network and the second action value network include an embedding representation layer 310, a state representation layer 320 and an output layer 330, wherein the state representation layer 320 is connected to the embedding representation layer 310 and the output layer 330 respectively; wherein,

[0183] The embedding representation layer 310 is used to assign a corresponding sequence vector to each item in the interactive item sequence;

[0184] A state representation layer 320, used to convert the sequence vector into a hidden state vector; the hidden state vector and the sequence vector have the same dimension;

[0185] The output layer 330 is used to map the hidden state vector to the item space and output the action value of all items.

[0186] Specifically, it can be understood that the structure of the second action value network is consistent with that of the first action value network. The operation process of inputting the sequence of interactive items into the second action value network can refer to the first action value network. Here, the calculation process of the first action value network is taken as an example, and the operation process in the second action value network will not be repeated.

[0187] Optionally, before inputting the interaction item sequences of each item into the first action value network, the lengths of the interaction item sequences of each item may be unified first, for example, unified to a length of l. Specifically, a sequence with insufficient interaction item sequence length may be filled with element 0, and a sequence with a length exceeding l may be truncated with the l most recent items, and then input into the embedded representation layer 310 in the trained first action value network.

[0188] Furthermore, the embedding representation layer 310 assigns a different vector to each item by looking up a table. The vector is called the embedding representation of the item. The interaction sequence as a string of digital inputs is converted into a sequence vector with a shape of l×d (where d is the dimension of the sequence vector and l is the length of the interaction item sequence) after being processed by the embedding representation layer 310, and then used as the input of the state representation layer 320.

[0189] Furthermore, the state representation layer 320 converts the above-mentioned sequence vector of shape l×d into a hidden state vector required for reinforcement learning, wherein the hidden state vector and the sequence vector have the same dimension. Specifically, the function of the state representation layer 320 is to convert the input sequence vector into a hidden state vector required for reinforcement learning. The performance of the state representation layer can affect the quality of the strategy finally learned by the reinforcement learning agent. We can use several classic deep learning sequence recommendation models as the state representation layer 320, such as convolutional neural networks, recurrent neural networks, and transform Transformer neural networks, etc., to extract sequence features and obtain better state representation. The output of the final state representation layer 320 is the hidden state vector of the sequence, which can be expressed as h t It means that the shape of the hidden state vector is 1×d (where d is the dimension of the vector).

[0190] Furthermore, the output layer 330 can map the above hidden state vector to the item space (i.e., action space), and finally output the action value of all items. Specifically, the final output of the first action value network is a 1×N vector (where N represents a total of N items), which represents the state of the reinforcement learning agent at time t. t When , take all possible actions a t The action value at that time is the action value of all possible recommended items.

[0191] For example, the output of the first action-value network is Q all ,

[0192] Q all =Wh t +b=[Q1,Q2,…,Q N ];

[0193] Among them, W represents weight and b represents bias;

[0194] The value with subscript n corresponds to the agent h at time t. t Recommended Items t is i n The corresponding action value is as follows:

[0195] Q n =Q(s t ,a t ),a t =i n ,n∈N;

[0196] Among them, a t represents the agent h at time t t Recommended items, t The latent vector representation obtained after the user-item interaction sequence at time t passes through the state representation layer 320.

[0197] In the method provided by the embodiment of the present invention, the structure of the first action value network and the second action value network includes an embedding representation layer, a state representation layer and an output layer. The embedding representation layer 310 is used to assign a corresponding sequence vector to each item in the interactive item sequence. Then, the state representation layer 320 converts the sequence vector into a hidden state vector, wherein the hidden state vector and the sequence vector have the same dimension. Finally, the output layer 330 maps the hidden state vector to the item space and outputs the action value of all items. The first action value network obtained by offline reinforcement learning is used to solve the sequential decision problem, that is, the item recommendation problem, while capturing the dynamic changes of user preferences and optimizing long-term benefits. It can take into account the long-term and short-term benefits of users. The recommendation performance of using the first action value network to predict recommended items is better.

[0198] The offline item recommendation device provided by the present invention is described below. The offline item recommendation device described below and the offline item recommendation method described above can be referred to each other.

[0199] Figure 4 A schematic diagram of the structure of the offline item recommendation device provided by the present invention is shown in FIG. Figure 4 As shown, the device comprises:

[0200] An acquisition module 410 is used to acquire a sequence of interactive items of a user at a current moment;

[0201] The calculation module 420 is used to calculate the action value of all items in the interactive item sequence using the trained first action value network; the action value represents the user's preference for the item; the first action value network is trained in an offline environment;

[0202] The determination module 430 is used to determine the item corresponding to the largest action value among all the action values ​​of the items as the recommended item.

[0203] In the device provided by this embodiment, the acquisition module 410 is used to obtain the interactive item sequence of the user at the current moment, and then the calculation module 420 uses the trained first action value network to calculate the action value of all items in the interactive item sequence, wherein the action value represents the user's preference for the item, and the first action value network is trained in an offline environment. Finally, the determination module 430 determines the item corresponding to the largest action value among the action values ​​of all items as the recommended item. In the device provided by the present invention, based on the first action value network trained in an offline environment, the interactive item sequence of the user at the current moment is calculated to obtain the action value of all items in the interactive item sequence, so as to recommend items to the user according to the size of the action value of all items, and the recommendation efficiency is high. Moreover, since the first action value network is trained in an offline environment, that is, the first action value network can be trained without interacting with the real recommendation environment, the recommendation result of the trained first action value network is more accurate, and the user experience is better.

[0204] Optionally, the device may further include a training module;

[0205] The training module is used to: use the trained first action value network to calculate the action values ​​of all items in the interactive item sequence.

[0206] Input the item interaction sequence at time t into the first action value network to obtain the action value corresponding to the item to be recommended; t is an integer greater than or equal to 0;

[0207] Input the item interaction sequence at time t+1 into the first action value network, and obtain the item corresponding to the largest action value among the action values ​​output by the first action value network;

[0208] Based on the second action value network, determine the target action value of the item corresponding to the maximum action value;

[0209] Determine the target loss function based on the target action value, the action value corresponding to the item to be recommended, the action value corresponding to all items in the item interaction sequence at time t, and the action value of the negative sample;

[0210] Based on the target loss function and the preset learning rate, the first parameters corresponding to the first action value network and the second action value network are updated.

[0211] Optionally, the training module is specifically used to:

[0212] Calculate the difference loss value based on the target action value and the action value corresponding to the recommended item;

[0213] Calculate the first loss function based on the action value corresponding to the item to be recommended and the action value corresponding to all items in the item interaction sequence at time t;

[0214] Calculate the second loss function based on the action value corresponding to the item to be recommended and the action value of the negative sample;

[0215] Based on the differential loss value, the first loss function, and the second loss function, a target loss function is determined.

[0216] Optionally, the training module is specifically used to:

[0217] Before calculating the second loss function based on the action values ​​corresponding to the recommended items and the action values ​​of the negative samples, multiple negative samples are collected based on the action values ​​of all items output by the first action value network; the negative samples are items that the user has not interacted with based on the negative samples at time t;

[0218] The second loss function is calculated based on the action value corresponding to the item to be recommended and the action value of the negative sample, specifically including:

[0219] Based on the action value corresponding to the item to be recommended and the action value of the negative sample, the second loss function is calculated in the form of positive and negative sample pairs.

[0220] Optionally, the training module is specifically used to:

[0221] Updating a first parameter of the first action value network to obtain an updated first parameter of the first action value network;

[0222] Based on the updated first parameter and the second parameter, the first parameter of the second action value network is updated to obtain the updated first parameter of the second action value network; the second parameter is used to control the update amplitude of the first parameter of the second action value network.

[0223] Optionally, the structures of the first action value network and the second action value network include an embedding representation layer, a state representation layer and an output layer, wherein the state representation layer is connected to the embedding representation layer and the output layer respectively; wherein,

[0224] The embedding representation layer is used to assign a corresponding sequence vector to each item in the interactive item sequence;

[0225] The state representation layer is used to convert the sequence vector into a hidden state vector; the hidden state vector and the sequence vector have the same dimension;

[0226] The output layer is used to map the hidden state vector to the item space and output the action value of all items.

[0227] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communications interface 520 and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the offline item recommendation method, which includes:

[0228] Get the user's interactive item sequence at the current moment;

[0229] The trained first action value network is used to calculate the action value of all items in the interactive item sequence; the action value represents the user's preference for the item; the first action value network is trained in an offline environment;

[0230] The item corresponding to the largest action value among all the action values ​​of the items is determined as the recommended item.

[0231] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0232] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the offline item recommendation method provided by the above methods, the method comprising:

[0233] Get the user's interactive item sequence at the current moment;

[0234] The trained first action value network is used to calculate the action value of all items in the interactive item sequence; the action value represents the user's preference for the item; the first action value network is trained in an offline environment;

[0235] The item corresponding to the largest action value among all the action values ​​of the items is determined as the recommended item.

[0236] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the offline item recommendation method provided by the above methods is implemented, and the method includes:

[0237] Get the user's interactive item sequence at the current moment;

[0238] The trained first action value network is used to calculate the action value of all items in the interactive item sequence; the action value represents the user's preference for the item; the first action value network is trained in an offline environment;

[0239] The item corresponding to the largest action value among all the action values ​​of the items is determined as the recommended item.

[0240] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0241] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0242] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An offline item recommendation method, characterized in that: include: Get the user's interactive item sequence at the current moment; The action values ​​of all items in the interactive item sequence are calculated using the trained first action value network; the action value represents the user's preference for the item; the first action value network is trained in an offline environment; Determine the item corresponding to the largest action value among all the action values ​​of the items as the recommended item; Before calculating the action values ​​of all items in the interactive item sequence using the trained first action value network, the method further includes: Input the item interaction sequence at time t into the first action value network to obtain the action value corresponding to the item to be recommended; t is an integer greater than or equal to 0; Input the item interaction sequences at time t+1 into the first action value network respectively, and obtain the item corresponding to the largest action value among the action values ​​output by the first action value network; Determining a target action value of the item corresponding to the maximum action value based on the second action value network; Determine a target loss function based on the target action value, the action value corresponding to the item to be recommended, the action value corresponding to all items in the item interaction sequence at time t, and the action value of negative samples; Based on the target loss function and a preset learning rate, updating the first parameters corresponding to the first action value network and the second action value network respectively; The determining of the target loss function based on the target action value, the action value corresponding to the to-be-recommended item, the action value corresponding to all items in the item interaction sequence at time t, and the action value of the negative sample includes: Calculating a differential loss value based on the target action value and the action value corresponding to the item to be recommended; Calculating a first loss function based on the action value corresponding to the item to be recommended and the action values ​​corresponding to all items in the item interaction sequence at time t; Calculating a second loss function based on the action value corresponding to the item to be recommended and the action value of the negative sample; Determine the target loss function based on the differential loss value, the first loss function, and the second loss function; Before calculating the second loss function based on the action value corresponding to the item to be recommended and the action value of the negative sample, the method further includes: A plurality of negative samples are collected based on the action values ​​of all items output by the first action value network; the negative samples are obtained by sampling a certain amount of items that the user has not interacted with at time t, and the items that the user has not interacted with are items with negative action values; The calculating the second loss function based on the action value corresponding to the item to be recommended and the action value of the negative sample includes: The second loss function is calculated in a positive-negative sample pair manner based on the action value corresponding to the to-be-recommended item and the action value of the negative sample.

2. The offline item recommendation method according to claim 1, characterized in that: The updating of first parameters corresponding to the first action value network and the second action value network respectively includes: Updating a first parameter of the first action value network to obtain an updated first parameter of the first action value network; Based on the updated first parameter and the second parameter, the first parameter of the second action value network is updated to obtain the updated first parameter of the second action value network; the second parameter is used to control the update amplitude of the first parameter of the second action value network.

3. The offline item recommendation method according to claim 1, characterized in that: The structures of the first action value network and the second action value network include an embedding representation layer, a state representation layer and an output layer, and the state representation layer is connected to the embedding representation layer and the output layer respectively; The embedding representation layer is used to assign a corresponding sequence vector to each item in the interactive item sequence; The state representation layer is used to convert the sequence vector into a hidden state vector; the hidden state vector and the sequence vector have the same dimension; The output layer is used to map the hidden state vector to the item space and output the action values ​​of all items.

4. An offline item recommendation device, characterized in that: include: The acquisition module is used to obtain the user's interactive item sequence at the current moment; A calculation module, used to calculate the action value of all items in the interactive item sequence using a trained first action value network; the action value represents the user's preference for the item; the first action value network is trained in an offline environment; A determination module, configured to determine the item corresponding to the largest action value among the action values ​​of all the items as a recommended item; The device also includes a training module; The training module is used to input the item interaction sequence at time t into the first action value network before calculating the action values ​​of all items in the interactive item sequence using the trained first action value network, to obtain the action value corresponding to the item to be recommended; t is an integer greater than or equal to 0; input the item interaction sequence at time t+1 into the first action value network, to obtain the item corresponding to the largest action value among the action values ​​output by the first action value network; and determine the target action value of the item corresponding to the largest action value based on the second action value network; Based on the target action value, the action value corresponding to the item to be recommended, the action value corresponding to all items in the item interaction sequence at time t, and the action value of the negative sample, a target loss function is determined; based on the target loss function and a preset learning rate, the first parameters corresponding to each of the first action value network and the second action value network are updated; the target loss function is determined based on the target action value, the action value corresponding to the item to be recommended, the action value corresponding to all items in the item interaction sequence at time t, and the action value of the negative sample, including: calculating the differential loss value based on the target action value and the action value corresponding to the item to be recommended; calculating the first loss function based on the action value corresponding to the item to be recommended and the action value corresponding to all items in the item interaction sequence at time t; calculating the differential loss value based on the action value corresponding to the item to be recommended The second loss function is calculated based on the action value of the recommended item and the action value of the negative sample; the target loss function is determined based on the differential loss value, the first loss function and the second loss function; before calculating the second loss function based on the action value corresponding to the recommended item and the action value of the negative sample, it also includes: collecting multiple negative samples based on the action values ​​of all items output by the first action value network; the negative samples are obtained by sampling a certain amount of items that the user has not interacted with at time t, and the items that the user has not interacted with are items with negative action values; the second loss function is calculated based on the action value corresponding to the recommended item and the action value of the negative sample, including: calculating the second loss function in the form of positive and negative sample pairs based on the action value corresponding to the recommended item and the action value of the negative sample.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the offline item recommendation method according to any one of claims 1 to 3 is implemented.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the offline item recommendation method according to any one of claims 1 to 3 is implemented.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the offline item recommendation method according to any one of claims 1 to 3 is implemented.

Images