Recommendation method, device and storage medium

Abstract

A recommendation method, device and storage medium, the method comprising: obtaining feature information of a target user, the feature information comprising user portrait information and historical behavior information; inputting the feature information into an item recommendation model to determine a predicted score for each candidate item in a candidate item set for the target user, the item recommendation model comprising k sub-recommendation models, each sub-recommendation model being obtained by training a neural network based on k weight parameters and a training sample set corresponding to the target user, the k weight parameters being determined based on an actor-critic model and the training sample set, and k being a positive integer; determining a target candidate item from the candidate item set based on the predicted score, and pushing the target candidate item to a client corresponding to the target user, wherein the actor-critic model can be used to determine the user's personalized weight parameters to improve the accuracy of item recommendations.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a recommendation method, device, and storage medium. Background Art

[0002] With the rapid development of the internet, the amount of information is exploding, leading to a growing problem of information overload. This means that users' ability to filter and select information is significantly lagging, making it difficult to quickly retrieve desired or interesting content from the vast amount of information. Recommender systems are an effective tool for addressing this information overload problem, providing personalized recommendations to meet the individual needs of different users.

[0003] In the actual recommendation process, the recommendation system often has more than one business indicator. For example, in the e-commerce field, it is necessary to consider the optimization of two business indicators: recommendation accuracy (Click-Through-Rate, CTR) and total transaction volume. Improving recommendation accuracy but reducing total transaction volume will directly reduce the company's revenue. Increasing total transaction volume but significantly decreasing recommendation accuracy will also have a negative impact on the business. In the information flow field or video recommendation field, in addition to recommendation accuracy, user browsing time, conversion rate, sharing rate, etc. can also be used as business indicators of the recommendation system. At present, when solving the optimization problem of multiple business indicators, it is based on manually defining the weight parameters of each business indicator, and then solving based on the defined weight parameters. As a result, the items recommended by the recommendation system may not be what the user wants or is interested in. Summary of the Invention

[0004] The embodiments of the present invention provide a recommendation method, device, and storage medium, which can determine a user's personalized weight parameters through an actor-critic model to improve the accuracy of item recommendations.

[0005] In one aspect, an embodiment of the present invention provides a recommendation method, the method comprising:

[0006] Obtaining target user's characteristic information, including user profile information and historical behavior information;

[0007] Inputting the feature information into an item recommendation model to determine a predicted score for each candidate item in a candidate item set for the target user, wherein the item recommendation model includes k sub-recommendation models, wherein each sub-recommendation model is obtained by training a neural network based on k weight parameters corresponding to the target user and a training sample set, wherein the k weight parameters are determined based on an actor-critic model and the training sample set, and k is a positive integer;

[0008] A target candidate item is determined from the candidate item set according to the predicted score, and the target candidate item is pushed to a client corresponding to the target user.

[0009] In another aspect, an embodiment of the present invention provides a recommendation device, comprising:

[0010] An acquisition module is used to obtain characteristic information of a target user, including user profile information and historical behavior information;

[0011] a processing module, configured to input the feature information into an item recommendation model to determine a predicted score for each candidate item in a candidate item set for the target user, wherein the item recommendation model includes k sub-recommendation models, wherein each sub-recommendation model is obtained by training a neural network based on k weight parameters corresponding to the target user and a training sample set, wherein the k weight parameters are determined based on an actor-critic model and the training sample set, and k is a positive integer;

[0012] The processing module is further configured to determine a target candidate item from the candidate item set according to the predicted score, and push the target candidate item to a client corresponding to the target user.

[0013] Accordingly, an embodiment of the present invention provides a computer device, which includes a processor, a communication interface and a memory, wherein the processor, the communication interface and the memory are interconnected, wherein the memory stores executable program code, and the processor is used to call the executable program code to execute the recommended method described in any of the above possible implementation methods.

[0014] Accordingly, an embodiment of the present invention provides a computer-readable storage medium storing a computer program, and the processor executes the program involved in the recommendation method described in any possible implementation manner.

[0015] Accordingly, embodiments of the present invention provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the recommendation method described in any of the possible implementations described above.

[0016] In an embodiment of the present invention, the characteristic information of the target user is first obtained, and then the predicted score of each candidate item in the candidate item set for the target user is determined based on the characteristic information of the target user and the item recommendation model. Then, the target candidate item can be determined from the candidate item set based on the predicted score, and the target candidate item is pushed to the client corresponding to the target user, so that the item recommendation model can make personalized recommendations for the target user. At the same time, the k sub-recommendation models included in the item recommendation model are obtained by training a neural network based on the k weight parameters and training sample sets corresponding to the target user, wherein the k weight parameters are determined based on the actor-critic model and the training sample set. In this way, the personalized weight parameters corresponding to the user can be determined through the actor-critic model, so that the item recommendation model can more effectively make personalized recommendations and improve the accuracy of item recommendations. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A schematic diagram of the architecture of a recommendation system provided in an embodiment of the present application;

[0019] Figure 2 A flowchart of a recommended method provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of the network architecture of a sub-recommendation model provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of a process for establishing an item recommendation model provided in an embodiment of the present application;

[0022] Figure 5 A flowchart of a recommended method provided in an embodiment of the present application;

[0023] Figure 6 is a structural diagram of a recommended device provided by an embodiment of the present invention;

[0024] Figure 7 It is a structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0026] It should be noted that the terms "first" and "second" in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature designated as "first" or "second" may explicitly or implicitly include at least one such feature.

[0027] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, cloud storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0028] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0029] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields. The solutions provided in the embodiments of this application involve artificial intelligence machine learning technology and other technologies, which are specifically explained through the following embodiments.

[0030] See also Figure 1 , Figure 1: is a schematic diagram of a recommendation system provided in an embodiment of the present application. The recommendation system can be specifically integrated into an electronic device, which can be a terminal device or a server. The terminal device is also called a terminal (Terminal), user equipment (UE), access terminal, user unit, mobile device, user terminal, wireless communication device, user agent or user device. The terminal device can be a smart TV, a handheld device with wireless communication function (such as a smart phone, tablet computer), a computing device (such as a personal computer (PC), a vehicle-mounted device, a wearable device or other smart device, etc., but is not limited thereto.

[0031] The server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal devices and servers can be connected directly or indirectly via wired or wireless communications, which is not limited in this application.

[0032] It is understandable that the recommendation method of this embodiment may be executed on the terminal device, may be executed on the server, or may be executed jointly by the terminal device and the server.

[0033] In one embodiment, a server-executed recommendation method is used as an example. The recommendation system includes a terminal device 101 and a server 102, and the terminal device 101 and the server 102 are connected via a network, such as a wireless network connection. The server 102 obtains the characteristic information of the target user through the network, and thereby determines the predicted score of each candidate item in the candidate item set for the target user based on the characteristic information of the target user and the item recommendation model. Then, the target candidate item can be determined from the candidate item set based on the predicted score, and the target candidate item can be pushed to the client corresponding to the target user, so that the item recommendation model can make personalized recommendations for the target user. At the same time, the k sub-recommendation models included in the item recommendation model are obtained by the server 102 training the neural network based on the k weight parameters and training sample set corresponding to the target user, wherein the k weight parameters are determined based on the actor-critic model and the training sample set. In this way, the personalized weight parameters of each user can be determined by the actor-critic model, so that the item recommendation model can make personalized recommendations more effectively, and improve the accuracy of item recommendations and the promotion benefits of merchants.

[0034] It can be understood that the architectural diagram of the system described in the embodiment of the present application is intended to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0035] In one embodiment, Figure 2 As shown, this embodiment of the application is based on Figure 1 A recommendation method provided by the recommendation system. Figure 1 The server 102 mentioned in is taken as an example. Figure 2 The method of the embodiment of the present application is described.

[0036] S201. Obtain characteristic information of a target user, where the characteristic information includes user portrait information and historical behavior information.

[0037] In one embodiment, the target user can refer to any user. When making personalized recommendations for the target user, the server needs to obtain the target user's current feature information. Among them, user portrait information specifically includes the target user's age, gender, occupation, hobbies, place of residence, marital status, etc.; historical behavior information can include item information and corresponding click feature information and context information; click feature information needs to be determined based on actual business indicators. For example, when the business indicators of the item recommendation system are recommendation accuracy and consumption time, click feature information can be items previously clicked by the target user and the corresponding consumption time; item information is information or category labels related to the items clicked by the target user. For example, when the item is a commodity, the item information can be category labels such as clothes and shoes; context information mainly includes sequence information and time information of the items clicked by the target user.

[0038] The specific form of the item can be determined based on actual needs. For example, in the e-commerce field, the item form can be a commodity. In the information flow field or video flow field, the item form can be a video, news, article, etc., or an advertisement, picture, novel, etc., which is not limited in this application. The target user is the user for personalized recommendation, which can be one or more users, which is not limited in this application.

[0039] S202. Input the feature information into an item recommendation model to determine a predicted score for each candidate item in the candidate item set for the target user, wherein the item recommendation model includes k sub-recommendation models, wherein each sub-recommendation model is obtained by training a neural network based on k weight parameters corresponding to the target user and a training sample set, wherein the k weight parameters are determined based on an actor-critic model and the training sample set, and k is a positive integer.

[0040] In one embodiment, when making personalized recommendations for a target user, the server can determine a predicted score for each candidate item in the candidate item set based on the item recommendation model and the acquired target user's characteristic information. The predicted score reflects the expected benefit of recommending the candidate item to the target user.

[0041] Specifically, the item recommendation model includes k sub-recommendation models, k is a positive integer, each sub-recommendation model corresponds to a different business indicator, and the prediction score is determined by the scores of the k sub-recommendation models. In the actual recommendation process, a common recommendation system often has more than one business indicator, and can specifically include two or three or more business indicators. This application takes the information flow field as an example, and there are two main business indicators: recommendation accuracy and consumption time. The corresponding two sub-recommendation models can be obtained based on the recommendation accuracy and consumption time. According to the current feature information of the target user, the score of the recommendation accuracy and consumption time when executing any candidate item in the candidate item set can be determined, thereby determining the prediction score of any candidate item based on the score of the recommendation accuracy and consumption time of any candidate item.

[0042] The k sub-recommendation models included in the item recommendation model are obtained by training the neural network based on the k weight parameters corresponding to the target user and the training sample set. The k weight parameters are determined based on the actor-critic model and the training sample set. The k neural network models are used to learn for different business indicators to obtain sub-recommendation models. The item recommendation model includes k sub-recommendation models and weight parameters for each sub-recommendation model. The weight parameters of each of the k sub-recommendation models are determined by the k weight parameters corresponding to the target user.

[0043] S203: Determine a target candidate item from the candidate item set according to the predicted score, and push the target candidate item to a client corresponding to the target user.

[0044] In one embodiment, after determining the predicted score of each candidate item in the candidate item set, the candidate items with predicted scores greater than a threshold are determined as target candidate items based on the predicted score of each candidate item, or a preset number of candidate items are selected as target candidate items based on descending order of the predicted scores, and the target candidate items are pushed to the client corresponding to the target user. The client can be installed in Figure 1 On the terminal device 101 mentioned.

[0045] Through the embodiments of the present application, the characteristic information of the target user can be obtained, and the predicted score of each candidate item in the candidate item set for the target user can be determined based on the characteristic information of the target user and the item recommendation model. Then, the target candidate item can be determined from the candidate item set based on the predicted score, and the target candidate item can be pushed to the client corresponding to the target user, so that the item recommendation model can make personalized recommendations for the target user. At the same time, the k sub-recommendation models included in the item recommendation model are obtained by training the neural network based on the k weight parameters and training sample sets corresponding to the target user, wherein the k weight parameters are determined based on the actor-critic model and the training sample set. In this way, the personalized weight parameters of each user can be determined by the actor-critic model, so that the item recommendation model can make personalized recommendations more effectively, and improve the accuracy of item recommendations and the promotion benefits of merchants.

[0046] In one embodiment, the historical behavior information includes item information and corresponding click feature information and context information. Inputting the feature information into the item recommendation model to determine the predicted score of each candidate item in the candidate item set for the target user includes: performing feature cross-processing on the user portrait information, the item information and the corresponding click feature information to obtain a first feature of the target user; encoding the context information to obtain a second feature of the target user; and inputting the first feature and the second feature into the item recommendation model to determine the predicted score of each candidate item in the candidate item set for the target user.

[0047] In one embodiment, historical behavior information includes item information and corresponding click feature information and contextual information. Click feature information needs to be determined based on actual business indicators. For example, when the business indicators of the item recommendation system are recommendation accuracy and consumption time in the information flow field, click feature information can include items previously clicked by the target user and the corresponding consumption time. Item information includes information or category labels related to items clicked by the target user. For example, when the items are commodities, the item information can include category labels such as clothing and shoes. Contextual information mainly includes sequence information and time information of items clicked by the target user. When inputting the acquired target user's current feature information into the item recommendation model, in cases where the data dimensions of the user profile information, item information, and corresponding click feature information included in the acquired feature information are particularly high and the data is particularly sparse, feature cross-processing can be performed to automatically learn high-order feature combinations to obtain the target user's first feature, thereby capturing high-order nonlinear relationships between features. The feature cross-processing method can adopt the AutoInt model (automatically learning feature combinations through the self-attention mechanism), the Attentional Factorization Machine (AFM) model, the Product-based Neural Network (PNN), etc., which is not limited in this application. At the same time, for the context information corresponding to the item information included in the feature information, the Transformer model can be used to encode the context information corresponding to the item information included in the feature information. Specifically, the Transformer model uses sine and cosine functions to construct sequence encoding or time encoding so that the attention mechanism can perceive the sequence information or time information, thereby obtaining the second feature of the target user. The first feature and the second feature are combined and input into the item recommendation model to determine the predicted score of each candidate item in the candidate item set for the target user. When the feature information of the target user is input into the item recommendation model, the feature information of the target user is subjected to feature cross-processing and encoding processing, which can facilitate the item recommendation model to better utilize the feature information of the target user, thereby more effectively making personalized recommendations for the target user.

[0048] In one embodiment, the feature information is input into the item recommendation model to determine the predicted score of each candidate item in the candidate item set for the target user, including: for any candidate item in the candidate item set, the feature information and the any candidate item are input into each of the k sub-recommendation models respectively, and the score of the target user on the business indicator corresponding to each sub-recommendation model output by each sub-recommendation model is obtained to obtain the score of the target user on the k business indicators, wherein different sub-recommendation models correspond to different business indicators; based on the score of the target user on the k business indicators and the k weight parameters, the predicted score of the any candidate item for the target user is determined.

[0049] In one embodiment, for any candidate item in the candidate item set, the target user's feature information and any candidate item are input into each of the k sub-recommendation models in the item recommendation model. The feature information of the target user is determined. When recommending any candidate item, the score of each of the k sub-recommendation models on the corresponding business indicator is determined. Different sub-recommendation models correspond to different business indicators. The predicted score of any candidate item for the target user is determined by weighting the target user's score on the k business indicators and the weight parameters of the sub-recommendation models corresponding to each business indicator and summing them. For example, if k = 3, there are three sub-recommendation models. The scores of the business indicators corresponding to each of the three sub-recommendation models are k1, k2, and k3. The weight parameters of each sub-recommendation model are w1, w2, and w3. The predicted score is k1×w1+k2×w2+k3×w3.

[0050] See Figure 3 , Figure 3 The following example illustrates a network architecture diagram of a sub-recommendation model actually in use. When obtaining the sub-recommendation model, that is, when training the neural network, first, after obtaining the original data related to the target user, the original data needs to be preprocessed to obtain the characteristic information of the target user. Furthermore, feature cross-processing and encoding processing can be performed through the encoder in the AutoInt model and the transformer model to obtain the first feature s1 and the second feature s2 of the target user. By combining the first feature s1 and the second feature s2 of the target user and inputting them into the gated recurrent unit (GRU) network + fully connected layers (FC) in the neural network, the characteristic information s of the target user can be determined. t Recommend any candidate item a in the candidate item set tThe first predicted reward value Q1(s t ,a t ). The neural network can adopt any network structure, as long as it uses the network update method proposed in this application to learn, so this application does not limit the network structure of the neural network. Figure 3 When the sub-recommendation model is shown, the first predicted reward value obtained by the neural network can be found in the following formula (1):

[0051] Q1(s t ,a t )=FC(GRU([s1,s2])) (1)

[0052] As shown in the following formula (2), the loss function of the neural network is:

[0053] L(θ)=MSE(r1(S t ,a t ),Q1(S t ,a t )) (2)

[0054] Among them, r1(s t ,a t ) is the first real reward value of the target user for the business indicator corresponding to the sub-recommendation model or neural network, and θ is the network parameter of the neural network.

[0055] The sub-recommendation model is determined by training the neural network, and the characteristic information of the target user and any candidate item are input into the sub-recommendation model, so that the characteristic information of the target user can be obtained through the sub-recommendation model. When recommending any candidate item, the expected benefit can be obtained from the target user.

[0056] In order to obtain the sub-recommendation model, the neural network needs to be trained. During training, the Deep Q-Network (DQN) method can be used to update the network. The specific update process can be referred to the following steps:

[0057] 1. Initialize the experience replay pool D with a capacity of N.

[0058] 2. Initialize the online value network Q with random weights θ, and at the same time initialize the weights θ' of the target value network Q', where θ' = θ.

[0059] 3. Loop through the episodes (episode = 1, 2, ..., M).

[0060] 1) Initialize feature information s t .

[0061] 2) Loop through each step in the episode (t=1,2,…,T).

[0062] (1) Use the ε strategy to randomly select a recommended item a t , or select a t =argmaxaQ(s,a;θ).

[0063] (2) Execute recommended item a t , receiving score r t , and new feature information s t+1 .

[0064] (3) The memory unit (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool D.

[0065] (4) Randomly extract a mini-batch of memory units from the experience replay pool D.

[0066] (5) Then:

[0067]

[0068] (6) To ((y j -Q(s j ,a j ;θ)) 2 The gradient descent method is used to update θ.

[0069] (7) Update the target value network Q'=Q every C steps.

[0070] In one embodiment, a document list approach (ListWise) is used when training a neural network. This approach uses the feedback from target users after the item recommendation model makes a recommendation as a training example for the neural network training. Furthermore, reinforcement learning is employed during the neural network training process based on Deep Q-learning (DQN), taking into account the long-term benefits of item recommendations. In addition to ListWise, other approaches, such as Pointwise and Pairwise, may also be employed, though this application does not limit these approaches.

[0071] In one embodiment, before inputting the feature information into the item recommendation model to determine the predicted score of each candidate item in the candidate item set for the target user, it is necessary to establish an item recommendation model, such as Figure 4 As shown, the following steps are included:

[0072] S1. Obtain a training sample set, where each sample in the training sample set includes an item clicked by a target user, a corresponding first real reward value, and feature information of the target user, where the first real reward value is obtained based on indicator data of the target user on the k business indicators.

[0073] S2. Input the item clicked by the target user and the corresponding characteristic information of the target user included in each sample into k neural networks respectively to obtain a second real reward value of the item clicked by the target user.

[0074] S3. Determine an updated weight parameter of each of the k neural networks based on the second true reward value and the actor-critic model.

[0075] S4. Adjust the network parameters of the k neural networks based on the updated weight parameters and loss function of each neural network to obtain k sub-recommendation models, and determine the item recommendation model based on the k sub-recommendation models.

[0076] In one embodiment, each sample in the training sample set includes an item clicked by a target user, a corresponding first true reward value, and target user feature information. The first true reward value is derived based on the target user's performance data for the item clicked on k business metrics. For example, in the information flow domain, when the business metrics are recommendation accuracy and consumption time, the first true reward value includes information on whether the target user clicked on the item and the duration of their viewing. The target user's feature information includes user profile information, item information, corresponding click feature information, and contextual information. User profile information specifically includes the target user's age, gender, occupation, hobbies, place of residence, marital status, etc. Click feature information needs to be determined based on the actual business metrics. For example, when the business metrics of the item recommendation system are recommendation accuracy and consumption time in the information flow domain, click feature information can include items previously clicked by the target user and the corresponding consumption time. Item information includes information or category labels related to the items clicked by the target user. For example, when the items are commodities, the item information can include category labels such as clothing or shoes. Contextual information primarily includes sequence information and time information of the items clicked by the target user.

[0077] In one embodiment, the items clicked by the target user and the corresponding characteristic information of the target user included in each sample in the training sample set are respectively input into k neural networks, where each neural network corresponds to a different business indicator to determine the characteristic information of the target user. When recommending the item clicked by the target user, each of the k neural networks obtains a first predicted reward value based on the corresponding business indicator. The first predicted reward value reflects the expected profit of recommending the item clicked by the target user to the target user. A second true reward value for the item clicked by the target user is then obtained based on the first predicted reward value, the first true reward value, the loss function of each neural network, and the weight parameters.

[0078] In one embodiment, the updated weight parameters of each of the k neural networks are determined based on the second real reward value and the actor-critic model. Since the k neural networks correspond to k different business indicators, there are k loss functions {L1(θ), L2(θ), ..., L k (θ)},L i (θ) corresponds to the loss function of the i-th neural network. In order to solve the optimization problem of multiple business indicators, Pareto multi-objective optimization is introduced. Pareto multi-objective optimization refers to an optimal state in which it is impossible to improve another indicator at the expense of other indicators. Therefore, this application uses multiple business indicators, that is, the weight parameters w in multiple neural networks. i (θ) is a learnable parameter. Since the Pareto stationary point is a necessary condition for the Pareto optimal solution, the Pareto optimal solution is obtained by solving the Pareto stationary point, thereby determining the weight parameter w of each business indicator. i (θ). The Pareto stationary point is defined as follows: Assume that there are k business indicators and the loss function of the i-th business indicator is L i (θ sh ,θ i ), where θ sh is a shared parameter, θ i is the parameter of the i-th business indicator, then w that satisfies the following conditions i (θ) is the Pareto stationary point:

[0079] 1. There are weights w1,…,w k ≥0 such that∑w i (θ) = 1 and

[0080] 2. For each business indicator, make

[0081] Furthermore, the Pareto stationary point can be considered as an optimization problem by the following formula (3):

[0082]

[0083] Specifically, the actor-critic model is used to determine k business indicators, i.e., the weight parameters of each of the k sub-reference models. The optimization goal of the actor-critic model determined based on the Pareto optimal solution is to maximize G(θ), where G(θ) is as shown in the following formula (4):

[0084] Among them, ∑w i (φ)=1,w i (φ)≥0 (4)

[0085] Different from existing solutions in the industry, this application considers the long-term benefits of G(θ), namely the following formula (5):

[0086] Q(s,a;θ)=G t (θ)+γG t+1 (θ)+γ T-t G T (θ) (5)

[0087] Among them, s is the feature information, and a is the action performed, that is, the recommended item.

[0088] Then, the optimization objective of the actor-critic model is transformed into the following formula (6):

[0089] maxQ(s,a;θ)where ∑w i =1,w i ≥0 (6)

[0090] The optimization goal of the critic network is to maximize Q(s,a;θ), and the goal of the actor network is to determine the weight parameters of the business indicators [w1,w2,…,w k ], the critic model is used to guide the learning of the actor model. According to formula (6), the training [w1,w2,…,w k ] will optimize along the direction of simultaneous gradient descent of multiple business indicators, or directly obtain the optimal solution that satisfies the KKT (Karush-Kuhn-Tucker) condition. In optimization theory, the KKT condition is a necessary condition for the optimal solution of nonlinear programming. The KKT condition generalizes the equality-constrained optimization problem in the Lagrange multiplier method to inequality constraints.

[0091] In one embodiment, in each round of training, the k weight parameters determined by the actor model are used as the updated weight parameters of each neural network, and the k neural networks are retrained based on the updated weight parameters and loss function of each business indicator, thereby adjusting the network parameters of the k neural networks. By using the training stop condition, the final k sub-recommendation models can be obtained, where each sub-recommendation model includes weight parameters that can be obtained by the weight parameters finally determined by the actor-critic model when training stops. The item recommendation model is determined by the obtained k sub-recommendation models and the weight parameters of each sub-recommendation model, where the k sub-recommendation models correspond to k different business indicators.

[0092] In one embodiment, the step of inputting the item clicked by the target user included in each sample and the corresponding characteristic information of the target user into k neural networks respectively to obtain the second real reward value of the item clicked by the target user includes: inputting the item clicked by the target user included in each sample and the corresponding characteristic information of the target user into k neural networks respectively to obtain the first predicted reward value output by each of the k neural networks; and determining the second real reward value of the item clicked by the target user based on the first predicted reward value output by each neural network, the first real reward value, and the loss function and weight parameters of each neural network.

[0093] In one embodiment, the items clicked by the target user and the corresponding characteristic information of the target user included in each sample in the training sample set are input into k neural networks respectively, wherein each neural network corresponds to a different business indicator to determine the characteristic information for the target user. When recommending the items clicked by the target user, each of the k neural networks obtains a first predicted reward value based on the corresponding business indicator. The first predicted reward value reflects the expectation of gaining benefits by recommending the items clicked by the target user to the target user. The second real reward value of the item clicked by the target user is obtained based on the first predicted reward value, the first real reward value, the loss function and the weight parameter of each neural network. The second real reward value r2(s t ,a t )Please refer to the following formulas (7)-(8):

[0094]

[0095] L i (θ)=MSE(r1 i (s t ,a t ),Q1 i (s t ,a t )) (8)

[0096] Among them, r1 i (s t ,a t ) is the first real reward value of the i-th neural network, Q1 i (s t ,a t ) is the first predicted reward value of the i-th neural network, L i (θ) is the loss function of the i-th neural network, w i (θ) is the weight parameter of the i-th neural network.

[0097] When considering two business indicators in the information flow field: recommendation accuracy and consumption time, the second real reward value is as shown in the following formula (9):

[0098]

[0099] in, is the weight parameter of the neural network corresponding to the recommendation accuracy, is the weight parameter of the neural network corresponding to the consumption time, L click (s t ,a t |θ) is the loss function of the neural network corresponding to the recommendation accuracy, L dwell (s t ,a t |θ) is the loss function of the neural network corresponding to the consumption time.

[0100] In one embodiment, determining the weight parameters of each of the k neural networks based on the second true reward value and the actor-critic model includes: inputting the characteristic information of the target user corresponding to the item clicked by the target user included in each sample into the actor model in the actor-critic model to obtain k predicted weight parameters output by the actor model; inputting the item clicked by the target user included in each sample and the corresponding characteristic information of the target user, and the k predicted weight parameters into the critic model in the actor-critic model to obtain the second predicted reward value of the item clicked by the target user output by the critic model; adjusting the network parameters of the actor-critic model according to the second true reward value, the second predicted reward value, the k predicted weight parameters and the total loss function of the actor-critic model; and determining the updated weight parameters of each of the k neural networks based on the actor model in the adjusted actor-critic model.

[0101] In one embodiment, the feature information of the target user corresponding to the item clicked by the target user included in each sample is input into the actor model in the actor-critic model to obtain k prediction weight parameters w output by the actor model pred , as shown in the following formula (10):

[0102] w pred =Softmax(FC(GRU([s1,s2]))) (10)

[0103] Furthermore, the item clicked by the target user included in each sample, the corresponding target user's feature information, and the k prediction weight parameters output by the actor model are input into the critic model in the actor-critic model to obtain a second predicted reward value output by the critic model for the item clicked by the target user. The network parameters of the actor model can be adjusted based on the second predicted reward value so that the output k prediction weight parameters can reach a Pareto optimal solution.

[0104] Furthermore, the network parameters of the actor-critic model are adjusted based on the second real reward value, the second predicted reward value, the k predicted weight parameters, and the total loss function of the actor-critic model. When the total loss function of the actor-critic model converges or the number of training times meets the set number of times, the adjustment of the network parameters of the actor-critic model is stopped. During each round of training, the weight parameters output by the actor model in the adjusted actor-critic model are used as the updated weight parameters of each of the k neural networks, and the network parameters of the k neural networks are adjusted based on the updated weight parameters of each neural network and the loss function to obtain an item recommendation model.

[0105] In one embodiment, before adjusting the network parameters of the actor-critic model based on the second true reward value, the second predicted reward value, the k predicted weight parameters, and the total loss function of the actor-critic model, the method further includes: determining a loss function of the actor model and a loss function of the critic model, where the loss function of the actor model includes a policy gradient loss function and a weight parameter loss function; and determining the total loss function of the actor-critic model based on the loss function of the actor model and the loss function of the critic model.

[0106] In one embodiment, the loss function of the actor model and the loss function of the critic model need to be determined to determine the total loss function of the actor-critic model.

[0107] The output of the critic model is the second predicted reward value, and the second true reward value is used to construct the loss function L1(θ Q), as shown in the following formula (11):

[0108] L1(θ Q )=MSE(r2(s t ,a t ),Q2(s t ,a t )) (11)

[0109] Among them, Q2(s t ,a t ) is the second predicted reward value, θ Q are the network parameters of the critic model.

[0110] The learning objectives of the actor model are twofold. The first is the policy gradient returned by the critic model. Specifically in this scenario, That is, the gradient value of the second predicted reward value of the critic model to the predicted weight parameter of the actor model. By maximizing this gradient, the actor model moves towards maximizing Q2(s t ,a t ) direction of learning; the second is w pred and the weight parameters w of k neural networks real The mean square error loss ensures the convergence and stability of learning. Therefore, the loss function of the actor model includes the policy gradient loss function L21(θ μ ) and weight parameter loss function L22(θ μ ), as shown in the following formulas (12)-(13):

[0111]

[0112] L22(θ μ )=MSE(w pred ,w real ) (13)

[0113] Among them, θ μ are the network parameters of the actor model, It is the strategy probability of the weight parameter obtained based on formula (10) when the feature information is given.

[0114] When considering two business indicators in the information flow field: recommendation accuracy and consumption time, the weight parameter loss function L22(θ) is as shown in the following formula (14):

[0115]

[0116] Among them, w click and w dwell The prediction weight parameters for the actor model for recommendation accuracy and consumption time.

[0117] Then, the total loss function L of the actor-critic model is all As shown in the following formula (15):

[0118] L all =L21(θ Q )+β(L21(θ μ )+L22(θ μ )) (15)

[0119] Here, β can be 1.

[0120] See Figure 5 , a flowchart of a recommendation method provided in an embodiment of the present application, wherein the two neural networks included in the item recommendation model are only used as examples, and multiple neural networks may be included, and the shared bottom layer may be a multi-gate mixture of experts (MMoE) model, a single-gate mixture of experts (OMOE) model, a shared bottom layer (Shared-Bottom) model, etc., which is mainly used to prevent the item recommendation model from overfitting during the training process. The second predicted reward value output by the critic model is r2(s t ,a t ), the input includes: 1) the gradient of the loss function of the neural network; 2) the weight parameters in use of the neural network; 3) the feature information of the target user and the predicted weight parameters of the actor model. The critic model is based on the learning of the weight parameters in use of the neural network, so that the weight parameters in the item recommendation model reach the Pareto optimal solution. The initial weight parameters of the neural network can be set manually. The critic model will perform feature cross-processing and encoding on the feature information of the target user to obtain the first feature and the second feature, and at the same time splice the predicted weight parameters of the actor model. Finally, the critic model is based on L1(θ Q ) to update the network parameters. The output of the actor model is the predicted weight parameters of multiple neural networks. The input and model structure are similar to the critic model. The actor model is based on the policy gradient loss function L21(θ μ ) and weight parameter loss function L22(θ μ ) to update network parameters.

[0121] In one embodiment, adjusting the network parameters of the actor-critic model according to the second true reward value, the second predicted reward value, the k predicted weight parameters, and the total loss function of the actor-critic model includes: determining a loss value of the critic model according to the second predicted reward value, the second true reward value, and the loss function of the critic model; determining a first loss value of the actor model according to the second predicted reward value, the policy probability of the k predicted weight parameters, and the policy gradient loss function of the actor model; determining a second loss value of the actor model according to the weight parameters of the k neural networks, the k predicted weight parameters, and the weight parameter loss function of the actor model; determining a total loss value according to the loss value of the critic model, the first and second loss values ​​of the actor model, and the total loss function of the actor-critic model, and adjusting the network parameters of the actor-critic model based on the total loss value.

[0122] In one embodiment, the second predicted reward value and the second true reward value are input into the loss function of the critic model to determine the loss value of the critic model. The second predicted reward value and the policy probability of k predicted weight parameters are input into the policy gradient loss function of the actor model to determine the first loss value of the actor model; the weight parameters of k neural networks and k predicted weight parameters are input into the weight parameter loss function of the actor model to determine the second loss value of the actor model; finally, the loss value of the critic model, the first loss value and the second loss value of the actor model are input into the total loss function of the actor-critic model to determine the total loss value, and the network parameters of the actor-critic model are adjusted based on the total loss value. The process of updating the network parameters of the actor-critic model can refer to the following steps:

[0123] 1. Use random weights θ Q Initialize the critic model Q(s,a|θ Q ), and using random weights θ μ Initialize the actor model μ(s|θ μ ).

[0124] 2. Use weight θ Q′ and θ μ′ Initialize the target critic model Q' and target actor model μ', θ Q′ ←θ Q ,θ μ′ ←θ μ .

[0125] 3. Initialize the experience replay pool R.

[0126] 4. Loop through the episodes (episode = 1, 2, ..., M).

[0127] 1) Initialize a random process T for action exploration.

[0128] 2) Receive initial feature information s1.

[0129] 3) Loop through each step in the episode (t=1,2,…,T).

[0130] (1) Select a recommended item a t =μ(s t |θ μ )+T t .

[0131] (2) Execute recommended item a t , receive reward r t , and new feature information s t+1 .

[0132] (3) The memory unit (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool R.

[0133] (4) Randomly extract a mini-batch of memory units from the experience replay pool R.

[0134] (5) Then:

[0135] y j =r j +γQ′(s j+1 ,μ′(s j+1 |θ μ′ )|θ Q′ )

[0136] (6) By minimizing L = 1 / N∑ j ((y j -Q(s j ,a j |θ Q )) 2 Update the critic model.

[0137] (7) By minimizing the policy gradient loss function L21(θ μ ) and weight parameter loss function L22(θ μ )Update the actor model.

[0138] (8) Update the target critic model Q' and the target actor model μ':

[0139] θ Q′ ←τθ Q +(1-τ)θQ′

[0140] θ μ′ ←τθ μ +(1-τ)θ μ′

[0141] In one embodiment, the specific training process of the item recommendation method proposed in the embodiment of the present application includes the following steps:

[0142] 1. Calculate the loss function of the neural network corresponding to k business indicators {L1(θ), L2(θ), ..., L k (θ)} and the corresponding gradient

[0143] 2. Update the network parameters of the actor-critic model and the updated weight parameters of the neural network corresponding to the prediction of k business indicators.

[0144] 3. Use the updated weight parameters to update the neural network corresponding to the k business indicators to obtain the item recommendation model.

[0145] In one embodiment, after obtaining the updated weight parameters of the neural network, the gradient of each neural network is calculated. Update the network parameters of a single neural network according to the following formula (16):

[0146]

[0147] in, is the learning rate of a single neural network.

[0148] In one embodiment, the present application compares some commonly used recommendation models based on single business indicators and recommendation models based on multiple business indicators in the industry. Recommendation models based on single business indicators include: Factorization Machine (FM), Neural Factorization Machine (NFM), Deep Factorization Machines (DeepFM), AutoInt model, and Adaptive Factorization Network (AFN). Recommendation models based on multiple business indicators include: PO-EA (Pareto-efficient) model and POW-RL (Personalized Objective Weighting with Reinforcement Learning) model. Among them, the business indicators of the item recommendation model include recommendation accuracy and consumption time. The comparison indicators of recommendation accuracy include hit rate@K (H@K), mean average precision (MAP), and area under the curve (AUC). The comparison indicators of consumption time include WHIT@K (W@K) and WMAP obtained based on residence time, as well as AUC.

[0149] The comparison results are shown in Table 1 below. The model corresponding to CTR in the table represents a model that only uses recommendation accuracy as a business indicator, ModelA is a sub-recommendation model that only uses recommendation accuracy as a business indicator, the model corresponding to DT represents a model that only uses consumption time as a business indicator, ModelB is a sub-recommendation model that only uses consumption time as a business indicator, the PO-EA model and the POW-RL model are recommendation models that use two business indicators, recommendation accuracy and consumption time, and objModel is the item recommendation model proposed in this application. From the experimental results, it can be seen that the sub-recommendation models in the item recommendation model proposed in this application are basically the best when compared on their respective single business indicators. At the same time, the item recommendation model established using the weight parameters determined by the actor-critic also performs better than the recommendation model based on a single business indicator.

[0150] Table 1 Comparison results

[0151]

[0152]

[0153] like Figure 6 As shown, Figure 6: is a schematic diagram of a recommended device provided in an embodiment of the present application, the device comprising:

[0154] Acquisition module 601, used to acquire characteristic information of a target user, the characteristic information including user portrait information and historical behavior information;

[0155] Processing module 602 is configured to input the feature information into an item recommendation model to determine a predicted score for each candidate item in the candidate item set for the target user, wherein the item recommendation model includes k sub-recommendation models, wherein each sub-recommendation model is obtained by training a neural network based on k weight parameters corresponding to the target user and a training sample set, wherein the k weight parameters are determined based on an actor-critic model and the training sample set, and k is a positive integer;

[0156] The processing module 602 is further configured to determine a target candidate item from the candidate item set according to the predicted score, and push the target candidate item to a client corresponding to the target user.

[0157] In one embodiment, the processing module 602 is specifically configured to:

[0158] Performing feature cross-processing on the user portrait information, the item information, and the corresponding click feature information to obtain a first feature of the target user;

[0159] encoding the context information to obtain a second feature of the target user;

[0160] The first feature and the second feature are input into an item recommendation model to determine a predicted score for each candidate item in a candidate item set for the target user.

[0161] In one embodiment, the processing module 602 is specifically configured to:

[0162] For any candidate item in the candidate item set, the feature information and the candidate item are input into each of the k sub-recommendation models, and the score of the target user on the business indicator corresponding to each sub-recommendation model output by each sub-recommendation model is obtained to obtain the score of the target user on the k business indicators, wherein different sub-recommendation models correspond to different business indicators;

[0163] Based on the scores of the target user on the k business indicators and the k weight parameters, a predicted score of any candidate item for the target user is determined.

[0164] In one embodiment, the processing module 602 is specifically configured to:

[0165] Obtaining a training sample set, where each sample in the training sample set includes an item clicked by a target user and a corresponding first real reward value and feature information of the target user, where the first real reward value is obtained based on indicator data of the target user on the k business indicators;

[0166] Inputting the item clicked by the target user and the corresponding characteristic information of the target user included in each sample into k neural networks respectively to obtain a second real reward value of the item clicked by the target user;

[0167] Determining an updated weight parameter for each of the k neural networks based on the second true reward value and the actor-critic model;

[0168] The network parameters of the k neural networks are adjusted based on the updated weight parameters and loss function of each neural network to obtain k sub-recommendation models, and an item recommendation model is determined based on the k sub-recommendation models.

[0169] In one embodiment, the processing module 602 is specifically configured to:

[0170] Inputting the item clicked by the target user and the corresponding characteristic information of the target user included in each sample into k neural networks respectively to obtain a first predicted reward value output by each of the k neural networks;

[0171] Determine a second real reward value of the item clicked by the target user based on the first predicted reward value output by each neural network, the first real reward value, and the loss function and weight parameters of each neural network.

[0172] In one embodiment, the processing module 602 is specifically configured to:

[0173] Inputting the feature information of the target user corresponding to the item clicked by the target user included in each sample into the actor model in the actor-critic model to obtain k prediction weight parameters output by the actor model;

[0174] Inputting the item clicked by the target user, the corresponding feature information of the target user, and the k prediction weight parameters included in each sample into the critic model in the actor-critic model to obtain a second predicted reward value for the item clicked by the target user output by the critic model;

[0175] Adjusting network parameters of the actor-critic model according to the second true reward value, the second predicted reward value, the k prediction weight parameters, and the total loss function of the actor-critic model;

[0176] An updated weight parameter of each of the k neural networks is determined based on the actor model in the adjusted actor-critic model.

[0177] In one embodiment, the processing module 602 is specifically configured to:

[0178] determining a loss function of the actor model and a loss function of the critic model, wherein the loss function of the actor model includes a policy gradient loss function and a weight parameter loss function;

[0179] An overall loss function of the actor-critic model is determined according to the loss function of the actor model and the loss function of the critic model.

[0180] In one embodiment, the processing module 602 is specifically configured to:

[0181] determining a loss value of the critic model according to the second predicted reward value, the second true reward value, and a loss function of the critic model;

[0182] determining a first loss value of the actor model based on the second predicted reward value, the policy probabilities of the k predicted weight parameters, and the policy gradient loss function of the actor model;

[0183] determining a second loss value of the actor model based on the weight parameters of the k neural networks, the k predicted weight parameters, and a weight parameter loss function of the actor model;

[0184] A total loss value is determined according to the loss value of the critic model, the first loss value and the second loss value of the actor model, and a total loss function of the actor-critic model, and network parameters of the actor-critic model are adjusted based on the total loss value.

[0185] Through the embodiments of the present application, the characteristic information of the target user can be obtained, and the predicted score of each candidate item in the candidate item set for the target user can be determined based on the characteristic information of the target user and the item recommendation model. Then, the target candidate item can be determined from the candidate item set based on the predicted score, and the target candidate item can be pushed to the client corresponding to the target user, so that the item recommendation model can make personalized recommendations for the target user. At the same time, the k sub-recommendation models included in the item recommendation model are obtained by training the neural network based on the k weight parameters and training sample sets corresponding to the target user, wherein the k weight parameters are determined based on the actor-critic model and the training sample set. In this way, the personalized weight parameters of each user can be determined by the actor-critic model, so that the item recommendation model can make personalized recommendations more effectively, and improve the accuracy of item recommendations and the promotion benefits of merchants.

[0186] like Figure 7 As shown, Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The internal structure of the device is as follows Figure 7 As shown, it includes: one or more processors 701, memory 702, and communication interface 702. The processor 701, memory 702, and communication interface 702 can be connected via a bus 704 or other means. The embodiment of the present application takes the connection via the bus 704 as an example.

[0187] Among them, the processor 701 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device. It can parse various instructions in the computer device and process various data of the computer device. For example, the CPU can be used to parse the power on and off instructions sent by the user to the computer device and control the computer device to perform power on and off operations; for another example, the CPU can transmit various interactive data between the internal structures of the computer device, etc. The communication interface 702 can optionally include a standard wired interface, a wireless interface (such as Wi-Fi, mobile communication interface, etc.), which is controlled by the processor 701 to send and receive data. The memory 702 (Memory) is a memory device in the computer device for storing programs and data. It can be understood that the memory 702 here can include both the built-in memory of the computer device and the extended memory supported by the computer device. The memory 702 provides storage space, which stores the operating system of the computer device, which may include but is not limited to: Windows system, Linux system, etc., and this application is not limited to this.

[0188] In one embodiment, the processor 701 is specifically configured to:

[0189] Obtaining target user's characteristic information, including user profile information and historical behavior information;

[0190] Inputting the feature information into an item recommendation model to determine a predicted score for each candidate item in a candidate item set for the target user, wherein the item recommendation model includes k sub-recommendation models, wherein each sub-recommendation model is obtained by training a neural network based on k weight parameters corresponding to the target user and a training sample set, wherein the k weight parameters are determined based on an actor-critic model and the training sample set, and k is a positive integer;

[0191] A target candidate item is determined from the candidate item set according to the predicted score, and the target candidate item is pushed to a client corresponding to the target user.

[0192] In one embodiment, the processor 701 is specifically configured to:

[0193] Performing feature cross-processing on the user portrait information, the item information, and the corresponding click feature information to obtain a first feature of the target user;

[0194] encoding the context information to obtain a second feature of the target user;

[0195] The first feature and the second feature are input into an item recommendation model to determine a predicted score for each candidate item in a candidate item set for the target user.

[0196] In one embodiment, the processor 701 is specifically configured to:

[0197] For any candidate item in the candidate item set, the feature information and the candidate item are input into each of the k sub-recommendation models, and the score of the target user on the business indicator corresponding to each sub-recommendation model output by each sub-recommendation model is obtained to obtain the score of the target user on the k business indicators, wherein different sub-recommendation models correspond to different business indicators;

[0198] Based on the scores of the target user on the k business indicators and the k weight parameters, a predicted score of any candidate item for the target user is determined.

[0199] In one embodiment, the processor 701 is specifically configured to:

[0200] Obtaining a training sample set, where each sample in the training sample set includes an item clicked by a target user and a corresponding first real reward value and feature information of the target user, where the first real reward value is obtained based on indicator data of the target user on the k business indicators;

[0201] Inputting the item clicked by the target user and the corresponding characteristic information of the target user included in each sample into k neural networks respectively to obtain a second real reward value of the item clicked by the target user;

[0202] Determining an updated weight parameter for each of the k neural networks based on the second true reward value and the actor-critic model;

[0203] The network parameters of the k neural networks are adjusted based on the updated weight parameters and loss function of each neural network to obtain k sub-recommendation models, and an item recommendation model is determined based on the k sub-recommendation models.

[0204] In one embodiment, the processor 701 is specifically configured to:

[0205] Inputting the item clicked by the target user and the corresponding characteristic information of the target user included in each sample into k neural networks respectively to obtain a first predicted reward value output by each of the k neural networks;

[0206] Determine a second real reward value of the item clicked by the target user based on the first predicted reward value output by each neural network, the first real reward value, and the loss function and weight parameters of each neural network.

[0207] In one embodiment, the processor 701 is specifically configured to:

[0208] Inputting the feature information of the target user corresponding to the item clicked by the target user included in each sample into the actor model in the actor-critic model to obtain k prediction weight parameters output by the actor model;

[0209] Inputting the item clicked by the target user, the corresponding feature information of the target user, and the k prediction weight parameters included in each sample into the critic model in the actor-critic model to obtain a second predicted reward value for the item clicked by the target user output by the critic model;

[0210] Adjusting network parameters of the actor-critic model according to the second true reward value, the second predicted reward value, the k prediction weight parameters, and the total loss function of the actor-critic model;

[0211] An updated weight parameter of each of the k neural networks is determined based on the actor model in the adjusted actor-critic model.

[0212] In one embodiment, the processor 701 is specifically configured to:

[0213] determining a loss function of the actor model and a loss function of the critic model, wherein the loss function of the actor model includes a policy gradient loss function and a weight parameter loss function;

[0214] An overall loss function of the actor-critic model is determined according to the loss function of the actor model and the loss function of the critic model.

[0215] In one embodiment, the processor 701 is specifically configured to:

[0216] determining a loss value of the critic model according to the second predicted reward value, the second true reward value, and a loss function of the critic model;

[0217] determining a first loss value of the actor model based on the second predicted reward value, the policy probabilities of the k predicted weight parameters, and the policy gradient loss function of the actor model;

[0218] determining a second loss value of the actor model based on the weight parameters of the k neural networks, the k predicted weight parameters, and a weight parameter loss function of the actor model;

[0219] A total loss value is determined according to the loss value of the critic model, the first loss value and the second loss value of the actor model, and a total loss function of the actor-critic model, and network parameters of the actor-critic model are adjusted based on the total loss value.

[0220] Through the embodiments of the present application, the characteristic information of the target user can be obtained, and the predicted score of each candidate item in the candidate item set for the target user can be determined based on the characteristic information of the target user and the item recommendation model. Then, the target candidate item can be determined from the candidate item set based on the predicted score, and the target candidate item can be pushed to the client corresponding to the target user, so that the item recommendation model can make personalized recommendations for the target user. At the same time, the k sub-recommendation models included in the item recommendation model are obtained by training the neural network based on the k weight parameters and training sample sets corresponding to the target user, wherein the k weight parameters are determined based on the actor-critic model and the training sample set. In this way, the personalized weight parameters of each user can be determined by the actor-critic model, so that the item recommendation model can make personalized recommendations more effectively, and improve the accuracy of item recommendations and the promotion benefits of merchants.

[0221] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described preferred method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0222] One or more embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps performed in the above-described method embodiments.

[0223] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A recommendation method, characterized in that: The method comprises: Obtaining target user's characteristic information, including user profile information and historical behavior information; Inputting the feature information into an item recommendation model to determine a predicted score for each candidate item in the candidate item set for the target user; Determine a target candidate item from the candidate item set according to the prediction score, and push the target candidate item to a client corresponding to the target user; The method for determining the item recommendation model includes: Obtaining a training sample set, where each sample in the training sample set includes an item clicked by a target user, a corresponding first real reward value, and feature information of the target user, where the first real reward value is obtained based on indicator data of the target user on k business indicators; Inputting the item clicked by the target user and the corresponding characteristic information of the target user included in each sample into k neural networks respectively to obtain a second real reward value of the item clicked by the target user; Inputting the feature information of the target user corresponding to the item clicked by the target user included in each sample into the actor model in the actor-critic model to obtain k prediction weight parameters output by the actor model; Inputting the item clicked by the target user, the corresponding feature information of the target user, and the k prediction weight parameters included in each sample into the critic model in the actor-critic model to obtain a second predicted reward value for the item clicked by the target user output by the critic model; adjusting the network parameters of the actor-critic model according to the second true reward value, the second predicted reward value, the k predicted weight parameters, and the total loss function of the actor-critic model; and determining updated weight parameters of each of the k neural networks based on the actor model in the adjusted actor-critic model; The network parameters of the k neural networks are adjusted based on the updated weight parameters and loss function of each neural network to obtain k sub-recommendation models, and an item recommendation model is determined based on the k sub-recommendation models.

2. The method according to claim 1, characterized in that The historical behavior information includes item information and corresponding click feature information and context information. Inputting the feature information into an item recommendation model to determine a predicted score for each candidate item in a candidate item set for the target user includes: Performing feature cross-processing on the user portrait information, the item information, and the corresponding click feature information to obtain a first feature of the target user; encoding the context information to obtain a second feature of the target user; The first feature and the second feature are input into an item recommendation model to determine a predicted score for each candidate item in a candidate item set for the target user.

3. The method according to claim 1, characterized in that Inputting the feature information into an item recommendation model to determine a predicted score for each candidate item in a candidate item set for the target user includes: For any candidate item in the candidate item set, the feature information and the candidate item are input into each of the k sub-recommendation models, and the score of the target user on the business indicator corresponding to each sub-recommendation model output by each sub-recommendation model is obtained to obtain the score of the target user on the k business indicators, wherein different sub-recommendation models correspond to different business indicators; Based on the score of the target user on the k business indicators and the updated weight parameters of each neural network in the k neural networks, a predicted score of any candidate item for the target user is determined.

4. The method according to claim 1, wherein Inputting the item clicked by the target user and the corresponding characteristic information of the target user included in each sample into k neural networks respectively to obtain a second real reward value of the item clicked by the target user, including: Inputting the item clicked by the target user and the corresponding characteristic information of the target user included in each sample into k neural networks respectively to obtain a first predicted reward value output by each of the k neural networks; Determine a second real reward value of the item clicked by the target user based on the first predicted reward value output by each neural network, the first real reward value, and the loss function and weight parameters of each neural network.

5. The method according to claim 1, wherein Before adjusting the network parameters of the actor-critic model according to the second true reward value, the second predicted reward value, the k prediction weight parameters, and the total loss function of the actor-critic model, the method further includes: determining a loss function of the actor model and a loss function of the critic model, wherein the loss function of the actor model includes a policy gradient loss function and a weight parameter loss function; An overall loss function of the actor-critic model is determined according to the loss function of the actor model and the loss function of the critic model.

6. The method according to claim 1, characterized in that The adjusting the network parameters of the actor-critic model according to the second true reward value, the second predicted reward value, the k prediction weight parameters, and the total loss function of the actor-critic model includes: determining a loss value of the critic model according to the second predicted reward value, the second true reward value, and a loss function of the critic model; determining a first loss value of the actor model based on the second predicted reward value, the policy probabilities of the k predicted weight parameters, and the policy gradient loss function of the actor model; determining a second loss value of the actor model based on the weight parameters of the k neural networks, the k predicted weight parameters, and a weight parameter loss function of the actor model; A total loss value is determined according to the loss value of the critic model, the first loss value and the second loss value of the actor model, and a total loss function of the actor-critic model, and network parameters of the actor-critic model are adjusted based on the total loss value.

7. A recommendation device, characterized in that: The device comprises: An acquisition module is used to obtain characteristic information of a target user, including user profile information and historical behavior information; a processing module, configured to input the feature information into an item recommendation model to determine a predicted score for each candidate item in the candidate item set for the target user; The processing module is further configured to determine a target candidate item from the candidate item set according to the predicted score, and push the target candidate item to a client corresponding to the target user; The processing module is further configured to obtain a training sample set, wherein each sample in the training sample set includes an item clicked by a target user and a corresponding first real reward value and feature information of the target user, wherein the first real reward value is obtained based on the indicator data of the target user on k business indicators; the item clicked by the target user included in each sample and the corresponding feature information of the target user are respectively input into k neural networks to obtain a second real reward value of the item clicked by the target user; the feature information of the target user corresponding to the item clicked by the target user included in each sample is input into the actor model in the actor-critic model to obtain k prediction weight parameters output by the actor model; the item clicked by the target user included in each sample and the corresponding feature information of the target user are respectively input into k neural networks to obtain a second real reward value of the item clicked by the target user; the feature information of the target user corresponding to the item clicked by the target user included in each sample is input into the actor model in the actor-critic model to obtain k prediction weight parameters output by the actor model; the item clicked by the target user included in each sample and the corresponding feature information of the target user are respectively input into k neural networks to obtain k prediction weight parameters output by the actor model; The characteristic information of the target user and the k prediction weight parameters are input into the critic model in the actor-critic model to obtain the second predicted reward value of the item clicked by the target user output by the critic model; the network parameters of the actor-critic model are adjusted according to the second real reward value, the second predicted reward value, the k prediction weight parameters and the total loss function of the actor-critic model; the updated weight parameters of each neural network in the k neural networks are determined based on the actor model in the adjusted actor-critic model; the network parameters of the k neural networks are adjusted based on the updated weight parameters of each neural network and the loss function to obtain k sub-recommendation models, and an item recommendation model is determined based on the k sub-recommendation models.

8. A computer device, characterized in that: The invention comprises a processor, a communication interface and a memory, wherein the processor, the communication interface and the memory are connected to each other, wherein the memory stores an executable program code, and the processor is used to call the executable program code to execute the recommendation method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the recommendation method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product includes a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, it is used to implement the recommendation method according to any one of claims 1 to 6.

Images