Recommended content sorting method, sorting model construction method, equipment and program product

Abstract

The invention discloses a recommended content sorting method, a sorting model construction method, equipment and a program product, and the method comprises the steps: firstly carrying out the strategy decomposition processing of a main strategy based on the attribute of to-be-sorted content, and obtaining at least one subtask through decomposition; obtaining a multi-task data set according to the user historical sequence data of the to-be-sorted contents; then, constructing a first multi-task strategy enhancement model by utilizing each sub-task, and training by utilizing the multi-task data set to obtain a second multi-task strategy enhancement model; for each piece of to-be-sorted content, performing policy value evaluation on the to-be-sorted content by using a second multi-task policy enhancement model to obtain a content income value; and sorting the recommended contents according to the content revenue value of each to-be-sorted content. The recommendation content sorting method based on strategy enhancement has the advantages of being good in recommendation sorting effect and low in cost for small-flow product recommendation positions.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a method for ranking recommended content, a method for building a ranking model, a device, and a program product. Background Art

[0002] For the hotel reservation business, users mainly search with clear intentions, and hotel search ranking accounts for a large part of the traffic in the overall application. Hotel recommendation ranking is a relatively small module in the hotel reservation business, which is scattered in multiple businesses. The exposure traffic of many recommended locations accounts for a small proportion of the traffic in the overall application, and there are many recommendation scenarios.

[0003] Existing recommendation ranking models generally directly model product exposure and conversion as optimization goals. This modeling method is not applicable to recommendation positions with low traffic.

[0004] For hotel recommendation ranking modeling with many recommended positions and a small proportion of traffic, first of all, the exposure-conversion modeling method cannot be used. If conversion target modeling is performed for each product recommendation position, and there are more than ten recommended product positions, there will be more than ten models, and the manpower, material and time costs are huge. The output income of some positions cannot cover their costs; secondly, the model effect will be subject to the exposure traffic of the recommended product position and the scale of training data. Because the traffic of the recommended product position is small, the model training effect for the product position is difficult to guarantee.

[0005] Therefore, it is urgent to invent a method for sorting recommended content for multiple low-traffic recommended product positions to solve the problems of poor sorting effect and high cost in the prior art for multiple low-traffic recommended product positions. Summary of the invention

[0006] In view of this, embodiments of the present invention provide a method for ranking recommended content, a method for building a ranking model, a device, and a program product, which at least partially solve the problems existing in the prior art.

[0007] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0008] In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions:

[0009] According to a first aspect of an embodiment of the present invention, a method for ranking recommended content is provided, the method comprising:

[0010] Decomposing the main strategy based on the attributes of the content to be sorted to obtain at least one subtask;

[0011] Constructing an initial data set based on historical user sequence data of the content to be sorted, and performing data decomposition processing on the initial data set based on each subtask to obtain a multi-task data set;

[0012] Using each subtask to build a first multi-task strategy reinforcement model, based on the experience replay mechanism, using the multi-task data set to train the first multi-task strategy reinforcement model, to obtain a trained second multi-task strategy reinforcement model;

[0013] For each content to be sorted, using the second multi-task strategy reinforcement model to evaluate the strategy value of the content to be sorted, and obtain the content benefit value;

[0014] The recommended contents are sorted according to the content revenue value of each content to be sorted.

[0015] Furthermore, the main strategy is decomposed based on the attributes of the content to be sorted, and at least one subtask is obtained by decomposing the main strategy, including:

[0016] According to the basic attributes of the content to be sorted, the policy space corresponding to the content to be sorted is decomposed to obtain at least one sub-policy space, and the sub-policy space is used as a subtask.

[0017] Furthermore, an initial data set is constructed based on the user historical sequence data of the content to be sorted, and the initial data set is decomposed based on each subtask to obtain a multi-task data set, including:

[0018] Construct the initial data set D based on the user historical sequence data of the content to be sorted n =(s n ,a n ,r n ), where s n For user input status, a n is the user policy, r n is the strategy return, n is a positive integer;

[0019] The initial data set is decomposed based on each subtask to obtain a multi-task data set D' n =([SC,S A ] n ,[A,B,...] n ,[R A ,R B ,...] n ), where [A,V,...] n is the subtask data after strategy decomposition, [R A ,R B ,...] nis the strategy benefit corresponding to the subtask data after strategy decomposition, S C is the shared basic feature of each subtask in the current state, S A It is the status feature of each subtask.

[0020] Further, a first multi-task strategy reinforcement model is constructed using each subtask, and based on the experience replay mechanism, the first multi-task strategy reinforcement model is trained using the multi-task data set to obtain a trained second multi-task strategy reinforcement model, including:

[0021] Using each subtask to build a first multi-task strategy reinforcement model, wherein the first multi-task strategy reinforcement model includes a long short-term memory network LSTM and a deep neural network DNN;

[0022] For each subtask, the subtask state features and shared basic features in the multi-task dataset are input into a long short-term memory network LSTM to obtain a state feature vector;

[0023] Performing feature splicing processing using the state feature vector and the target strategy basic feature to obtain a feature splicing result, wherein the target strategy basic feature is an interval feature of the subtask data;

[0024] The feature splicing result is input into the deep neural network DNN, and the subtask strategy parameters are updated based on the principle of maximizing the subtask strategy benefit until the multi-task strategy benefit loss function is minimized, thereby obtaining a second multi-task strategy reinforcement model.

[0025] Furthermore, for each content to be sorted, the second multi-task strategy reinforcement model is used to evaluate the strategy value of the content to be sorted to obtain the content benefit value, including:

[0026] For each to-be-sorted content, the loss weight of each subtask in the model is strengthened according to the second multi-task strategy to obtain the interval distribution probability corresponding to each subtask;

[0027] Using the interval distribution probability corresponding to each subtask, the attribute corresponding to the content to be sorted is evaluated to obtain the attribute benefit value;

[0028] The attribute benefit value is used to obtain the content benefit value of the content to be sorted.

[0029] Furthermore, using the attribute benefit value to obtain the content benefit value of the content to be sorted, the method further includes:

[0030] A weighted summation process is performed on all attribute revenue values ​​of the content to be sorted, and a weighted weight value is set based on the business demand of the recommended product position to obtain the content revenue value of the content to be sorted.

[0031] Furthermore, the method is used for hotel recommendation ranking, comprising:

[0032] The main strategy is decomposed based on the attributes of the hotels to be sorted, and at least one hotel subtask is decomposed. The attributes of the hotels to be sorted include price, star rating, brand, location code, POI type, distance and review quality. The main strategy is to order the hotel;

[0033] An initial data set is constructed based on the user historical order sequence data of the hotels to be sorted, and the initial data set is decomposed based on each hotel subtask to obtain a multi-task data set;

[0034] Using each hotel subtask to build a first multi-task strategy reinforcement model, based on the experience replay mechanism, using the multi-task data set to train the first multi-task strategy reinforcement model, to obtain a trained second multi-task strategy reinforcement model;

[0035] For each hotel to be sorted, using the second multi-task strategy reinforcement model to evaluate the strategy value of the hotel to be sorted, and obtain the hotel recommendation revenue value;

[0036] The recommended hotels are ranked according to the hotel recommendation revenue value of each hotel to be ranked.

[0037] According to a second aspect of an embodiment of the present invention, a method for constructing a ranking model is provided, the method comprising: performing a policy decomposition process on a main policy based on attributes of data to be trained, and decomposing the main policy to obtain at least one subtask;

[0038] According to the historical sequence data of the data to be trained, an initial data set is constructed, and data decomposition processing is performed on the initial data set based on each subtask to obtain a multi-task data set;

[0039] A first multi-task strategy reinforcement model is constructed using each subtask, and based on an experience replay mechanism, the first multi-task strategy reinforcement model is trained using the multi-task data set to obtain a trained second multi-task strategy reinforcement model.

[0040] According to a third aspect of an embodiment of the present invention, a device is provided, the device comprising: a processor and a memory;

[0041] The memory is used to store one or more program instructions;

[0042] The processor is used to run one or more program instructions to execute the steps of a recommended content ranking method as described in any one of the above items or a ranking model construction method as described in the above item.

[0043] According to a fourth aspect of an embodiment of the present invention, a computer program product is provided, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, which, when executed by a computer, enable the computer to implement a method for ranking recommended content as described in any one of the above items or a method for building a ranking model as described above.

[0044] The embodiments of the present invention have the following advantages:

[0045] 1. Based on the strategy decomposition method of the attributes of the content to be recommended, a single infinite or large-scale action strategy space is decomposed into multiple task-limited strategy spaces; the decomposed multi-task strategy reduces the problem of poor model training effect of the original single-task infinite strategy, so that the policy gradient reinforcement learning algorithm can be effectively trained and obtain better performance.

[0046] 2. The policy gradient decomposition reinforcement learning algorithm aims to select a strategy that maximizes revenue. Different from the supervised learning algorithm with conversion as the goal, it does not need to collect a large amount of exposure conversion data for product positions, thus solving the problem of small traffic in product recommendation positions, insufficient training data and poor performance when modeling with conversion as the goal.

[0047] 3. The policy gradient decomposition reinforcement learning algorithm is trained based on the data of the entire site. Therefore, it has the natural cross-scenario recommendation sorting characteristics and can be applied to the sorting of all recommendation scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0049] Figure 1 A flowchart of a method for ranking recommended content provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of the principle of constructing a data set provided by an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of the structure of a multi-task strategy reinforcement model provided by an embodiment of the present invention;

[0052] Figure 4 A schematic diagram for comparing the recommended sorting effects provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0054] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0056] Reinforcement Learning (RL): A machine learning paradigm in which an agent learns how to take actions to maximize the expected cumulative reward by interacting with its environment.

[0057] E&E: usually refers to "Exploration and Exploitation", which is a core concept in reinforcement learning algorithms.

[0058] Policy Gradient: It is a reinforcement learning (RL) technique used to directly optimize the policy function, directly parameterize the policy, that is, the behavior of the agent, and optimize the policy to obtain the model parameters through the gradient ascent method. Assume that the target policy π θ is a stochastic strategy and is differentiable everywhere, where θ is the corresponding parameter. For such π θ, a model can be used to model this policy function. The input of this model is the current state s of the model, and the output is the probability distribution of the next action a. Based on the probability distribution of the action, find an optimal strategy and maximize the expected return of this strategy in the environment. The objective function of policy learning is as follows:

[0059]

[0060] Among them, s is the current state of the agent, The value function in the policy environment, E is the expected value. The meaning of policy gradient is to choose the strategy with the maximum benefit in the current state. The formula for solving the gradient of the objective function is as follows:

[0061]

[0062] in, is the value of action a under the current policy π when in state s.

[0063] The above formula is the policy gradient formula. The iteration of the gradient allows the policy to obtain Q with greater value. For the value evaluation of limited actions a, the parameters can be estimated by exploring or recycling the environment. However, for large-scale actions a, most of the data is sparse, and it is impractical and costly to evaluate all actions.

[0064] In the process of interactive exploration with the environment, reinforcement learning requires a portion of traffic to be random, among which the most typical recommendation algorithm is E&E. The E&E algorithm balances the trade-off between trying new strategies (exploration) and sticking to the best known strategy (exploitation) during model learning. When a reinforcement learning agent "exploits" in the environment, it selects those strategies that are considered to bring the highest reward based on current knowledge. Exploitation ensures that the agent achieves better performance in a known environment. "Exploration" refers to the behavior when the agent tries new behaviors to understand the environment. This may involve choosing actions that do not necessarily bring the greatest reward immediately. In the long run, exploration can help the agent learn more about the environment and may discover better strategies than the current strategy. The experience replay mechanism has the following advantages: first, experience can be reused to improve data efficiency; second, there is a strong correlation between samples in online learning, and the offline experience replay mechanism reduces the strong correlation between samples, which can reduce the variance of model updates; second, the batch sample replay mechanism can smoothly avoid fluctuations or divergence in parameters.

[0065] E&E is an online learning method, which is opposite to offline learning. In reinforcement learning, offline learning uses the experience replay mechanism. The experience of the agent at each time step is stored in the data, and many rounds are aggregated into a replay memory. If the data set D = e1,...,e x , where e t =(s t ,a t ,r t ,s t+1 ), s i Represents the state, a i is the action strategy adopted, r i is the gain obtained, s i+1 It is the next state. In model training, the training sample updated by the policy gradient randomly samples the experience sample e from the stored sample data set D, and inputs the sample e into the model for training.

[0066] MMoE (Multi-gate Mixture of Experts): A model architecture for multi-task learning (MTL) that aims to improve the learning effect of multiple related tasks through shared representations and task-specific expert networks. It achieves collaboration and information sharing between tasks by introducing multiple expert networks and multiple gate networks. Multi-task and multi-objective supervised learning algorithms, such as MMOE, are a popular algorithm that minimizes the error between multiple target predictions and true labels. Supervised learning and policy gradient learning are two different paradigms in mechanism learning. They have similarities. If the fixed policy return r is 1 and the learning sequence is a single step, the policy gradient can be forced to be converted into supervised learning for training. Therefore, supervised learning and policy gradient learning algorithms are essentially different in model building ideas, and there is not much difference in target learning, or supervised learning is a special case of policy gradient learning.

[0067] Supervised Learning: A machine learning paradigm in which the model makes predictions by learning from labeled training data. The training data consists of a set of input-output pairs, and the goal is to enable the model to accurately predict new, unseen data by minimizing the error between the predicted output and the true label.

[0068] Conventional policy gradient-based reinforcement learning algorithms are mainly used for single-target and limited action strategies. However, large-scale or infinite action spaces often fail to converge due to very sparse data and insufficient samples, resulting in relatively poor model effects.

[0069] To this end, an embodiment of the present invention adopts a target action decomposition algorithm to decompose single-task reinforcement learning into multi-task reinforcement learning, and weightedly combines the sub-task strategy values ​​of multiple tasks as the value assessment of the target strategy, so that the intelligent agent adopts a strategy that maximizes the current benefit.

[0070] Figure 1 FIG. 2 shows a flow chart of a method for ranking recommended contents according to an embodiment of the present invention.

[0071] like Figure 1 As shown, the recommended content ranking method based on policy enhancement according to an embodiment of the present invention may include step S100, step S200, step S300, step S400 and step S500.

[0072] In step S100, the main policy is decomposed based on the attributes of the content to be sorted, and at least one subtask is obtained by decomposing the main policy.

[0073] Specifically, the above steps include: decomposing the policy space corresponding to the content to be sorted according to the basic attributes of the content to be sorted, obtaining at least one sub-policy space, and using the sub-policy space as a subtask.

[0074] Next, in step S200, an initial data set is constructed according to the user historical sequence data of the content to be sorted, and data decomposition processing is performed on the initial data set based on each subtask to obtain a multi-task data set.

[0075] refer to Figure 2 The above steps specifically include: according to the user history sequence data of the content to be sorted, assuming that a certain time of the user is t, then the time before t is the past history, and by randomly selecting t, the user history sequence data is randomly divided into the initial data set D n =(s n ,a n ,r n ), where s n For user input status, a n is the user policy, r n is the strategy return, and n is a positive integer.

[0076] The initial data set is decomposed based on each subtask to obtain the multi-task data set D' n =([S C ,S A ] n ,[A,B,...] n ,[R A ,B B ,...] n ), where [A,B,...] n is the subtask data after strategy decomposition, [RA ,R B ,...] n is the strategy benefit corresponding to the subtask data after strategy decomposition, S C is the shared basic feature of each subtask in the current state, S A It is the status feature of each subtask.

[0077] Next, in step S300, a first multi-task strategy reinforcement model is constructed using each subtask, and based on the experience replay mechanism, the first multi-task strategy reinforcement model is trained using a multi-task data set to obtain a trained second multi-task strategy reinforcement model.

[0078] The above steps specifically include: first, using each subtask to construct a first multi-task strategy reinforcement model, the first multi-task strategy reinforcement model is based on the long short-term memory network LSTM and the deep neural network DNN as the network structure, the multi-task strategy reinforcement model is as follows: Figure 3 shown.

[0079] For each subtask, the subtask state feature S in the multi-task dataset is A and shared basic features S C Input to the long short-term memory network LSTM to obtain the state feature vector; the input of LSTM is S C and S A The output is S C With S A The state feature vector of C Refers to the context state of the model input, that is, the basic features of the current state, which is shared by each subtask. A Refers to the characteristics of the current subtask; then the state feature vector and the target strategy basic features are used for feature concatenation to obtain the feature concatenation result. The target strategy basic features are the interval features of the subtask data; the feature concatenation result is input into the 2-layer deep neural network DNN to obtain the strategy benefit. The above strategy benefit is the model in the input state S C With S A The revenue value generated when .

[0080] Based on the principle of maximizing the subtask strategy benefit, the subtask strategy parameters are updated until the multitask strategy benefit loss function is minimized, and the second multitask strategy reinforcement model is obtained.

[0081] Among them, the overall loss of the model is the weighted sum of the losses of each subtask strategy. The setting of the loss weight of the present invention is based on unifying the losses of each subtask strategy to the same scale. The multi-task strategy profit loss function is as follows:

[0082]

[0083] Among them, σ is the loss variance. The loss weight with large variance σ is updated less during learning. The variance is limited by log(σ) to prevent it from being infinite.

[0084] The embodiment of the present invention performs model training through an experience replay mechanism. If online learning is used to explore with random traffic, the overall revenue will be affected, which is not allowed in actual recommendation business applications. Therefore, replaying historical data is the best choice.

[0085] In step S400, for each content to be sorted, a second multi-task policy reinforcement model is used to perform a policy value assessment on the content to be sorted to obtain the content revenue value.

[0086] The above steps specifically include: for each content to be sorted, the loss weight of each subtask in the model is strengthened according to the second multi-task strategy to obtain the interval distribution probability corresponding to each subtask; then, using the interval distribution probability corresponding to each subtask, the attributes corresponding to the content to be sorted are evaluated and processed to obtain the attribute benefit value; finally, the attribute benefit value is used to accumulate the content benefit value of the content to be sorted.

[0087] Alternatively, in another example, a weighted summation process may be performed on all attribute revenue values ​​of the content to be sorted, and a weighted value may be set based on the business requirements of the recommended product position to obtain the content revenue value of the content to be sorted.

[0088] Alternatively, in another example, a linear model may be built based on the conversion data of the product positions to obtain the content revenue value of the content to be ranked.

[0089] Finally, in step S500, the recommended contents are ranked according to the content revenue value of each content to be ranked.

[0090] Figure 4 The figure shows a schematic diagram of the sorting effects of the recommended content sorting method provided by an embodiment of the present invention and the existing sorting method, wherein the sorting effect of the method provided by an embodiment of the present invention is an experimental group, and the sorting effect of the existing sorting model is a control group, and the order conversion rate in the four product recommendation positions is between 10% and 30%.

[0091] The embodiment of the present invention discloses a method for ranking recommended content, which includes: firstly, performing a policy decomposition process on the main policy based on the attributes of the content to be ranked, and decomposing to obtain at least one subtask; then obtaining a multi-task data set according to the user historical sequence data of the content to be ranked; then using each subtask to construct a first multi-task policy reinforcement model, and using the multi-task data set to train, to obtain a second multi-task policy reinforcement model; for each content to be ranked, using the second multi-task policy reinforcement model to evaluate the policy value of the content to be ranked, and obtain the content revenue value; and ranking the recommended content according to the content revenue value of each content to be ranked. The recommended content ranking method based on policy reinforcement disclosed in the present invention has the advantages of good recommendation ranking effect and low cost for the recommendation position of small-traffic products. The embodiment of the present invention copes with the challenges brought by a single infinite action policy space, and provides new possibilities for reinforcement learning in a wider range of application fields. Future work can optimize the action space decomposition, and conduct deeper exploration and experiments on the optimization value strategy selection evaluation method.

[0092] In addition, an embodiment of the present invention also provides a method for ranking recommended content for hotel recommendation. In the hotel recommendation business, hotel recommendation has obvious characteristics. On the one hand, there are many product recommendation positions, and on the other hand, the traffic of each position is small and unevenly distributed. The existing modeling based on the exposure conversion of product recommendation positions is difficult to achieve good results due to the small scale of training data, and it is impossible to make a separate recommendation model for each small business, which will incur great costs and little benefits. However, the entire site has a large amount of user historical data, such as exposure, clicks, and orders, and it is urgent to use the data of the entire site to train a model that can be applied to all recommendation scenarios.

[0093] An embodiment of the present invention provides a method for sorting recommended content for hotel recommendation sorting, which specifically includes: first, performing strategy decomposition processing on a main strategy based on the attributes of a hotel to be sorted, and decomposing to obtain at least one hotel subtask, wherein the attributes of the hotel to be sorted include price, star rating, brand, location code, POI type, distance, and review quality, and the main strategy is an order hotel.

[0094] Then, based on the historical order sequence data of the users of the hotels to be sorted, an initial data set is constructed, and the initial data set is decomposed based on each hotel subtask to obtain a multi-task data set.

[0095] The above steps specifically include: according to the user's historical order sequence data of the hotel to be sorted, assuming that a certain time of the user is t, then the time before t is the past history, and by randomly selecting t, the user's historical order sequence data is randomly divided into the initial data set D n =(s n ,a n ,r n ), where sn For user input status, a n Order strategy for users, r n is the strategy return, and n is a positive integer.

[0096] Next, the initial data set is decomposed based on each subtask to obtain the multi-task data set D' n =([S C ,S A ] n ,[A,B,...] n ,[R A ,R B ,...] n ), where [A,B,...] n Subtask data after strategy decomposition, such as price data, star data, brand data, etc. A ,R B ,...] n is the strategy benefit corresponding to the subtask data after strategy decomposition. For example, if the price range of the hotel ordered by the user is 300-400 yuan, then the strategy benefit of the price subtask in the range of 300-400 yuan is 1. C is the shared basic features of each subtask in the current state, such as city, S A It is the status feature of each subtask. For example, the status feature of the price subtask is the user's historical order price.

[0097] A first multi-task strategy reinforcement model is constructed using each hotel sub-task. Based on the experience replay mechanism, the first multi-task strategy reinforcement model is trained using a multi-task data set to obtain a trained second multi-task strategy reinforcement model.

[0098] The above steps specifically include: first, using each subtask to construct a first multi-task strategy reinforcement model, the first multi-task strategy reinforcement model is based on the long short-term memory network LSTM and the deep neural network DNN as the network structure, the multi-task strategy reinforcement model is as follows: Figure 3 For each subtask, the subtask state feature S in the multi-task dataset is A and shared basic features S C Input to the long short-term memory network LSTM to obtain the state feature vector; the input of LSTM is S C and S A The output is S C With S A The state feature vector of C Refers to the context state of the model input, that is, the basic features of the current state, which is shared by each subtask. ARefers to the characteristics of the current subtask; then the state feature vector and the target strategy basic features are used for feature concatenation to obtain the feature concatenation result. The target strategy basic features are the interval features of the subtask data; the feature concatenation result is input into the 2-layer deep neural network DNN to obtain the strategy benefit. The above strategy benefit is the model in the input state S C With S A When the profit value is generated, the subtask strategy parameters are updated based on the principle of maximizing the profit of the subtask strategy until the multi-task strategy profit loss function is minimized, and the second multi-task strategy reinforcement model is obtained. The recommended business takes the order as the main optimization goal. Based on the current state, if an order is placed, the profit is 1, and if no order is placed, the profit is 0. It is not used as a learning directory.

[0099] For each hotel to be sorted, the second multi-task strategy reinforcement model is used to evaluate the strategic value of the hotel to be sorted, and the hotel recommendation revenue value is obtained.

[0100] Finally, the recommended hotels are ranked according to the hotel recommendation revenue value of each hotel to be ranked.

[0101] In addition, an embodiment of the present invention also provides a method for constructing a sorting model, which specifically includes: performing strategy decomposition processing on the main strategy based on the attributes of the data to be trained, and decomposing it to obtain at least one sub-task; constructing an initial data set based on the historical sequence data of the data to be trained, and performing data decomposition processing on the initial data set based on each sub-task to obtain a multi-task data set; using each sub-task to construct a first multi-task strategy reinforcement model, and based on the experience replay mechanism, using the multi-task data set to train the first multi-task strategy reinforcement model to obtain a trained second multi-task strategy reinforcement model.

[0102] In addition, an embodiment of the present invention also provides a device, which includes: a processor and a memory; the memory is used to store one or more program instructions; the processor is used to run one or more program instructions to execute the steps of a recommended content ranking method as described in any of the above items or a ranking model construction method as described above.

[0103] In addition, an embodiment of the present invention further provides a computer program product, which includes computer program instructions, and when the computer program instructions are executed by a processor, the steps of a method for ranking recommended content as described in any of the above items or a method for building a ranking model as described above are implemented.

[0104] In the embodiment of the present invention, the processor may be an integrated circuit chip with signal processing capability. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware. The storage medium may be a memory, for example, a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DRRAM).The storage medium described in the embodiment of the present invention is intended to include but is not limited to these and any other suitable types of memory. Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by a combination of hardware and software. When the software is applied, the corresponding function can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication medium includes any medium that is convenient for transmitting a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access. Although the present invention has been described in detail above with general descriptions and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to it on the basis of the present invention. Therefore, these modifications or improvements made on the basis of not departing from the spirit of the present invention all belong to the scope of protection claimed in the present invention.

Claims

1. A method for ranking recommended content, characterized in that: The method comprises: Decomposing the main strategy based on the attributes of the content to be sorted to obtain at least one subtask; Constructing an initial data set based on historical user sequence data of the content to be sorted, and performing data decomposition processing on the initial data set based on each subtask to obtain a multi-task data set; Using each subtask to build a first multi-task strategy reinforcement model, based on the experience replay mechanism, using the multi-task data set to train the first multi-task strategy reinforcement model, to obtain a trained second multi-task strategy reinforcement model; For each content to be sorted, using the second multi-task strategy reinforcement model to evaluate the strategy value of the content to be sorted, and obtain the content benefit value; The recommended contents are sorted according to the content revenue value of each content to be sorted.

2. A method for ranking recommended content according to claim 1, characterized in that: The main policy is decomposed based on the attributes of the content to be sorted, and at least one subtask is obtained by decomposing the main policy, including: According to the basic attributes of the content to be sorted, the policy space corresponding to the content to be sorted is decomposed to obtain at least one sub-policy space, and the sub-policy space is used as a subtask.

3. A method for ranking recommended content according to claim 1, characterized in that: An initial data set is constructed based on the user historical sequence data of the content to be sorted, and the initial data set is decomposed based on each subtask to obtain a multi-task data set, including: Construct the initial data set D based on the user historical sequence data of the content to be sorted n =(s n , a n , r n ), where s n For user input status, a n is the user policy, r n is the strategy return, n is a positive integer; The initial data set is decomposed based on each subtask to obtain a multi-task data set D′n=([S C , S A ] n , [A, B, ...] n ,[R A , R B , ...] n ), where [A, B, ...] n is the subtask data after strategy decomposition, [R A , R B , ...] n is the strategy benefit corresponding to the subtask data after strategy decomposition, S C is the shared basic feature of each subtask in the current state, S A It is the status feature of each subtask.

4. A method for ranking recommended content according to claim 1, characterized in that: Using each subtask to build a first multi-task strategy reinforcement model, based on the experience replay mechanism, using the multi-task data set to train the first multi-task strategy reinforcement model, to obtain a trained second multi-task strategy reinforcement model, including: Using each subtask to build a first multi-task strategy reinforcement model, wherein the first multi-task strategy reinforcement model includes a long short-term memory network LSTM and a deep neural network DNN; For each subtask, the subtask state features and shared basic features in the multi-task dataset are input into a long short-term memory network LSTM to obtain a state feature vector; Performing feature splicing processing using the state feature vector and the target strategy basic feature to obtain a feature splicing result, wherein the target strategy basic feature is an interval feature of the subtask data; The feature splicing result is input into the deep neural network DNN, and the subtask strategy parameters are updated based on the principle of maximizing the subtask strategy benefit until the multi-task strategy benefit loss function is minimized, thereby obtaining a second multi-task strategy reinforcement model.

5. A method for ranking recommended content according to claim 1, characterized in that: For each content to be sorted, the second multi-task strategy reinforcement model is used to evaluate the strategy value of the content to be sorted, and the content benefit value is obtained, including: For each to-be-sorted content, the loss weight of each subtask in the model is strengthened according to the second multi-task strategy to obtain the interval distribution probability corresponding to each subtask; Using the interval distribution probability corresponding to each subtask, the attribute corresponding to the content to be sorted is evaluated to obtain the attribute benefit value; The attribute benefit value is used to obtain the content benefit value of the content to be sorted.

6. A method for ranking recommended content as claimed in claim 5, characterized in that: Using the attribute benefit value to obtain the content benefit value of the content to be sorted, further comprising: A weighted summation process is performed on all attribute revenue values ​​of the content to be sorted, and a weighted weight value is set based on the business demand of the recommended product position to obtain the content revenue value of the content to be sorted.

7. A method for ranking recommended content according to any one of claims 1 to 6, characterized in that: The method is used for hotel recommendation ranking, and includes: The main strategy is decomposed based on the attributes of the hotels to be sorted, and at least one hotel subtask is decomposed. The attributes of the hotels to be sorted include price, star rating, brand, location code, POI type, distance and review quality. The main strategy is to order the hotel; An initial data set is constructed based on the user historical order sequence data of the hotels to be sorted, and the initial data set is decomposed based on each hotel subtask to obtain a multi-task data set; Using each hotel subtask to build a first multi-task strategy reinforcement model, based on the experience replay mechanism, using the multi-task data set to train the first multi-task strategy reinforcement model, to obtain a trained second multi-task strategy reinforcement model; For each hotel to be sorted, using the second multi-task strategy reinforcement model to evaluate the strategy value of the hotel to be sorted, and obtain the hotel recommendation revenue value; The recommended hotels are ranked according to the hotel recommendation revenue value of each hotel to be ranked.

8. A method for constructing a ranking model, characterized in that: The method comprises: Decomposing the main strategy based on the attributes of the data to be trained to obtain at least one subtask; According to the historical sequence data of the data to be trained, an initial data set is constructed, and data decomposition processing is performed on the initial data set based on each subtask to obtain a multi-task data set; A first multi-task strategy reinforcement model is constructed using each subtask, and based on an experience replay mechanism, the first multi-task strategy reinforcement model is trained using the multi-task data set to obtain a trained second multi-task strategy reinforcement model.

9. A device, characterized in that: The device comprises: a processor and a memory; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the steps of a recommended content ranking method as described in any one of claims 1 to 7 or a ranking model construction method as described in claim 8.

10. A computer program product, characterized in that The computer program product includes computer program instructions, which, when executed by a processor, implement the steps of a recommended content ranking method according to any one of claims 1 to 7 or a ranking model construction method according to claim 8.

Images