Strategy model training method and device, content sorting method and device and computer equipment

Abstract

The invention relates to a strategy model training method and device, a content sorting method and device and computer equipment. The method comprises the following steps: determining a plurality of sample user state representations based on user information obtained in a historical time period; determining a value label corresponding to each sample user state representation based on a plurality of recommended contents in a historical time period; respectively inputting the plurality of sample user state representations into a trained model, and determining sample responses of the sample user state representations for the plurality of recommended contents in the historical time period according to an output result of the trained model; and optimizing model parameters of the trained model according to the plurality of sample user state representations, the plurality of sample responses and the plurality of value labels to obtain a strategy model. By adopting the method, the recommendation efficiency and the recommendation effect can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of multi-objective sorting, and in particular to a training method for a strategy model, a content sorting method, an apparatus and a computer device. Background Art

[0002] With the development of Internet technology, various applications have emerged in an endless stream, bringing a lot of convenience to people's lives and work.

[0003] In some applications, the homepage will recommend content to users, which involves the problem of multi-objective sorting. At present, multi-objective sorting mainly determines the final score based on the estimated scores of multiple sub-objectives and the weight coefficients of each sub-objective, and then sorts according to the final score.

[0004] However, the weight coefficients in the above method need to be adjusted manually and regularly, otherwise the sorting will be unreasonable, affecting the recommendation effect. Summary of the invention

[0005] Based on this, it is necessary to provide a strategy model training method, content sorting method, device and computer equipment to address the above-mentioned technical problems. The training strategy model can quickly determine the target fusion coefficient and sorting results in practical applications, thereby improving the recommendation efficiency and effect.

[0006] In a first aspect, the present application provides a method for training a policy model, the method comprising:

[0007] Determine multiple sample user status representations based on user information obtained during historical periods;

[0008] Based on multiple recommended contents in a historical period, determine the value label corresponding to each sample user status representation;

[0009] Inputting a plurality of sample user status representations into the trained model respectively, and determining a sample response of each sample user status representation to a plurality of recommended contents in a historical period according to an output result of the trained model;

[0010] The model parameters of the trained model are optimized according to multiple sample user state representations, multiple sample responses and multiple value labels to obtain a policy model.

[0011] In some of the embodiments, the model parameters of the trained model are optimized according to multiple sample user state representations, multiple sample responses, and multiple value labels to obtain a policy model, including:

[0012] Multiple sample user state representations, multiple sample responses, and multiple value labels are substituted into the objective function for iterative calculation, and the optimized values ​​of the model parameters are determined according to the calculation results to obtain the strategy model; wherein, the objective function aims to maximize the average value of multiple users, and the objective function includes the value function of all fusion coefficients of the user under the model parameters.

[0013] In some of the embodiments, multiple sample user state representations, multiple sample responses, and multiple value tags are substituted into the objective function for iterative calculation, and the optimized values ​​of the model parameters are determined according to the calculation results, including:

[0014] Substitute multiple sample user state representations, multiple sample responses, multiple value labels and initial values ​​of model parameters into the objective function for calculation;

[0015] If the calculated average value of multiple users does not reach the maximum, the gradient ascent algorithm is used to adjust the value of the model parameter, and the next round of calculation is performed based on the adjusted value;

[0016] If the calculated average value of the plurality of users reaches the maximum, the value after the model parameter is adjusted is used to determine the optimized value of the model parameter.

[0017] In some embodiments, based on the ranking results of multiple recommended contents in a historical period, determining the value label corresponding to each sample user status representation includes:

[0018] Determine the actual interaction value based on the ranking of multiple recommended content in the historical period;

[0019] Determine the value label corresponding to each sample user status representation based on the actual interaction value and user retention characteristics.

[0020] In some of the embodiments, the user information includes at least one of user status information, content status information, historical interaction rate statistics, login information, and user type information.

[0021] In a third aspect, the present application also provides a content sorting method, the method comprising:

[0022] Determine the target user status representation according to the currently acquired user information;

[0023] For each recommended content to be recommended, determine the target fusion coefficient corresponding to each interactive target of the recommended content according to the target user state representation and the pre-trained strategy model;

[0024] According to the quantitative value of each interactive target of each recommended content and the target fusion coefficient, multiple recommended contents are sorted to obtain a target sorting result.

[0025] In some of the embodiments, determining the target fusion coefficient corresponding to each interactive target of the recommended content according to the target user state representation and the pre-trained strategy model includes:

[0026] The target user state representation is input into the policy model to obtain the probability distribution of the policy model output, and sampling is performed based on the probability distribution to obtain the target fusion coefficient corresponding to each interactive target.

[0027] In some embodiments, according to the quantified value of each interactive target of each recommended content and the target fusion coefficient, multiple recommended contents are sorted to obtain a target sorting result, including:

[0028] For each recommended content, a weighted sum calculation is performed based on the quantitative values ​​of multiple interactive targets and the target fusion coefficient to obtain the ranking score of each recommended content;

[0029] According to the ranking score of each recommended content, multiple recommended contents are ranked to obtain a target ranking result.

[0030] In a third aspect, the present application also provides a training device for a policy model, the device comprising:

[0031] A sample determination module, used to determine multiple sample user status representations based on user information acquired during a historical period;

[0032] A label determination module, used to determine the value label corresponding to each sample user status representation based on multiple recommended contents in a historical period;

[0033] A response determination module, used to input a plurality of sample user status representations into the trained model respectively, and determine a sample response of each sample user status representation to a plurality of recommended contents within a historical period according to an output result of the trained model;

[0034] The model optimization module is used to optimize the model parameters of the trained model according to multiple sample user state representations, multiple sample responses and multiple value labels to obtain a policy model.

[0035] In a fourth aspect, the present application further provides a content sorting device, the device comprising:

[0036] A state determination module, used to determine the state representation of the target user according to the currently acquired user information;

[0037] A coefficient determination module is used to determine the target fusion coefficient corresponding to each interactive target of the recommended content according to the target user state representation and the pre-trained strategy model for each recommended content currently to be recommended;

[0038] The sorting module is used to sort the multiple recommended contents according to the quantitative value of each interactive target of each recommended content and the target fusion coefficient to obtain the target sorting result.

[0039] In a fifth aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method as described in any one of the first aspect or the second aspect are implemented.

[0040] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method as described in any one of the first aspect or the second aspect are implemented.

[0041] In a seventh aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method as described in any one of the first aspect or the second aspect.

[0042] The training method, content sorting method, device and computer equipment of the above-mentioned strategy model determine multiple sample user status representations based on user information obtained in the historical period; determine the value label corresponding to each sample user status representation based on multiple recommended contents in the historical period; input the multiple sample user status representations into the trained model respectively, and determine the sample response of each sample user status representation to the multiple recommended contents in the historical period according to the output result of the trained model; optimize the model parameters of the trained model according to the multiple sample user status representations, the multiple sample responses and the multiple value labels to obtain the strategy model. In the technical scheme of the embodiment of the present application, the training strategy model can quickly determine the target fusion coefficient and the sorting result in practical applications, thereby improving the recommendation efficiency and recommendation effect, and the strategy model can be adjusted in time by using the user information obtained online for enhanced learning, thereby improving the accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A diagram of an application environment of a content sorting method in an embodiment;

[0045] Figure 2 A schematic diagram of a flow chart of a method for training a strategy model in an embodiment;

[0046] Figure 3 A schematic diagram of a flow chart of a step of determining an optimized value of a model parameter in one embodiment;

[0047] Figure 4 A flowchart of a step of determining a value tag in one embodiment;

[0048] Figure 5 A flowchart of a step of determining a sample user status representation in one embodiment;

[0049] Figure 6 A flowchart of a content sorting method in one embodiment;

[0050] Figure 7 A schematic diagram of a flow chart of steps for sorting a plurality of recommended contents in one embodiment;

[0051] Figure 8 is a structural block diagram of a training device for a strategy model in an embodiment;

[0052] Fig. 9 is a structural block diagram of a content sorting device in one embodiment;

[0053] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] First of all, before specifically introducing the technical solution of the embodiment of the present application, the technical background or technical evolution context on which the embodiment of the present application is based is introduced. First of all, with the development of Internet technology, various applications have emerged in an endless stream, bringing a lot of convenience to people's lives and work. In some applications, the homepage will recommend content to users, which involves the problem of multi-objective sorting. At present, multi-objective sorting mainly determines the final score based on the estimated scores of multiple sub-goals and the weight coefficients of each sub-goal, and then sorts according to the final score. However, the above method has the problem that the weight coefficient needs to be adjusted manually regularly, otherwise the sorting will be unreasonable, affecting the recommendation effect.

[0056] In response to the above problems, the embodiment of the present application provides a training method for a strategy model, which determines multiple sample user status representations based on user information obtained in a historical period; determines the value label corresponding to each sample user status representation based on multiple recommended contents in the historical period; inputs multiple sample user status representations into the trained model respectively, and determines the sample response of each sample user status representation to multiple recommended contents in the historical period according to the output result of the trained model; optimizes the model parameters of the trained model according to multiple sample user status representations, multiple sample responses and multiple value labels to obtain a strategy model. In the technical solution of the embodiment of the present application, the training strategy model can quickly determine the target fusion coefficient and the target ranking result in practical applications, thereby improving the recommendation efficiency and recommendation effect.

[0057] The content sorting method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 interacts with the user to obtain user information, and sends the user information to the server 104; the server 104 determines the target fusion coefficient based on the user information, and sorts the recommended content according to the target fusion coefficient to obtain the target sorting result; the terminal 102 displays multiple recommended contents according to the sorting result. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablet computers, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0058] In an exemplary embodiment, Figure 2 As shown, a training method for a policy model is provided, which is applied to Figure 1 Taking the server in as an example, the following steps may be included:

[0059] Step 201 : determining a plurality of sample user status representations based on user information acquired during a historical period.

[0060] The user information includes information related to the user's interaction with the terminal, such as whether the user has logged into the application, the user's interaction with different recommended content in the application, etc. It should be noted that the user information is not limited to the above examples, and may also include other information. The user status is represented by the amount of representation corresponding to the user information.

[0061] The server may process the user information of multiple users (user) in a historical period to obtain multiple sample user state representations (state). The processing method may include screening processing, filtering processing, feature extraction processing, encoding processing, etc.

[0062] For example, the server processes the user information of users u1, u2, ... un respectively to obtain sample user state representations S1, S2, ... Sn.

[0063] Step 202: Determine a value label corresponding to each sample user status representation based on a plurality of recommended contents in a historical period.

[0064] The value tag represents the value corresponding to the interaction goal between the user and the recommended content. The interaction goal includes at least one of like, favorite, forward, and comment.

[0065] During the historical period, users interact with recommended content, the server obtains the interaction target, and calculates the value of the interaction target to obtain the value label (reward) corresponding to each sample state representation.

[0066] For example, the interactions corresponding to the sample user state representations S1, S2...Sn are obtained, and then the value labels R1, R2...Rn corresponding to the sample user state representations S1, S2...Sn are calculated.

[0067] Step 203: Input a plurality of sample user state representations into the trained model respectively, and determine a sample response corresponding to each sample user state representation according to the output result of the trained model.

[0068] Among them, the sample response representation represents the predicted interaction target based on the sample user state representation.

[0069] After determining multiple sample user state representations, the multiple sample user state representations are respectively input into the trained model, and the trained model outputs the sample mean and sample variance; the sample Gaussian distribution is determined according to the sample mean and the sample variance, and the sample fusion coefficient corresponding to each interactive target can be obtained by randomly sampling the sample Gaussian distribution, and the sample response can be obtained by fusing the interactive targets according to the sample fusion coefficient.

[0070] For example, multiple sample user state representations S1, S2...Sn are respectively input into the trained model, and according to the output results of the trained model, the sample responses of users u1, u2...un can be determined to be A1, A2...An respectively.

[0071] Step 204 , optimizing the model parameters of the trained model according to the multiple sample user state representations, the multiple sample responses and the multiple value tags to obtain a policy model.

[0072] Loss calculation can be performed based on multiple sample user state representations, multiple sample responses, multiple value labels and a pre-set loss function, and the model parameters in the trained model are optimized based on the calculated loss values ​​to obtain the optimized values ​​of the model parameters, and the trained model using the optimized values ​​of the model parameters is determined as the strategy model.

[0073] It should be noted that the optimization method of the model parameters in the embodiment of the present application is not limited, and the method of loss calculation is not limited, and can be set according to actual conditions.

[0074] In the above embodiment, based on the user information obtained in the historical period, multiple sample user status representations are determined; based on the multiple recommended contents in the historical period, the value labels corresponding to each sample user status representation are determined; the multiple sample user status representations are respectively input into the trained model, and the sample responses of each sample user status representation to the multiple recommended contents in the historical period are determined according to the output results of the trained model; the model parameters of the trained model are optimized according to the multiple sample user status representations, the multiple sample responses and the multiple value labels to obtain the strategy model. In the technical solution of the embodiment of the present application, the training strategy model can quickly determine the target fusion coefficient and the target ranking result in practical applications, thereby improving the recommendation efficiency and recommendation effect; and, by using the user information obtained online for enhanced learning, the strategy model can be adjusted in time, thereby improving the accuracy of the model.

[0075] The above embodiment shows an online training method, and each sample must be sampled from the latest model. Therefore, during training, the training sample can only be used once, and the model cannot be updated in batches during large-scale training. In order to ensure the stability and convergence of model training, the embodiment of the present application optimizes the training method and converts the online training method into an offline training method.

[0076] The offline training method includes setting an objective function for calculating the loss according to the model parameters. The objective function aims to maximize the average value of multiple users, and the objective function includes the value function of all fusion coefficients of the user under the model parameters. The objective function can be shown as formula (1):

[0077] ----------------------------(1)

[0078] Among them, θ is the model parameter, u is the user, V(u,θ) is the value function of all fusion coefficients of the user under the model parameters, N is the number of users, and J(θ) is the average value of multiple users.

[0079] Based on the objective function, the above embodiment of "optimizing the model parameters of the trained model according to multiple sample user state representations, multiple sample responses and multiple value labels to obtain a strategy model" may include: substituting multiple sample user state representations, multiple sample responses, and multiple value labels into the objective function for iterative calculation, determining the optimized values ​​of the model parameters according to the calculation results, and obtaining the strategy model.

[0080] In each round of iterative calculation, multiple sample user state representations, multiple sample responses, and multiple value labels are substituted into the above objective function for calculation, and the value of the model parameter θ is adjusted according to the calculation result. The next round of calculation is performed according to the adjusted value of the model parameter until the average value of multiple users calculated according to the objective function reaches the maximum. The value of the model parameter θ at the end of the calculation is determined as the optimized value of the model parameter, and the trained model using the optimized value of the model parameter is determined as the policy model.

[0081] In the above embodiment, multiple sample user state representations, multiple sample responses, and multiple value tags are substituted into the objective function for iterative calculation, and the optimization values ​​of the model parameters are determined according to the calculation results to obtain the policy model. In the technical solution of the embodiment of the present application, the online training method is converted to the offline model training method, which can ensure the stability and convergence of the model training, and support the reuse of training samples and batch model update training.

[0082] In an exemplary embodiment, Figure 3 As shown, in the above embodiment, "substituting multiple sample user state representations, multiple sample responses, and multiple value tags into the objective function for iterative calculation, and determining the optimized values ​​of the model parameters according to the calculation results" may include the following steps:

[0083] Step 301, substitute multiple sample user state representations, multiple sample responses, multiple value labels and initial values ​​of model parameters into the objective function for calculation.

[0084] The model parameters are initialized to obtain the initial values ​​of the model parameters. Then, multiple sample user state representations, multiple sample responses, multiple value labels and the initial values ​​of the model parameters are substituted into the objective function to calculate and obtain the average value of multiple users.

[0085] Step 302: If the calculated average value of multiple users does not reach the maximum, the value of the model parameter is adjusted using a gradient ascent algorithm, and the next round of calculation is performed based on the adjusted value.

[0086] If the calculated average value of multiple users does not reach the maximum, the gradient ascent algorithm is used to adjust the value of the model parameters. Refer to formula (2):

[0087] -----------------------------------(2)

[0088] in, are the adjusted values ​​of the model parameters, is the adjusted value of the model parameters, α is the learning rate, u is the user, and N is the number of users.

[0089] Expanding formula (2) yields formula (3):

[0090] ----------------------(3)

[0091] Among them, ω is the fusion coefficient. Assuming ω is discrete, the probability is expressed as Represents. Transforming the right side of formula (3) yields formula (4):

[0092] ----------------(4)

[0093] in, Represents the actual value of the observed user under the fusion coefficient. Transforming the right side of formula (4) yields formula (5):

[0094] --------------(5)

[0095] in, is an expected form. Since the expectation is not computable, multiple Monte Carlo samplings are used to approximate the result, and formula (6) is obtained:

[0096] ------------------(6)

[0097] According to formula (6), the rise of model parameters can be calculated , and then the initial value of the model parameter and the rise amount are summed to obtain the adjusted value of the model parameter.

[0098] The objective function uses the adjusted values ​​of the model parameters, and then substitutes the sample user state representation, sample response, and value label into the objective function for the next round of iterative calculation.

[0099] Step 303: If the calculated average value of the plurality of users reaches the maximum, the adjusted value of the model parameter is used to determine the optimized value of the model parameter.

[0100] If the calculated average value of the plurality of users is the largest, the adjusted value of the model parameter is determined as the optimized value of the model parameter.

[0101] In the above embodiment, multiple sample user state representations, multiple sample responses, multiple value labels and initial values ​​of model parameters are substituted into the objective function for calculation; if the calculated average value of multiple users does not reach the maximum, the gradient ascent algorithm is used to adjust the value of the model parameter, and the next round of calculation is performed according to the adjusted value; if the calculated average value of multiple users reaches the maximum, the adjusted value of the model parameter is used to determine the optimized value of the model parameter. In the technical solution of the embodiment of the present application, the gradient ascent algorithm is used to adjust the model parameters, and the online training method can be converted to the offline training method, which not only ensures the stability and convergence of the model training, but also improves the efficiency of the model training.

[0102] In an exemplary embodiment, Figure 4 As shown, in the above embodiment, “determining the value label corresponding to each sample user status representation based on the ranking results of multiple recommended contents in the historical period” may include the following steps:

[0103] Step 401, determining the actual interaction value according to the ranking of multiple recommended contents in the historical period.

[0104] The terminal displays multiple recommended contents according to the sorting results in the historical period, and the user interacts with the recommended contents based on the display, such as viewing and collecting recommended content a, viewing and liking recommended content b, and not viewing recommended content c. The server can calculate the actual interaction value of the user under the sorting result based on the user's actual interaction.

[0105] Step 402: Determine the value label corresponding to each sample user status representation according to the actual interaction value and the user retention feature.

[0106] Among them, the user second retention feature indicates whether the new user is retained the next day. For example, if the new user still logs in to the application on the next day, the user second retention feature is retention; if the new user does not log in to the application on the next day, the user second retention feature is non-retention.

[0107] The server can combine the actual interaction value and the user's retention characteristics into a value label.

[0108] In some embodiments, benchmark data may be added to the value label. The benchmark data uses a regression network to learn the original value, thereby reducing the value variance and improving the training effect and stability.

[0109] In some embodiments, the value labels are normalized for learning. The original values ​​are generally positive numbers, that is, all strategies, whether good or bad, have positive return gradients, and after normalization, the model has negative return gradients and amplifies the differences between values, achieving the effect of accelerating model learning.

[0110] In the above embodiment, the actual interaction value is determined based on the ranking of multiple recommended contents in the historical period;

[0111] Determine the value label corresponding to each sample user status representation according to the actual interaction value and the user secondary retention feature. In the technical solution of the embodiment of the present application, adding the user secondary retention feature to the value label can improve the effect of the model in terms of retention; the combination of value labels not only highlights the core goal of the model, but also takes into account the interaction effect, and can neutralize fluctuations such as retention and duration.

[0112] In an exemplary embodiment, the user information includes at least one of user status information, content status information, historical interaction rate statistics information, login information, and user type information. Among them, the user status information represents the interaction status between the user and the terminal; the content status information includes the status of the recommended content; the historical interaction rate statistics information represents the ratio of interaction to the total amount in the historical period, such as the like rate; the login information represents whether the user has logged in to the application, and the user type information represents whether the user is a new user or an old user. Generally, a user is an old user after 7 days of registration.

[0113] like Figure 5 As shown, in the above embodiment, “determining the sample user status representation based on the user information obtained in the historical period” may include the following steps:

[0114] Step 501: input at least one of user status information and content status information into a pre-established behavior prediction model, and extract user features obtained by the behavior prediction model during the prediction process.

[0115] A behavior prediction model is established in advance. The server inputs at least one of the user status information and the content status information into the behavior prediction model, and the behavior prediction model makes predictions based on the above information and outputs the predicted interaction target. During the prediction process, the behavior prediction model encodes the user status information and the content status information, extracts the encoding results, and determines the encoding results as user features.

[0116] Step 502: Combine at least one of historical interaction rate statistics, login information, and user type information with user features to obtain a sample user state representation.

[0117] At least one of historical interaction rate statistics, login information, and user type information is concatenated with user features into a matrix of one row and multiple columns, and the matrix is ​​used as a sample user state representation.

[0118] In the above embodiment, at least one of the user status information and the content status information is input into a pre-established behavior prediction model to extract the user features obtained by the behavior prediction model during the prediction process; at least one of the historical interaction rate statistics, login information, and user type information is combined with the user features to obtain a sample user status representation. In the technical solution of the embodiment of the present application, user features are extracted from the behavior prediction model, which can theoretically maintain the density of the training data space while better maintaining the user distinction; the use of historical interaction rate statistics, login information, and user type information can more directly represent the features of different user attributes.

[0119] In an exemplary embodiment, Figure 6 As shown, a content sorting method is provided, which is applied to Figure 1 Taking the server in as an example, the following steps may be included:

[0120] Step 601: determine the target user status representation according to the currently acquired user information.

[0121] During the process of interacting with the user, the terminal obtains user information and sends the currently obtained user information to the server. The server receives the user information, performs feature extraction processing on the user information, or encodes the user information to obtain the target user state representation. The target user state representation can be a vector or a matrix.

[0122] In actual applications, the server can determine the target user status representation corresponding to the user based on at least one user information within a preset time period (such as n minutes); the server can also determine the target user status representation corresponding to each recommended content based on the user information corresponding to each recommended content.

[0123] Step 602 : for each recommended content to be recommended, determine the target fusion coefficient corresponding to each interactive target of the recommended content according to the target user state representation and the pre-trained strategy model.

[0124] The strategy model is pre-trained, and the training method can refer to the above embodiment.

[0125] For each recommended content currently to be displayed on the terminal, the server can input the target user status representation into the policy model to obtain the target fusion coefficient corresponding to each interactive target output by the policy model. For example, the server inputs the target user status representation into the policy model, and the policy model outputs the target fusion coefficient corresponding to the like as β1, the target fusion coefficient corresponding to the collection as β2, the target fusion coefficient corresponding to the forwarding as β3, and the target fusion coefficient corresponding to the comment as β4.

[0126] Step 603 , sorting the plurality of recommended contents according to the quantized value of each interactive target of each recommended content and the target fusion coefficient to obtain a target sorting result.

[0127] The server pre-sets the quantitative value of each interactive target. For each recommended content to be recommended, the ranking score of the recommended content can be determined according to the quantitative value of each interactive target and the target fusion coefficient; then, multiple recommended contents are sorted according to the ranking score to obtain the target ranking result.

[0128] After the server obtains the target ranking result, it can generate a recommendation page based on the target ranking result and multiple recommended contents, send the recommendation page to the terminal, and the terminal displays the recommendation page. Alternatively, the server directly sends multiple recommended contents and the target ranking result to the terminal; the terminal generates a recommendation page based on the multiple recommended contents and the target ranking result, and displays the recommendation page.

[0129] In the above embodiment, the target user state representation is determined based on the currently acquired user information; for each recommended content to be recommended, the target fusion coefficient corresponding to each interactive target of the recommended content is determined based on the target user state representation and the pre-trained strategy model; according to the quantified value of each interactive target of each recommended content and the target fusion coefficient, multiple recommended contents are sorted to obtain the target sorting result. In the technical solution of the embodiment of the present application, the target fusion coefficient is adaptively adjusted according to the user and the recommended content, so that the sorting of the recommended content can be made more reasonable and more suitable for different users, thereby improving the recommendation effect.

[0130] In an exemplary embodiment, the above-mentioned embodiment of "determining the target fusion coefficient corresponding to each interactive target of the recommended content according to the target user state representation and the pre-trained strategy model" may include the following steps: inputting the target user state representation into the strategy model, obtaining the probability distribution of the strategy model output, and sampling based on the probability distribution to obtain the target fusion coefficient corresponding to each interactive target.

[0131] Among them, the probability distribution represents the distribution of the fusion coefficients. The probability distribution can be expressed by the mean and variance.

[0132] The server inputs the target user state representation into the policy model to obtain the target mean and target variance output by the policy model. The Gaussian distribution is determined based on the target mean and target variance, and the target fusion coefficient corresponding to each interactive target is obtained by sampling in the Gaussian distribution.

[0133] For example, the target user state representation is input into the policy model, and the policy model outputs the target mean μ and target variance σ. The Gaussian distribution is determined based on the target mean μ and target variance σ, and random sampling is performed in the Gaussian distribution to obtain the target fusion coefficients for likes as β1, favorites as β2, forwards as β3, and comments as β4.

[0134] In the above embodiment, the target user state representation is input into the policy model to obtain the probability distribution of the policy model output, and sampling is performed based on the probability distribution to obtain the target fusion coefficient corresponding to each interactive target. In the technical solution of the embodiment of the present application, the probability distribution is determined by the policy model, so that the target fusion coefficient can be more consistent with the actual probability distribution, thereby improving the rationality of the sorting and further improving the recommendation effect.

[0135] In an exemplary embodiment, Figure 7 As shown, in the above embodiment, "sorting the plurality of recommended contents to obtain a target sorting result according to the quantified value of each interactive target of each recommended content and the target fusion coefficient" may include the following steps:

[0136] Step 701 : For each recommended content, a weighted sum calculation is performed according to the quantitative values ​​of multiple interactive objectives and the objective fusion coefficient to obtain a ranking score of each recommended content.

[0137] The quantitative value of each interactive target is pre-set. For example, the quantitative value corresponding to the like is δ1, the quantitative value corresponding to the favorite is δ2, the quantitative value corresponding to the forwarding is δ3, and the quantitative value corresponding to the comment is δ4. The ranking score of each recommended content calculated by weighted summation is F=δ1*β1+δ2*β2+δ3*β3+δ4*β4.

[0138] Step 702: sort the multiple recommended contents according to the ranking score of each recommended content to obtain a target ranking result.

[0139] Sort multiple recommended contents in descending order of ranking scores to obtain an initial ranking result; adjust the candidate ranking results according to the user's interaction time, ranking improvement coefficient, etc. to obtain a target ranking result.

[0140] It should be noted that the sorting method is not limited to the above examples and can be set according to actual conditions.

[0141] In the above embodiment, for each recommended content, a weighted sum is calculated based on the quantized values ​​of multiple interactive targets and the target fusion coefficient to obtain a ranking score for each recommended content; and based on the ranking score of each recommended content, multiple recommended contents are sorted to obtain a target ranking result. In the technical solution of the embodiment of the present application, the target fusion coefficient is adaptively adjusted based on the user and the recommended content, so personalized recommendations can be made for the user, thereby improving the recommendation effect.

[0142] In an exemplary embodiment, a content ranking method is provided, wherein the method is applied to Figure 1 Taking the server in as an example, the following steps may be included:

[0143] Step 1: Determine multiple sample user status representations based on user information acquired during a historical period; determine a value label corresponding to each sample user status representation based on multiple recommended contents during the historical period.

[0144] Step 2: Input multiple sample user state representations into the trained model respectively, and determine the sample response corresponding to each sample user state representation according to the output result of the trained model.

[0145] Step 3, substitute multiple sample user state representations, multiple sample responses, multiple value labels and initial values ​​of model parameters into the objective function for calculation.

[0146] Step 4: If the calculated average value of multiple users does not reach the maximum, the gradient ascent algorithm is used to adjust the value of the model parameter, and the next round of calculation is performed based on the adjusted value.

[0147] Step 5: If the calculated average value of multiple users reaches the maximum, the adjusted value of the model parameter is used to determine the optimized value of the model parameter to obtain the policy model.

[0148] Step 6: Determine the target user status representation according to the currently acquired user information.

[0149] Step 7: Input the target user state representation into the policy model to obtain the target mean and target variance output by the policy model.

[0150] Step 8: Determine the Gaussian distribution according to the target mean and target variance, and sample from the Gaussian distribution to obtain the target fusion coefficient corresponding to each interactive target.

[0151] Step 9: For each recommended content, a weighted sum calculation is performed based on the quantitative values ​​of multiple interactive targets and the target fusion coefficient to obtain a ranking score for each recommended content.

[0152] Step 10: sort the multiple recommended contents according to the ranking score of each recommended content to obtain a target ranking result.

[0153] Step 11: Obtain the actual interaction value corresponding to the target ranking result.

[0154] Step 12: Determine the value label corresponding to each sample user status representation according to the actual interaction value and the user retention feature. The value label can be used to retrain the strategy model.

[0155] In the above embodiment, the strategy model is trained and the online training mode is converted into an offline model training mode, which can ensure the stability and convergence of the model training, and support the reuse of training samples and the model update training in batches. In practical applications, the use of the strategy model can make the target fusion coefficient more consistent with the actual probability distribution, thereby improving the rationality of the sorting and further improving the recommendation effect. Furthermore, adding user retention features to the value label can improve the effect of the model in terms of retention; the combination of value labels not only highlights the core goal of the model, but also takes into account the interactive effect, which can neutralize fluctuations such as retention and duration.

[0156] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0157] Based on the same inventive concept, the embodiment of the present application also provides a content sorting device for implementing the content sorting method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more content sorting device embodiments provided below can refer to the limitations on the content sorting method above, and will not be repeated here.

[0158] In an exemplary embodiment, Figure 8 As shown, a training device for a strategy model is provided, comprising:

[0159] A sample determination module 801 is used to determine a plurality of sample user status representations based on user information acquired during a historical period;

[0160] A label determination module 802 is used to determine a value label corresponding to each sample user status representation based on multiple recommended contents in a historical period;

[0161] A response determination module 803, for inputting a plurality of sample user status representations into the trained model respectively, and determining a sample response of each sample user status representation to a plurality of recommended contents within a historical period according to an output result of the trained model;

[0162] The model optimization module 804 is used to optimize the model parameters of the trained model according to multiple sample user state representations, multiple sample responses and multiple value tags to obtain a policy model.

[0163] In some of the embodiments, the model optimization module 804 is specifically used to substitute multiple sample user state representations, multiple sample responses, and multiple value labels into the objective function for iterative calculation, determine the optimized values ​​of the model parameters according to the calculation results, and obtain the strategy model; wherein the objective function aims to maximize the average value of multiple users, and the objective function includes the value function of all fusion coefficients of the user under the model parameters.

[0164] In some of the embodiments, the model optimization module 804 is specifically used to substitute multiple sample user state representations, multiple sample responses, multiple value labels and initial values ​​of model parameters into the objective function for calculation; if the calculated average value of multiple users does not reach the maximum, the gradient ascent algorithm is used to adjust the value of the model parameter, and the next round of calculation is performed based on the adjusted value; if the calculated average value of multiple users reaches the maximum, the adjusted value of the model parameter is used to determine the optimized value of the model parameter.

[0165] In some embodiments, the device further comprises:

[0166] A value determination module, used to determine the actual interaction value based on the ranking of multiple recommended contents in a historical period;

[0167] The label determination module is used to determine the value label corresponding to each sample user status representation based on the actual interaction value and user retention characteristics.

[0168] In some of the embodiments, the user information includes at least one of user status information, content status information, historical interaction rate statistics, login information, and user type information.

[0169] In an exemplary embodiment, Fig. 9 As shown, a content sorting device is provided, the device comprising:

[0170] A state determination module 901 is used to determine a target user state representation according to currently acquired user information;

[0171] The coefficient determination module 902 is used to determine the target fusion coefficient corresponding to each interactive target of the recommended content according to the target user state representation and the pre-trained strategy model for each recommended content to be recommended currently;

[0172] The sorting module 903 is used to sort the multiple recommended contents according to the quantitative value of each interactive target of each recommended content and the target fusion coefficient to obtain a target sorting result.

[0173] In some of the embodiments, the coefficient determination module 902 is specifically used to input the target user state representation into the policy model, obtain the probability distribution of the policy model output, and obtain the target fusion coefficient corresponding to each interactive target by sampling based on the probability distribution.

[0174] In some of the embodiments, the sorting module 903 is used to perform weighted sum calculation on each recommended content based on the quantitative values ​​of multiple interactive targets and the target fusion coefficient to obtain a ranking score for each recommended content; and sort the multiple recommended contents based on the ranking score of each recommended content to obtain a target sorting result.

[0175] Each module in the above content sorting device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0176] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store XX data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a content sorting method is implemented.

[0177] Those skilled in the art will understand that Fig.10The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0178] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by a processor of an electronic device to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0179] In an exemplary embodiment, a computer program product is also provided, and when the computer program is executed by a processor, the above method can be implemented. The computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, part or all of the above method can be implemented in whole or in part according to the process or function described in the embodiment of the present application.

[0180] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0181] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0182] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0183] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for a strategy model, characterized in that: The method comprises: Determine multiple sample user status representations based on user information obtained during historical periods; Determining a value label corresponding to each of the sample user status representations based on the plurality of recommended contents within the historical period; Inputting the plurality of sample user status representations into the trained model respectively, and determining a sample response of each sample user status representation to the plurality of recommended contents within the historical period according to an output result of the trained model; The model parameters of the trained model are optimized according to the multiple sample user state representations, the multiple sample responses and the multiple value tags to obtain the policy model.

2. The method according to claim 1, characterized in that The step of optimizing the model parameters of the trained model according to the plurality of sample user state representations, the plurality of sample responses and the plurality of value tags to obtain the policy model comprises: Substitute multiple sample user state representations, multiple sample responses, and multiple value labels into the objective function for iterative calculation, determine the optimized values ​​of the model parameters according to the calculation results, and obtain the strategy model; wherein, the objective function aims to maximize the average value of multiple users, and the objective function includes the value function of all fusion coefficients of the user under the model parameters.

3. The method according to claim 2, characterized in that Substituting the plurality of sample user status representations, the plurality of sample responses, and the plurality of value tags into the objective function for iterative calculation, and determining the optimized values ​​of the model parameters according to the calculation results, comprises: Substituting the plurality of sample user state representations, the plurality of sample responses, the plurality of value tags and the initial values ​​of the model parameters into the objective function for calculation; If the calculated average value of the plurality of users does not reach the maximum, the value of the model parameter is adjusted using a gradient ascent algorithm, and the next round of calculation is performed according to the adjusted value; If the calculated average value of the plurality of users reaches the maximum, the adjusted value of the model parameter is used to determine the optimized value of the model parameter.

4. The method according to claim 3, characterized in that The determining, based on the ranking results of the plurality of recommended contents in the historical period, a value label corresponding to each of the sample user status representations comprises: Determine the actual interaction value based on the ranking of the plurality of recommended contents in the historical period; The value label corresponding to each of the sample user status representations is determined according to the actual interaction value and the user retention feature.

5. The method according to claim 1, characterized in that: The user information includes at least one of user status information, content status information, historical interaction rate information, login information and user type information.

6. A content sorting method, characterized in that: The method comprises: Determine the target user status representation according to the currently acquired user information; For each recommended content to be recommended currently, determine the target fusion coefficient corresponding to each interactive target of the recommended content according to the target user state representation and the pre-trained strategy model; According to the quantified value of each interactive target of each recommended content and the target fusion coefficient, the plurality of recommended contents are sorted to obtain a target sorting result.

7. The method according to claim 6, characterized in that The step of determining the target fusion coefficient corresponding to each interactive target of the recommended content according to the target user state representation and the pre-trained strategy model includes: The target user state representation is input into the strategy model to obtain the probability distribution of the strategy model output, and sampling is performed based on the probability distribution to obtain the target fusion coefficient corresponding to each of the interactive targets.

8. The method according to claim 6, characterized in that The step of sorting the plurality of recommended contents according to the quantified value of each interactive target and the target fusion coefficient of each recommended content to obtain a target sorting result includes: For each of the recommended contents, a weighted sum calculation is performed according to the quantized values ​​of the plurality of interactive objectives and the objective fusion coefficients to obtain a ranking score for each of the recommended contents; According to the ranking score of each of the recommended contents, the plurality of recommended contents are ranked to obtain the target ranking result.

9. A training device for a strategy model, characterized in that: The device comprises: A sample determination module, used to determine multiple sample user status representations based on user information acquired during a historical period; A label determination module, used to determine a value label corresponding to each of the sample user status representations based on a plurality of recommended contents in the historical period; a response determination module, configured to input the plurality of sample user status representations into the trained model respectively, and determine, according to the output result of the trained model, a sample response of each of the sample user status representations to the plurality of recommended contents within the historical period; A model optimization module is used to optimize the model parameters of the trained model according to the multiple sample user state representations, the multiple sample responses and the multiple value tags to obtain the policy model.

10. A content sorting device, characterized in that: The device comprises: A state determination module, used to determine the state representation of the target user according to the currently acquired user information; A coefficient determination module, for determining, for each recommended content currently to be recommended, a target fusion coefficient corresponding to each interactive target of the recommended content according to the target user state representation and a pre-trained strategy model; The sorting module is used to sort the plurality of recommended contents according to the quantified value of each interactive target and the target fusion coefficient of each recommended content to obtain a target sorting result.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images