Information sorting method, device, electronic device and readable storage medium

By sorting user characteristics and candidate information and using reinforcement learning model to output mixed bucket numbers for random hashing, the problem of small changes in information in the e-commerce recommendation system is solved, and the exposure rate and accuracy of information recommendation of new merchants are improved. , meets users' needs for diversity and novelty.

CN110766513BActive Publication Date: 2024-11-29BEIJING SANKUAI ONLINE TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN201910889074.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-09-19
Publication Date
2024-11-29
Estimated Expiration
2039-09-19

AI Technical Summary

Technical Problem

The existing e-commerce recommendation system is difficult to meet users' demand for diverse information in the takeaway industry, resulting in small changes in recommendation information and unable to meet users' demand for novelty.

Method used

By first sorting the user characteristics and candidate information, and using the reinforcement learning model to output the number of mixed buckets, the candidate information is randomly hashed into the bucket for the second sorting, improving the exposure rate of new information and the accuracy of recommendations.

Benefits of technology

It improves the exposure rate and accuracy of information recommendations of new merchants, takes into account the user's conversion rate and information novelty, and provides recommendation information that is more suitable for users.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110766513B_ABST
    Figure CN110766513B_ABST
Patent Text Reader

Abstract

The embodiments of the present disclosure provide an information sorting method, device, electronic device and readable storage medium, the method comprising: performing a first sorting on candidate information that meets user characteristics to obtain a first sorted candidate information set; inputting the user characteristics and candidate information characteristics into a preset model to output the number of mixed buckets through the preset model; the preset model is a reinforcement learning model trained according to the probability of repeated acquisition of information by the user; randomly hashing the candidate information in the first sorted candidate information set into buckets with the number of mixed buckets; performing a second sorting on the candidate information in the bucket to obtain a second sorted candidate information set; wherein the second sorted candidate information set is sorted differently from the first sorted candidate information set. The embodiments of the present disclosure can provide users with more suitable recommended information, thereby improving the accuracy of information recommendation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of network technology, and in particular, to an information sorting method, device, electronic device, and readable storage medium. Background Art

[0002] With the rapid development of informatization, the information provided to users by the Internet has exploded, and users' needs are also increasing. How to enable users to obtain the required information in a timely and accurate manner from the massive amount of information has become an urgent problem to be solved.

[0003] Currently, e-commerce recommendation systems usually estimate the probability of users clicking on or placing an order for a product based on the user's historical behavior and preferences, and then sort and display the products by probability.

[0004] However, for the emerging food delivery industry, since users may frequently purchase products from a certain merchant, sorting products based on users' historical behaviors and preferences results in little variation in the recommended products each time. For users who prefer diversity, the recommended product information is difficult to meet user needs. Summary of the invention

[0005] The embodiments of the present disclosure provide an information sorting method, device, electronic device and readable storage medium to better explore user preferences. Different sorting methods can be used for different users, so that the information recommended to different users can contain new information with different proportions, that is, more suitable recommended information can be provided to users, thereby improving the accuracy of information recommendations.

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided an information sorting method, the method comprising:

[0007] Performing a first sorting on the candidate information that meets the user characteristics to obtain a first sorted candidate information set;

[0008] Input the user features and candidate information features into a preset model to output the number of mixed buckets through the preset model; the preset model is a reinforcement learning model trained according to the probability of repeated acquisition of information by the user;

[0009] Randomly hashing the candidate information in the first sorted candidate information set into buckets whose number is the number of shuffle buckets;

[0010] Perform a second sort on the candidate information in the bucket to obtain a second sorted candidate information set; wherein the second sorted candidate information set is sorted differently from the first sorted candidate information set.

[0011] According to a second aspect of an embodiment of the present disclosure, there is provided an information sorting device, the device comprising:

[0012] A first sorting module, used to perform a first sorting on the candidate information that meets the user characteristics to obtain a first sorted candidate information set;

[0013] A bucket number prediction module is used to input the user characteristics and candidate information characteristics into a preset model to output the mixed bucket number through the preset model; the preset model is a reinforcement learning model trained according to the probability of repeated acquisition of information by the user;

[0014] A random hashing module, used for randomly hashing the candidate information in the first sorted candidate information set into buckets whose number is the same as the number of shuffled buckets;

[0015] The second sorting module is used to perform a second sorting on the candidate information in the bucket to obtain a second sorted candidate information set; wherein the sorting of the second sorted candidate information set is different from that of the first sorted candidate information set.

[0016] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0017] A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the aforementioned information sorting method is implemented when the processor executes the program.

[0018] According to a fourth aspect of an embodiment of the present disclosure, a readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the aforementioned information sorting method.

[0019] The embodiments of the present disclosure provide an information sorting method, device, electronic device, and readable storage medium, the method comprising:

[0020] Performing a first sorting on the candidate information that meets the user characteristics to obtain a first sorted candidate information set;

[0021] Input the user features and candidate information features into a preset model to output the number of mixed buckets through the preset model; the preset model is a reinforcement learning model trained according to the probability of repeated acquisition of information by the user;

[0022] Randomly hashing the candidate information in the first sorted candidate information set into buckets whose number is the number of shuffle buckets;

[0023] Perform a second sort on the candidate information in the bucket to obtain a second sorted candidate information set; wherein the second sorted candidate information set is sorted differently from the first sorted candidate information set.

[0024] The disclosed embodiment performs a first sort on the candidate information that meets the user characteristics, and after obtaining the first sorted candidate information set, the user characteristics and the candidate information characteristics are input into a preset model to output a mixed bucket number through the preset model, and the sorting results in the first sorted candidate information set are scattered using the mixed bucket number and a second sort is performed to obtain a second sorted candidate information set that is different from the first sorting method, so as to increase the probability of new information being ranked at the front, thereby increasing the exposure rate of the new information.

[0025] In addition, the reinforcement learning model is trained based on the probability of users repeatedly acquiring information, and can output the number of mixed buckets corresponding to different users. The number of mixed buckets can be used to control the degree of dispersion of the sorting results in the first sorting candidate information set. Different users can correspond to different degrees of dispersion, that is, different users have different sorting methods, and the proportion of new information in the recommended information is also different. Therefore, the embodiment of the present disclosure can provide users with more suitable recommended information, thereby improving the accuracy of information recommendation. At the same time, both the user conversion rate and the novelty of information can be taken into account. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the description of the embodiments of the present disclosure will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0027] Figure 1 A flowchart showing a method for sorting information in one embodiment of the present disclosure is shown;

[0028] Figure 2 A schematic diagram showing a flow chart of an information sorting method according to an embodiment of the present disclosure;

[0029] Figure 3 A schematic diagram of a reinforcement learning modeling architecture in one embodiment of the present disclosure is shown;

[0030] Figure 4 A structural diagram of an information sorting device in one embodiment of the present disclosure is shown;

[0031] Figure 5 A structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present disclosure.

[0033] Embodiment 1

[0034] Reference Figure 1 , which shows a flowchart of the steps of an information sorting method in an embodiment of the present disclosure, including:

[0035] Step 101: Perform a first sort on candidate information that meets user characteristics to obtain a first sorted candidate information set;

[0036] Step 102: input the user features and candidate information features into a preset model to output the number of mixed buckets through the preset model; the preset model is a reinforcement learning model trained according to the probability of repeated acquisition of information by the user;

[0037] Step 103: randomly hash the candidate information in the first sorted candidate information set into buckets whose number is the number of the mixed arrangement buckets;

[0038] Step 104: perform a second sort on the candidate information in the bucket to obtain a second sorted candidate information set; wherein the sorting of the second sorted candidate information set is different from that of the first sorted candidate information set.

[0039] The information sorting method disclosed in the present invention can be applied to terminals, and the terminals specifically include but are not limited to: smart phones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, car computers, desktop computers, set-top boxes, smart TVs, wearable devices, etc.

[0040] It should be noted that the information in the present disclosure includes any information that a user can obtain through a terminal, which may be product information, merchant information, dish information, news information, entertainment, etc. For ease of description, the present disclosure embodiment takes candidate information as a candidate merchant as an example for explanation, and the information sorting process in other application scenarios can be referenced to each other.

[0041] The core of this disclosure is to propose a new information sorting mechanism, which can improve the exposure rate of new merchants while ensuring the platform user conversion rate.

[0042] Specifically, the disclosed embodiment first performs a first sort on the candidate information that meets the user characteristics to obtain a first sorted candidate information set. In actual applications, when a user triggers an information recommendation request through a terminal, the recommendation server can receive the information recommendation request sent by the terminal, and call the recommendation service with the user identifier, the user's current location, and the query term as input recommendation parameters. The recommendation server mines the candidate information that meets the user characteristics based on the received recommendation parameters and recalls the candidate set. Afterwards, the recommendation server extracts features and calculates recommendation scores for the candidate information in the recalled candidate set through a preset recommendation model, and performs a first sort on the candidate information in the recalled candidate set in descending order of the recommendation scores to obtain a first sorted candidate information set.

[0043] The recommendation score is calculated by a preset recommendation model, which can be trained based on user characteristics, merchant characteristics, user-merchant interaction characteristics, and context characteristics. Generally, the recommendation score is positively correlated with the click-through rate of the corresponding candidate information.

[0044] Since the first ranking candidate information set is sorted according to the recommendation score, which conforms to the user's historical behavior and preferences, the information recommended to the user changes little each time, and it is difficult for new merchants to be exposed, resulting in a lack of novelty in the recommended information. To solve this problem, after obtaining the first ranking candidate information set, the embodiment of the present disclosure breaks up the sorting results in the first ranking candidate information set and performs a second sorting to obtain a second ranking candidate information set that is different from the first sorting method, so as to increase the probability of new merchants being ranked at the front, thereby increasing the exposure rate of new merchants.

[0045] In the disclosed embodiment, a reinforcement learning model can be obtained in advance based on the probability of repeated acquisition of information by the user, and the user characteristics and the candidate information characteristics are input into the reinforcement learning model to output the number of mixed buckets, and the sorting results in the first sorting candidate information set can be scattered by the mixed bucket number. Specifically, the candidate information in the first sorting candidate information set is randomly hashed into buckets with the number of mixed buckets.

[0046] In an application example of the present disclosure, it is assumed that the first ranked candidate information set obtained includes: {poi1: 0.92, poi2: 0.91, poi3: 0.89, poi4: 0.86, poi5: 0.85, poi6: 0.82, poi7: 0.81, poi8: 0.80, poi9: 0.80...}, wherein poi1: 0.92 means that the recommendation score of the candidate information poi1 is 0.92, ranking the highest, poi2: 0.91 means that the recommendation score of the candidate information poi2 is 0.91, ranking the second, and so on.

[0047] Assuming that the user features of the current user and the candidate information features are input into the reinforcement learning model, and the number of mixed buckets is n (n is a positive integer), the candidate information in the first sorted candidate information set is randomly hashed into n buckets, and the candidate information in the n buckets is sorted for the second time to obtain a second sorted candidate information set, so that the second sorted candidate information set is sorted differently from the first sorted candidate information set. For example, after the sorting results in the first sorted candidate information set are scattered and sorted for the second time, the second sorted candidate information set is obtained as {poi3: 0.89, poi5: 0.85, poi1: 0.92, poi6: 0.82, poi8: 0.80, poi7: 0.81, poi4: 0.86, poi2: 0.91, poi9: 0.80...}. It can be seen that the ranking result in the second ranking candidate information set is different from the ranking result in the first ranking candidate information set, so that the candidate information originally ranked at the back has the opportunity to be ranked at the front to increase the exposure rate of new merchants.

[0048] Among them, the user features input into the reinforcement learning model may include the user's gender, age, location, occupation, preferences, historical behavior and other features. The candidate information features input into the reinforcement learning model may include the number of candidate merchants in the first sorted candidate information set, the number of high-quality merchants among the candidate merchants, and the average click-through rate, average conversion rate, average order rate and other features of all candidate merchants in the first sorted candidate information set. It can be understood that the above-mentioned user features and candidate information features are only used as an application example of the present disclosure, and the present disclosure does not limit the specific content of the user features and candidate information features.

[0049] The reinforcement learning model is trained based on the probability of users repeatedly acquiring information, and the repeated acquisition in the probability of users repeatedly acquiring information may at least include the following behaviors: multiple clicks, multiple browsing, multiple purchases, etc. Taking the scenario where the candidate information is a candidate merchant as an example, the probability of users repeatedly acquiring information may refer to the repurchase rate of the merchant's products by the user.

[0050] The reinforcement learning model can output the number of mixed buckets corresponding to different users, and the number of mixed buckets can be used to control the degree of dispersion of the sorting results in the first sorting candidate information set. For example, for users with a higher repurchase rate, the probability of repeatedly purchasing the same merchant's goods is higher, so the number of mixed buckets for this user can be less, so that the degree of dispersion of the sorting results is lower, so as to ensure its user conversion rate. For users with a lower repurchase rate, they may prefer diversity, so the number of mixed buckets can be more, so that the degree of dispersion of the sorting results is higher, so as to increase the exposure rate of new merchants. Therefore, for different users, different degrees of dispersion can be corresponding, that is, different users correspond to different sorting methods, and the proportion of new information contained in the recommended information is also different. Therefore, the disclosed embodiment can provide users with more suitable recommended information, thereby improving the accuracy of information recommendation. At the same time, both the user's conversion rate and the novelty of the information can be taken into account.

[0051] In an optional embodiment of the present disclosure, performing a second sort on the candidate information in the bucket to obtain a second sorted candidate information set may specifically include:

[0052] Each time, the candidate information with the highest first ranking score is taken out from each bucket in turn, and arranged in the order of taking out to obtain a second ranking candidate information set.

[0053] For example, after randomly hashing the candidate information in the above-mentioned first sorted candidate information set into n buckets, the candidate information in each bucket can be sorted according to the first sorting score (such as the recommendation score) to obtain the sorted buckets, and then the candidate information is taken out from the 1st to the nth buckets in turn, one at a time, and arranged in the order of taking out to obtain the second sorted candidate information set.

[0054] Assume that n=3, after randomly hashing the candidate information in the above-mentioned first sorted candidate information set into three buckets, sort the candidate information in each bucket from high to low according to the recommendation score to obtain the sorted buckets. For the first time, take out the first candidate information from the three sorted buckets respectively, and rank the candidate information taken out from the first bucket as top1, the candidate information taken out from the second bucket as top2, and the candidate information taken out from the third bucket as top3. Take out the first candidate information from the three sorted buckets again for the second time, and this time rank the candidate information taken out from the first bucket as top4, the candidate information taken out from the second bucket as top5, and the candidate information taken out from the third bucket as top6, and so on, until the candidate information in all buckets is taken out and arranged to obtain the second sorted candidate information set. Refer to Figure 2 , showing a flow chart of an information sorting method according to an embodiment of the present disclosure.

[0055] In an optional embodiment of the present disclosure, after the step of obtaining the second ranking candidate information set, the method may further include:

[0056] Determine a recommendation result list according to the second sorting candidate information set.

[0057] After obtaining the second ranking candidate information set, a recommendation result list can be determined based on the second ranking candidate information set. For example, a recommendation result list is generated from the topk (k is a positive integer) candidate information in the second ranking candidate information set, and the recommendation result list is pushed to the user. The difference between the recommendation result list and the recommendation result list directly obtained based on the first ranking candidate information set is that the recommendation result list contains more newly exposed information, which can improve the novelty of the recommended information.

[0058] In an optional embodiment of the present disclosure, the preset model can be obtained by training through the following steps:

[0059] Step S11, acquiring offline sample data according to the historical candidate information set and the historical behavior data generated by the user on the historical candidate information set; the offline sample data includes: user features, historical candidate information set features, user behavior features generated on the historical candidate information set, and context features;

[0060] Step S12: Using the offline sample data as the state of the reinforcement learning model;

[0061] Step S13, using the initial preset model output predicted mixed bucket number as the action of the reinforcement learning model;

[0062] Step S14, determining a reward value obtained by taking the action in the state according to a preset reward function;

[0063] Step S15: According to the state, action, and reward value, the initial preset model is trained using a reinforcement learning algorithm to obtain a trained preset model.

[0064] In general, a reinforcement learning system can include an agent and an execution environment. The agent continuously learns and optimizes its strategy through interaction and feedback with the execution environment. Specifically, the agent observes and obtains the state of the execution environment, and determines the behavior or action to be taken according to the current state of the execution environment based on a certain strategy. Such behavior acts on the execution environment, changes the state of the execution environment, and generates feedback to the agent, which is also called a reward or reward score. The agent determines whether the previous behavior is correct and whether the strategy needs to be adjusted based on the reward score obtained, and then updates its strategy. By repeatedly observing the state, determining the behavior, and receiving feedback, the agent can continuously update its strategy. The ultimate goal is to be able to learn a strategy that maximizes the cumulative reward score obtained.

[0065] More specifically, the process of agent learning and training includes strategy evaluation and strategy improvement. In strategy evaluation, state value function V or state-action value function Q is often used to represent the expected cumulative reward. The state-action value function is also called Q function. For example, Q(S, A) represents the cumulative reward score after starting from state S, executing action A and then using the strategy.

[0066] For example, according to the Q-learning method, the Q function is iteratively calculated using the following formula:

[0067] Q(S, A)←(1-α)*Q(S, A)+α*[R+γ*maxQ(S′, a)] (1)

[0068] Among them, α is the learning rate and γ is the discount factor. In the case of training reinforcement learning models, the samples generally include<S,A,R,S’> A sequence of, where S represents the state of the environment, A represents the action performed in the S state, R represents the reward score obtained after performing action A, and S' represents the new state to which the environment migrates after performing action A. For the sake of simplicity and clarity, S is referred to as the first state, A is referred to as the first action, and S' is referred to as the second state. Therefore, the historical sample for offline training of the reinforcement learning model can be a sequence consisting of the first state, the first action, the reward score, and the second state. The second state is the state to which the environment migrates after applying the first action when the environment is in the first state. Among them, the first state S is a state that the environment has been in historically, and the first action A is the action taken in the historical sample when the environment is in the first state. The second action indicates what action the intelligent experience will take when facing the same state S for the reinforcement learning model to be trained currently.

[0069] It is understandable that the embodiments of the present disclosure do not limit the training method of the reinforcement learning model. The method of training the reinforcement learning model may include an on-policy method and an off-policy method. The difference between the two is that the strategy for generating samples and the target strategy for optimization in the on-policy method are the same, while the strategy for generating samples and the target strategy for optimization in the off-policy method are not the same.

[0070] Among them, the historical samples trained by the same strategy method are the same strategy samples, that is, the second action S' is the same as the first action S, which means that the reinforcement learning model to be trained will decide the same action A when facing the same state S as in history. The historical samples trained by the different strategy method are different strategy samples, that is, the second action S' is different from the first action S, which means that the reinforcement learning model to be trained will decide different actions A when facing the same state S as in history.

[0071] Specifically, the embodiment of the present disclosure first obtains offline sample data for training the reinforcement learning model based on the historical candidate information set and the historical behavior data generated by the user in response to the historical candidate information set. The offline sample data may include: user characteristics, historical candidate information set characteristics, behavior characteristics generated by the user in response to the historical candidate information set, and context characteristics.

[0072] In an optional embodiment of the present disclosure, the historical candidate information set features may specifically include: average click rate, average conversion rate, average order conversion rate, maximum click rate, maximum conversion rate, maximum order conversion rate and other features of the historical candidate information in the historical candidate information set;

[0073] The behavioral characteristics of the user on the historical candidate information set may specifically include: the number of clicks, the number of orders, and the probability of repeated acquisition generated by the user on the historical candidate information in the historical candidate information set;

[0074] The context features may specifically include: time, user distance, weather and other features.

[0075] The embodiment of the present disclosure takes the same-strategy sample training method as an example, takes the offline sample data as the state S of the reinforcement learning model, and takes the initial preset model output predicted mixed bucket number as the action A of the reinforcement learning model, and determines the reward value R obtained by taking the action under the state according to the preset reward function, and then can use the reinforcement learning algorithm to train the initial preset model according to the state, action, and reward value to obtain the trained preset model.

[0076] Reference Figure 3, which shows a schematic diagram of the reinforcement learning modeling architecture in an embodiment of the present disclosure. In an application example of the present disclosure, the Environment represents the current environment, which may specifically include the user user and the merchant set pois. The Agent represents the mixed arrangement system, which may specifically include the preset model DQN and the storage module Memory, and the Memory can be used to store the training sample data of the online DQN model. The state represents the state in the reinforcement learning model quadruple, which may specifically include user features and merchant set features, and the merchant set refers to the set of candidate merchants. The Action represents the action in the reinforcement learning model quadruple, which may specifically include outputting the mixed arrangement bucket number through the preset model DQN. The Reward represents the reward value in the reinforcement learning model quadruple, which is calculated by a preset reward function, and can take into account both the user's conversion rate and the novelty of the information.

[0077] In an optional embodiment of the present disclosure, determining the reward value obtained by taking the action in the state according to a preset reward function may specifically include:

[0078] According to the historical behavior data generated by the user for the historical candidate information set, the click data and / or order data generated by the user for the historical recommendation result list are obtained;

[0079] If the historical behavior data includes click data generated by the user on historical candidate information in the historical recommendation result list, the reward value is determined according to the number of newly exposed information in the historical recommendation result list and the number of information corresponding to the click data; or

[0080] If the historical behavior data includes click data and order data generated by users for historical candidate information in the historical recommendation result list, the reward value is determined according to the order price corresponding to the order data, the number of newly exposed information, and the number of information corresponding to the click data; or

[0081] If the historical behavior data does not contain click data and order data generated by the user for the historical candidate information in the historical recommendation result list, the reward value is determined according to the number of newly exposed information.

[0082] During the training process of the reinforcement learning model, after taking a certain action (number of mixed buckets n) according to the current state (features of the user and merchant set), the system calculates the corresponding reward value for this action based on the reward function. The reward value serves as a feedback on the quality of the current strategy of the reinforcement learning model.

[0083] The disclosed embodiment obtains the click data and / or order data generated by the user on the historical recommendation result list based on the historical behavior data generated by the user on the historical candidate information set, and determines the reward function based on whether the user generates click data and order data on the historical candidate information in the historical recommendation result list, and the number of newly exposed information in the historical recommendation result list. This can balance the user's conversion rate and the novelty of the information, while ensuring the conversion rate, improving the novelty, so that the user can see more high-quality new merchants, and at the same time, more new merchants can get the opportunity to be exposed. Specifically, the preset reward function of the disclosed embodiment can be expressed as:

[0084]

[0085] In the disclosed embodiment, the number of newly exposed businesses in this exposure is used as a quantitative indicator of novelty, that is, the more newly exposed businesses there are in this exposure, the higher the novelty of this exposure.

[0086] As shown in formula (2) above, when there is no order / click for this PV (page view), it means that the user repurchase rate is low, so the newly exposed merchants should be rewarded, and the reward value is Min (number of newly exposed merchants, 5), that is, the smaller value between the number of newly exposed merchants and 5 is used as the reward value.

[0087] When this PV only has clicks, it is necessary to reward the newly exposed merchants and the merchants clicked by users. The reward value is: Min (number of newly exposed merchants, 5) + Min (number of merchants clicked by users, 5).

[0088] When this PV results in an order, rewards will be given based on the price of the order, as well as the number of new exposures and clicked merchants. This reward is mainly reflected in the conversion rate, and the reward value is Price + Min (number of new exposure merchants, 5) + Min (number of user clicked merchants, 5).

[0089] In the case where there is no order / click in this PV or this PV only has clicks, although no order is made, the newly exposed merchants are rewarded, which reflects the novelty. In the case where an order is made in this PV, the behavior of making an order is rewarded, which reflects the user's conversion rate while also taking into account the novelty of the information.

[0090] In an optional embodiment of the present disclosure, after obtaining the sorted candidate information set, the method may further include:

[0091] Step S21: Pushing the recommendation result list to the user;

[0092] Step S22: Generate online sample data based on the recommendation result list and the online behavior data generated by the user in response to the recommendation result list;

[0093] Step S23: training the preset model according to the online sample data to optimize the model parameters of the preset model to obtain an optimized preset model.

[0094] After the preset model is trained, the preset model can be used online in the recommendation system to display to the user a recommendation result list determined according to the second ranking candidate information set, and the recommendation result list can balance the user's conversion rate and the novelty of the information. After the recommendation result list is pushed to the user, the embodiment of the present disclosure can also obtain the user's online behavior data for the recommendation result list. The online behavior data can specifically include: data on whether the user clicks on the recommended merchants in the recommendation result list, data on whether the user places an order, etc.

[0095] The embodiment of the present disclosure may use the online behavior data as online sample data to continue training the preset model to optimize the model parameters of the preset model and obtain an optimized preset model.

[0096] In summary, the disclosed embodiment performs a first sort on the candidate information that meets the user characteristics, and after obtaining the first sorted candidate information set, the user characteristics and the candidate information characteristics are input into a preset model to output a mixed bucket number through the preset model, and the sorting results in the first sorted candidate information set are scattered using the mixed bucket number and a second sort is performed to obtain a second sorted candidate information set that is different from the first sorting method, so as to increase the probability of new merchants being ranked at the front, thereby increasing the exposure rate of new merchants.

[0097] In addition, the reinforcement learning model is trained based on the probability of users repeatedly acquiring information, and can output the number of mixed buckets corresponding to different users. The number of mixed buckets can be used to control the degree of dispersion of the sorting results in the first sorting candidate information set. Different users can correspond to different degrees of dispersion, that is, different users have different sorting methods, and the proportion of new information in the recommended information is also different. Therefore, the embodiment of the present disclosure can provide users with more suitable recommended information, thereby improving the accuracy of information recommendation. At the same time, both the user conversion rate and the novelty of information can be taken into account.

[0098] Embodiment 3

[0099] Reference Figure 4 , which shows a structural diagram of an information sorting device in an embodiment of the present disclosure, as follows.

[0100] A first sorting module 401 is used to perform a first sorting on the candidate information that meets the user characteristics to obtain a first sorted candidate information set;

[0101] The bucket number prediction module 402 is used to input the user characteristics and candidate information characteristics into a preset model to output the mixed bucket number through the preset model; the preset model is a reinforcement learning model trained according to the probability of repeated acquisition of information by the user;

[0102] A random hashing module 403, configured to randomly hash the candidate information in the first sorted candidate information set into buckets whose number is the number of shuffled buckets;

[0103] The second sorting module 404 is used to perform a second sorting on the candidate information in the bucket to obtain a second sorted candidate information set; wherein the sorting of the second sorted candidate information set is different from that of the first sorted candidate information set.

[0104] Optionally, the second sorting module 404 is specifically configured to retrieve candidate information with the highest first sorting score from each bucket in turn each time, and arrange the information in the retrieved order to obtain a second sorted candidate information set.

[0105] Optionally, the device further comprises:

[0106] A result determination module is used to determine a recommendation result list according to the second sorting candidate information set.

[0107] Optionally, the device further comprises:

[0108] A result push module, used to push the recommendation result list to the user;

[0109] A data acquisition module, used to generate online sample data according to the recommendation result list and the online behavior data generated by the user in response to the recommendation result list;

[0110] The model optimization module is used to train the preset model according to the online sample data to optimize the model parameters of the preset model to obtain the optimized preset model.

[0111] Optionally, the device further includes: a model training module, used to train the preset model; the model training module includes:

[0112] The sample acquisition submodule is used to acquire offline sample data according to the historical candidate information set and the historical behavior data generated by the user on the historical candidate information set; the offline sample data includes: user features, historical candidate information set features, user behavior features generated on the historical candidate information set, and context features; the offline sample data is used as the state of the reinforcement learning model;

[0113] A first determination submodule is used to use the initial preset model output predicted mixed bucket number as an action of the reinforcement learning model;

[0114] A second determination submodule is used to determine a reward value obtained by taking the action in the state according to a preset reward function;

[0115] The model training submodule is used to train the initial preset model according to the state, action, and reward value using a reinforcement learning algorithm to obtain a trained preset model.

[0116] Optionally, the characteristics of the historical candidate information set include: average click rate, average conversion rate, average order conversion rate, maximum click rate, maximum conversion rate, and maximum order conversion rate of the historical candidate information in the historical candidate information set;

[0117] The user's behavior characteristics on the historical candidate information set include: the number of clicks, the number of orders, and the probability of repeated acquisition generated by the user on the historical candidate information in the historical candidate information set;

[0118] The context features include: time, user distance, and weather.

[0119] Optionally, the second determining submodule includes:

[0120] A data acquisition unit, configured to acquire click data and / or order data generated by the user on the historical recommendation result list according to the historical behavior data generated by the user on the historical candidate information set;

[0121] A first determining unit is configured to determine a reward value according to the number of newly exposed information in the historical recommendation result list and the number of information corresponding to the click data if the historical behavior data includes click data generated by the user on historical candidate information in the historical recommendation result list; or

[0122] A second determining unit is configured to determine a reward value according to an order price corresponding to the order data, the number of newly exposed information, and the number of information corresponding to the click data if the historical behavior data includes click data and order data generated by the user for historical candidate information in the historical recommendation result list; or

[0123] The third determining unit is used to determine the reward value according to the number of newly exposed information if there is no click data and order data generated by the user for the historical candidate information in the historical recommendation result list in the historical behavior data.

[0124] In summary, the embodiment of the present disclosure provides an information sorting device, the device comprising: a first sorting module 401, used to perform a first sorting on candidate information that meets user characteristics to obtain a first sorted candidate information set; a bucket number prediction module 402, used to input the user characteristics and candidate information characteristics into a preset model to output a mixed bucket number through the preset model; the preset model is a reinforcement learning model trained according to the probability of repeated acquisition of information by the user; a random hashing module 403, used to randomly hash the candidate information in the first sorted candidate information set into buckets with the number of mixed buckets; a second sorting module 404, used to perform a second sorting on the candidate information in the bucket to obtain a second sorted candidate information set; wherein the second sorted candidate information set is sorted differently from the first sorted candidate information set. The information sorting device of the embodiment of the present disclosure can increase the probability of new merchants being ranked at the front, thereby increasing the exposure rate of new merchants.

[0125] The embodiment of the present disclosure also provides an electronic device, see Figure 5 , including: a processor 501, a memory 502, and a computer program 5021 stored in the memory and executable on the processor, wherein the processor implements the information sorting method of the aforementioned embodiment when executing the program.

[0126] An embodiment of the present disclosure further provides a readable storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the information sorting method of the aforementioned embodiment.

[0127] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0128] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems may also be used together with the teachings based thereon. According to the above description, it is apparent that the structure required for constructing such systems is. In addition, the embodiments of the present disclosure are not directed to any particular programming language either. It should be understood that the contents of the embodiments of the present disclosure described herein may be realized using various programming languages, and the description of the specific languages ​​above is intended to disclose the best mode of implementation of the embodiments of the present disclosure.

[0129] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the embodiments of the present disclosure can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0130] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present disclosure, the various features of the embodiments of the present disclosure are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: that the claimed embodiments of the present disclosure require more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, the inventive aspects lie in less than all the features of the single embodiment disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself as a separate embodiment of the embodiments of the present disclosure.

[0131] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0132] The various component embodiments of the embodiments of the present disclosure may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the sorting device according to the embodiments of the present disclosure. The embodiments of the present disclosure may also be implemented as a device or apparatus program for executing part or all of the methods described herein. Such a program implementing an embodiment of the present disclosure may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0133] It should be noted that the above embodiments illustrate rather than limit the embodiments of the present disclosure, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The embodiments of the present disclosure may be implemented by means of hardware including several different elements and by means of appropriately programmed computers. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0134] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0135] The above description is only a preferred embodiment of the embodiments of the present disclosure and is not intended to limit the embodiments of the present disclosure. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the embodiments of the present disclosure should be included in the protection scope of the embodiments of the present disclosure.

[0136] The above is only a specific implementation of the embodiment of the present disclosure, but the protection scope of the embodiment of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed by the embodiment of the present disclosure, which should be included in the protection scope of the embodiment of the present disclosure. Therefore, the protection scope of the embodiment of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for sorting information, characterized in that: The method comprises: Performing a first sorting on the candidate information that meets the user characteristics to obtain a first sorted candidate information set; Input the user features and candidate information features into a preset model to output the number of mixed buckets through the preset model; the preset model is a reinforcement learning model trained according to the probability of repeated acquisition of information by the user; Randomly hashing the candidate information in the first sorted candidate information set into buckets whose number is the number of shuffle buckets; Performing a second sort on the candidate information in the bucket to obtain a second sorted candidate information set; wherein the second sorted candidate information set is sorted differently from the first sorted candidate information set; The performing a second sorting on the candidate information in the bucket to obtain a second sorted candidate information set includes: taking out the candidate information with the highest first sorting score from each bucket in turn each time, and arranging them in the order of taking out to obtain the second sorted candidate information set.

2. The method according to claim 1, characterized in that After the step of obtaining the second sorting candidate information set, the method further includes: Determine a recommendation result list according to the second sorting candidate information set.

3. The method according to claim 2, characterized in that After the step of determining the recommendation result list, the method further includes: Pushing the recommendation result list to the user; Generate online sample data according to the recommendation result list and the online behavior data generated by the user in response to the recommendation result list; The preset model is trained according to the online sample data to optimize the model parameters of the preset model to obtain an optimized preset model.

4. The method according to claim 1, characterized in that: The preset model is obtained by training through the following steps: Obtain offline sample data based on the historical candidate information set and the historical behavior data generated by the user on the historical candidate information set; the offline sample data includes: user features, historical candidate information set features, user behavior features generated on the historical candidate information set, and context features; the offline sample data is used as the state of the reinforcement learning model; The initial preset model output predicts the number of mixed buckets as the action of the reinforcement learning model; Determine, according to a preset reward function, a reward value obtained by taking the action in the state; According to the state, action, and reward value, the initial preset model is trained using a reinforcement learning algorithm to obtain a trained preset model.

5. The method according to claim 4, characterized in that The characteristics of the historical candidate information set include: the average click rate, average conversion rate, average order conversion rate, maximum click rate, maximum conversion rate, and maximum order conversion rate of the historical candidate information in the historical candidate information set; The user's behavior characteristics on the historical candidate information set include: the number of clicks, the number of orders, and the probability of repeated acquisition generated by the user on the historical candidate information in the historical candidate information set; The context features include: time, user distance, and weather.

6. The method according to claim 4, characterized in that Determining the reward value obtained by taking the action in the state according to a preset reward function includes: According to the historical behavior data generated by the user for the historical candidate information set, the click data and / or order data generated by the user for the historical recommendation result list are obtained; If the historical behavior data contains click data generated by users on historical candidate information in the historical recommendation result list, the reward value is determined based on the number of newly exposed information in the historical recommendation result list and the number of information corresponding to the click data; or if the historical behavior data contains click data and order data generated by users on historical candidate information in the historical recommendation result list, the reward value is determined based on the order price corresponding to the order data, the number of newly exposed information, and the number of information corresponding to the click data; or if the historical behavior data does not contain click data and order data generated by users on historical candidate information in the historical recommendation result list, the reward value is determined based on the number of newly exposed information.

7. An information sorting device, characterized in that: The device comprises: A first sorting module, used to perform a first sorting on the candidate information that meets the user characteristics to obtain a first sorted candidate information set; A bucket number prediction module is used to input the user characteristics and candidate information characteristics into a preset model to output the mixed bucket number through the preset model; the preset model is a reinforcement learning model trained according to the probability of repeated acquisition of information by the user; A random hashing module, used for randomly hashing the candidate information in the first sorted candidate information set into buckets whose number is the same as the number of shuffled buckets; A second sorting module is used to perform a second sorting on the candidate information in the bucket to obtain a second sorted candidate information set; wherein the second sorted candidate information set is sorted differently from the first sorted candidate information set; performing a second sorting on the candidate information in the bucket to obtain a second sorted candidate information set includes: Each time, the candidate information with the highest first ranking score is taken out from each bucket in turn, and arranged in the order of taking out to obtain a second ranking candidate information set.

8. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the information sorting method as described in one or more of claims 1-6 is implemented.

9. A readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the information sorting method as described in one or more of method claims 1-6.

Citation Information

Patent Citations

  • Information recommendation system and information recommendation method

    CN106327227A

  • Product recommendation method, device and electronic equipment

    CN107424043A