Information sorting method, device, electronic device and storage medium

Abstract

The present disclosure provides an information sorting method, device, electronic device and storage medium, which relate to the field of information recommendation technology, and in particular to the field of information sorting technology. The specific implementation scheme is: obtaining information to be sorted and a first sorting value corresponding to the information to be sorted; adjusting the first sorting value of the information to be sorted based on the adjustment parameter corresponding to the reference recommended information to obtain the second sorting value of the information to be sorted, and the information to be sorted and the reference recommended information belong to the same category; determining the sorting position of each information to be sorted based on the second sorting value of each information to be sorted. In the embodiment of the present disclosure, on the basis of the first sorting value, the adjustment parameter of the reference recommended information of the same category by the user is further used to adjust the first sorting value, thereby realizing the adjustment of the first sorting value based on the information of the same category, thereby improving the accuracy of sorting.

Description

Technical Field

[0001] The present disclosure relates to the field of information recommendation technology, and in particular to the field of information ranking technology. Background Art

[0002] With the development of Internet technology, web pages, documents, audio and video content have become content that Internet users often consume.

[0003] In related technologies, information to be sorted is first filtered from a large amount of content, and then sorted based on preset rules before being recommended to users in the sorted order. The accuracy of the sorting of information will directly affect the recommendation effect. Summary of the Invention

[0004] The present disclosure provides an information sorting method, device, electronic device and storage medium.

[0005] According to a first aspect of the present disclosure, there is provided an information sorting method, comprising:

[0006] Obtaining the information to be sorted and the first sort value corresponding to the information to be sorted;

[0007] Adjusting the first ranking value of the information to be sorted based on the adjustment parameter corresponding to the reference recommendation information to obtain a second ranking value of the information to be sorted, where the information to be sorted and the reference recommendation information belong to the same category;

[0008] Based on the second ranking value of each piece of information to be sorted, a ranking position of each piece of information to be sorted is determined.

[0009] According to a second aspect of the present disclosure, there is provided an information sorting device, comprising:

[0010] An acquisition module, configured to acquire the information to be sorted and a first sort value corresponding to the information to be sorted;

[0011] an adjustment module, configured to adjust a first ranking value of the information to be sorted based on an adjustment parameter corresponding to the reference recommendation information to obtain a second ranking value of the information to be sorted, wherein the information to be sorted and the reference recommendation information belong to the same category;

[0012] The sorting module is used to determine the sorting position of each piece of information to be sorted based on the second sorting value of each piece of information to be sorted.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the first aspect mentioned above.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method of the aforementioned first aspect.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method of the aforementioned first aspect when executed by a processor.

[0019] The solution provided in this embodiment further adjusts the first ranking value based on the first ranking value in the disclosed embodiment by using the adjustment parameters of the user's reference recommendation information of the same category, thereby adjusting the first ranking value based on the same category information, thereby improving the accuracy of the ranking.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a flowchart of an information sorting method according to an embodiment of the present disclosure;

[0023] Figure 2 is another flowchart of an information sorting method according to another embodiment of the present disclosure;

[0024] Figure 3 is a schematic diagram of a framework of an information sorting method according to another embodiment of the present disclosure;

[0025] Figure 4 This is a schematic diagram of the structure of an information sorting device according to an embodiment of the present disclosure;

[0026] Figure 5 is another structural diagram of an information sorting device according to another embodiment of the present disclosure;

[0027] Figure 6 It is a block diagram of an electronic device used to implement the information sorting method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0029] The terms "first," "second," and "third," etc., in the embodiments and claims of the present disclosure are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, for example, including a series of steps or elements. A method, system, product, or apparatus is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, product, or apparatus.

[0030] The present disclosure provides an information sorting method that can be applied to electronic devices, including but not limited to fixed devices and / or mobile devices. For example, fixed devices include but are not limited to servers, which can be cloud servers or ordinary servers. Mobile devices include but are not limited to one or more of mobile phones and tablet computers.

[0031] It should be noted that the acquisition, storage and application of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0032] The first embodiment of the present disclosure provides an information sorting method, such as Figure 1 The flowchart of the method is shown, which includes:

[0033] Step S101: Obtain information to be sorted and a first sort value corresponding to the information to be sorted.

[0034] In the disclosed embodiments, the information to be sorted may be information recommended to the user based on the user's search request, or information proactively recommended to the user by the recommendation system in the absence of a search request. The recommendation system may filter out information that the user may be interested in based on CF (Collaborative Filtering). The CF algorithm includes a user-based CF algorithm (User-Base CF). The recommendation principle adopted by User-Base CF is to find a user group with the same or similar hobbies for a target user, and when recommending, recommend content that the user group is interested in to the target user.

[0035] The CF algorithm primarily considers features such as the likelihood of a piece of information being clicked and the estimated viewing time. The content recalled by the CF algorithm can be information to be ranked. This information is fed into a ranking model, such as a Deep Neural Network (DNN) model, to obtain an estimated score for the information. This score serves as the first ranking value, representing the user's level of interest.

[0036] Information can be managed by categories, such as information in the film and television category, sports category, etc. The first ranking value does not take into account the user's interest feedback on information of different categories. However, users may have different interests in different time periods. If the user's recent interest needs can be quickly captured and the recall results of each category can be adjusted to meet the user's interest consumption in a timely manner, the user experience and consumption time benefits will be improved. Therefore, in step S102 of the embodiment of the present disclosure, based on the adjustment parameters corresponding to the reference recommendation information, the first ranking value of the information to be sorted is adjusted to obtain the second ranking value of the information to be sorted, and the information to be sorted and the reference recommendation information belong to the same category.

[0037] Step S103: determining the sorting position of each piece of information to be sorted based on the second sorting value of each piece of information to be sorted.

[0038] From the above, it can be seen that in the embodiment of the present disclosure, based on the first ranking value, the first ranking value is further adjusted using the adjustment parameters of the user's reference recommendation information of the same category, thereby adjusting the first ranking value based on the same category information, thereby improving the accuracy of the ranking.

[0039] In the case where the information resources have categories, in step 102, adjusting the first ranking value of the information to be sorted based on the adjustment parameter corresponding to the reference recommendation information to obtain the second ranking value of the information to be sorted includes:

[0040] Based on the category to which the information to be sorted belongs, obtaining adjustment parameters corresponding to the reference recommended information from the adjustment parameters corresponding to the recommended information in each category, wherein the adjustment parameters corresponding to the recommended information in each category are determined based on user operation data corresponding to the recommended information in each category;

[0041] Based on the adjustment parameter, the first ranking value of the information to be sorted is adjusted to obtain the second ranking value of the information to be sorted.

[0042] It can be seen that in the embodiment of the present disclosure, the adjustment parameters of the user for the recommendation information of different categories can be determined from the category granularity, and the adjustment parameters are used to reflect the user's real feedback on the recommendation information of different categories. This can improve the accuracy of the adjustment of the first ranking value. In addition, the reference recommendation information in the embodiment of the present disclosure is the recommendation information that has been recommended to the user and is in the same category as the information to be sorted. The adjustment parameters corresponding to the reference recommendation information are generated based on the user's operation data on the recommendation information of the same category as the information to be sorted. The user operation data can reflect the user's real preferences, and the adjustment parameters determined thereby can reflect the user's real preferences. When the adjustment parameters of the reference recommendation information are used to adjust the first ranking value of the information to be sorted, it is achieved that the first ranking value is adjusted based on the user's real interests and hobbies in the information of the same category, thereby improving the accuracy of the adjustment.

[0043] In some embodiments, to improve ranking accuracy, the user operation data in the disclosed embodiments may utilize short-term data to explore users' short-term interests. For example, user operation data for each category of recommended information within a week may be collected. Based on this user operation data, users' short-term interests in recommended information of different categories can then be explored to generate adjustment parameters for each category of recommendation. This allows the adjustment of the first ranking value based on the adjustment parameters to better meet users' short-term individual needs.

[0044] In the embodiment of the present disclosure, the categories of information may include first-level categories and sub-level categories. Sub-level categories may have multiple levels, such as first-level sub-categories, second-level sub-categories, and so on. For example, film and television videos are first-level categories, and film and television videos can be divided into first-level sub-categories such as suspense dramas and comedy dramas. It should be noted that the level of sub-categories can be configured according to business needs, and the present disclosure does not impose any restrictions on this. Since the level of categories is deeper and the number of categories included is higher, when determining the adjustment parameters of the recommended information of each category, the amount of calculation and complexity will increase as the number of categories increases. Therefore, during implementation, you can determine which level of categories to use to determine the adjustment parameters according to your own needs.

[0045] A user's interest in a certain category is not constant; sometimes their interest in a category increases, and sometimes decreases. Therefore, the adjustment parameters corresponding to each category of recommended information may include an adjustment direction, which indicates whether the interest in the category is increased or decreased, and an adjustment strength, which indicates the degree of increase or decrease in interest. Thus, based on the adjustment parameters, adjusting the first ranking value of the information to be sorted to obtain the second ranking value of the information to be sorted may be implemented as at least one of the following:

[0046] In the case where the adjustment direction is increasing, the adjustment strength is used as the increase ratio of the first ranking value to increase the first ranking value to the second ranking value;

[0047] When the adjustment direction is decreasing, the adjustment strength is used as a reduction ratio of the first ranking value to reduce the first ranking value to the second ranking value.

[0048] For example, the primary categories of multimedia resources include movies and TV. Assume that the adjustment parameter for movie and TV recommendations is 0.2. A positive adjustment parameter indicates increased interest, and an adjustment strength of 0.2 represents an increase of 0.2. Assuming the first ranking value of movie and TV recommendation A is 10, the second ranking value of A = 10 * (1 + 0.2) = 12. This increases the ranking value of the recommended information.

[0049] For another example, let's assume the adjustment parameter for movie and TV recommendations is -0.1. A negative adjustment parameter indicates a decrease in interest, and the adjustment strength is 0.1, meaning the reduction is 0.1. Assuming the first ranking value of movie and TV recommendation A is 10, the second ranking value of A is 10*(1-0.1)=9. This reduces the ranking value of the recommendation.

[0050] In another embodiment, the second ranking value can also be obtained by directly accumulating the adjustment parameters based on the first ranking value. For example, the first-level categories of multimedia resources include society, film and television, etc. Among them, the adjustment parameters corresponding to the society and film and television categories are 0.2 and -0.1 respectively. Among them, a positive number indicates that the user's interest in the corresponding category has increased, and a negative number indicates that the user's interest in the corresponding category has decreased. Therefore, the sign of the = integer parameter can indicate the adjustment direction. The absolute value of the adjustment parameter represents the adjustment strength. For example, 0.2 means that the interest level increases by a strength of 0.2, and -0.1 means that the interest level decreases by a strength of 0.1. The first ranking value is accumulated by 0.2 to increase the ranking value of the information to be sorted, and the first ranking value is accumulated by -0.1 to reduce the ranking value of the information to be sorted. If the ranking value of the information to be sorted changes, its sorting position may also change.

[0051] Therefore, in the embodiment of the present disclosure, by increasing and decreasing the ratio, the adjustment of the first ranking value can be achieved with a smaller amount of calculation and smaller resource consumption, and the adjustment direction is consistent with the change in the user's interest in the same type of recommendation, so that the final second ranking value is more in line with the user's interests and preferences. Based on the adjustment direction, a two-way adjustment of the interest level is achieved. In the case of increasing the interest level, the degree of increase in the user's interest in different categories may be different, and the degree of increase in the interest level of different categories will be different based on the adjustment strength. Similarly, when reducing the interest level, the adjustment strength of different categories will also be different. Therefore, the adjustment parameters of each category can measure the difference in interest changes of different categories.

[0052] When a user's interest in recommended information of different categories changes at different times, the adjustment parameters can accurately describe the differences in the user's interests in different categories. When the first ranking value of the information to be sorted is adjusted using the adjustment parameters of the recommended information of the category to which the information to be sorted belongs, the interest differences at the category level are mined. Compared with mining the interest differences of each information resource, mining the interest differences at the category level is more convenient and can more systematically describe the user's interest in a certain type of content, thereby making the sorting results more in line with the user's interests and hobbies.

[0053] In some embodiments, the adjustment parameters for each category of recommendation information are determined based on the following method:

[0054] Obtain the initial recommendation ranking and user operation data of each category of recommendation information;

[0055] Determine the number of times each category of recommended information has been operated based on user operation data;

[0056] Based on the number of times each category of recommended information is operated, the operation ranking of each category of recommended information is obtained;

[0057] Based on the initial recommendation ranking and operation ranking, determine the adjustment parameters corresponding to each category of recommendation information;

[0058] The recommended information of each category includes reference recommended information that belongs to the same category as the information to be sorted.

[0059] The initial ranking of recommended information in each category reflects the recommendation system's estimated ranking. This can be understood as the system's estimated ideal order. User action data, such as clicks, favorites, likes, and shares, reflects actual user feedback on the initial ranking. Therefore, the ranking of recommended information in each category based on user action data represents the user's actual ranking of recommended information in each category.

[0060] In summary, in the embodiments of the present disclosure, by analyzing user operation data, the user's operation ranking of each category of recommended information is obtained, and then the adjustment parameters of each category of recommended information are determined based on the operation ranking and the initial recommendation ranking, thereby determining the user's interest in different categories of recommended information based on the user's real feedback. The adjustment parameters determined in this way can reflect the user's interest and help improve the accuracy of the ranking.

[0061] The ranking of multiple information used in a recommendation is known, but there may be accidental errors in mining the interest differences at the category level based on the real feedback of users for a recommendation. In order to improve the accuracy of the feedback results, the embodiment of the present disclosure is based on the information recommended multiple times within a period of time, and the initial recommendation ranking of the recommended information is statistically analyzed, and then the adjustment parameters for the recommended information of each category are mined. The ranking of each piece of recommended information is known each time it is recommended, but when it is recommended multiple times, the ranking of the recommended information is not easy to determine. For example, the recommended information arranged in the recommended order for the first recommendation includes information A, information B, and information C. The recommended information arranged in the recommended order for the second recommendation is information D and information E, so the order of information AE is difficult to determine. Therefore, the embodiment of the present disclosure mines the interests of users of different categories at the category level. The initial recommendation order is also statistically analyzed based on the category granularity. During implementation, the initial recommendation ranking of the recommended information of each category is obtained, including:

[0062] Based on the number of times each category of recommended information is recommended, a recommendation ranking of the recommended information is obtained.

[0063] For example, consider multiple recommendations over a period of time, each of which belongs to a specific category. For example, at the first-level category level, we can mine the adjustment parameters corresponding to each category's recommendations. Suppose, within a week, the first-level category "Social" displayed 10 recommendations, and the first-level category "Film" displayed 3 recommendations. The first-level category "Social" received 10 recommendations, and the first-level category "Film" received 3 recommendations. The resulting ranking order is "Social," then "Film." This yields the initial ranking of recommendations for different categories over a period of time.

[0064] When ranking recommendations of different categories based on the number of recommendations, the statistical method is simple and feasible, and the recommendation ranking can be determined at the category level, so as to facilitate mining user feedback results at the category level.

[0065] Similarly, in the embodiment of the present disclosure, the user's actual ranking of each category of recommended information can be obtained based on the user's operation data. During implementation, the number of times each category of recommended information has been operated is determined based on the operation data; then, based on the number of times each category of recommended information has been operated, the operation ranking of each category of recommended information (i.e., the actual ranking of the recommended information at the user level) is obtained. When there is a difference between the initial recommendation ranking and the operation ranking, it means that the initial recommendation ranking estimated by the recommendation system is not accurate enough. The feedback result of the recommendation information can be determined based on the difference between the initial recommendation ranking and the operation ranking of the recommended information. In this way, the adjustment parameters of each category of recommended information are obtained.

[0066] In summary, the disclosed embodiments first generate an operation ranking for each category of recommended information based on the user operation data for each category of recommended information. The operation ranking represents the user's actual ranking of each category of recommended information, while the initial recommendation ranking for each category of recommended information represents an estimated ranking. Therefore, the difference between the operation ranking and the initial recommendation ranking can reflect the estimated interest deviation. Therefore, the adjustment parameters derived from the initial recommendation ranking and the operation ranking can truly reflect the user's interests and hobbies, thereby helping to improve the accuracy of the ranking.

[0067] In order to reasonably utilize the difference between the initial recommendation ranking and the operation ranking, and to mine the feedback results of different categories, the present embodiment introduces the ranking idea of ​​LambdaMart to measure the adjustment parameters corresponding to the recommendation information of each category. The aforementioned adjustment parameters corresponding to the recommendation information of each category are determined based on the initial recommendation ranking and the operation ranking, including:

[0068] Based on the initial recommendation ranking, determine the first labeled data of the recommended information in each category;

[0069] Determining second labeled data for recommended information of each category based on the operation ranking;

[0070] Determining a first normalized discounted cumulative gain (NDCG) of an initial recommendation ranking based on the first labeled data and the second labeled data;

[0071] After swapping the ranking positions of the recommendation ranking of one category with the recommendation rankings of other categories in the initial recommendation ranking, the candidate recommendation rankings are obtained;

[0072] Based on each candidate recommendation ranking and the first labeled data, obtaining third labeled data for each candidate recommendation ranking;

[0073] Determine the second labeled data and each third labeled data, and determine the second normalized loss cumulative gain of each candidate recommendation ranking;

[0074] An adjustment parameter corresponding to a category of recommendation information is determined based on the first normalized trade-off cumulative gain and each second normalized trade-off cumulative gain.

[0075] In this disclosed embodiment, the use of Lambda gradients in LambdaMart is utilized to quantify the direction and strength of adjustment parameters. This embodiment uses the normalized cumulative loss gain metric to measure the quality of category rankings, and then calculates the lambda score for each category to determine the strength and direction of adjustment parameters for each category's recommendation information, thereby improving the accuracy of parameter determination.

[0076] To understand how to determine the adjustment parameters of each category (also called category) based on LambdaMart, the following is combined with Figure 2 To illustrate this:

[0077] Step S201: Obtain display records and click records of recommended information of each category within a specified time period.

[0078] Step S202: Based on the display records of the recommended information of each category, count the number of times the recommended information of each category is recommended.

[0079] Step S203 : obtaining an initial recommendation ranking of each category of recommended information based on the number of times each category of recommended information is recommended, and obtaining first labeling data of each category of recommended information based on the initial recommendation ranking.

[0080] For example, the initial recommendation ranking of the first-level categories is [Health and Wellness, Society, Military, Film and Television, Food, Current Affairs, Automobile, International, Funny, Sports, Culture, Social Current Affairs, Fashion]. When marking, mark according to the ratio of 30%, 55%, and 15%. Among them, 30% represents the top 30% of the head categories in the initial recommendation ranking, 55% represents the categories belonging to the middle part of 31%-85% in the recommendation ranking, and 15% represents the categories ranked in the bottom 15%. Since three levels are divided (i.e., 30%, 55%, and 15% ratios), there are three values ​​for the marked data. In the embodiment of the present disclosure, 0 can be used to represent the category of the 15% ratio part, 1 can be used to represent the category of the 55% ratio part, and 2 can be used to represent the category of the top 30% ratio part. The first marked data of the first-level category obtained according to this marking method is, for example:

[0081] PredictLabel:[2,2,2,2,1,1,1,1,1,1,1,0,0]

[0082] Step S204: based on the click records of the recommended information of each category, count the number of clicks on the recommended information of each category.

[0083] Step S205 : obtaining an operation ranking of each category of recommended information based on the number of clicks on each category of recommended information, and obtaining second annotation data of each category of recommended information based on the operation ranking.

[0084] Similar to the annotation method of the initial recommendation ranking, the operation ranking in the embodiment of the present disclosure is also annotated according to the ratio of 30%, 55%, and 15%. Assume that the second annotation data of each category of recommendation information obtained based on the operation ranking is:

[0085] TrueLabel:[2,2,1,2,2,1,1,1,0,0,1,1,1]

[0086] Among them, the same sort position in TrueLabel and PredictLabel represents the same category.

[0087] Step S206 , using the second labeled data (TrueLabel), calculate the Discounted Cumulative Gain (DCG).

[0088] in,

[0089] In formula (1), p represents the total number of categories, p i is the labeled data of category i in the true label (TrueLabel). For example, in the above TrueLabel, the labeled data of the category in the first sort position is 2, and the labeled data of the category in the last sort position is 1.

[0090] Step S207: Calculate an ideal discounted cumulative gain (IDCG) based on the first labeled data.

[0091] in,

[0092] In formula (2), REL represents the total number of categories, rel i is the labeled data of category i in the ideal labeled data (i.e., PredictLabel). For example, in the above PredictLabel, the labeled data of the category in the first sort position is 2, and the labeled data of the category in the last sort position is 0.

[0093] Step S208: Calculate a first NDCG based on the DCG and the IDCG.

[0094] in,

[0095] The NDCG in formula (3) is the first NDCG.

[0096] In step S209, swapping any two categories i and j will yield a candidate recommendation ranking. The labeled data for the swapped categories i and j in the candidate recommendation ranking remains unchanged, with only the corresponding categories being changed. This yields the third labeled data for the candidate recommendation ranking. The NDCG is then recalculated to yield the second NDCG.

[0097] When calculating the adjustment parameters for category i, category i is swapped with each other category j, and a new candidate recommendation ranking is obtained with each swap. For example, if the initial recommendation ranking is (A, B, C), then A and B are swapped once to obtain a candidate recommendation ranking (B, A, C). A and C are swapped once to obtain another candidate recommendation ranking (C, B, A). For each candidate recommendation ranking, a second NDCG can be calculated with the second labeled data of the operation ranking.

[0098] Step S210 , calculating the difference between the first NDCG and the second NDCG before and after the positions of categories i and j are swapped.

[0099] Among them, ΔNDCG i,j =|NDCG(j,i)-NDCG(i,j)| (4)

[0100] In formula (4), ΔNDCG i,j It represents the difference between the first NDCG and the second NDCG before and after the positions of categories i and j are swapped. NDCG(j,i) represents the NDCG when category j is ranked before category i. NDCG(i,j) represents the NDCG when category j is ranked after category i.

[0101] Step S211, calculate lambda of categories i and j.

[0102] Among them, λ ij =a*|ΔNDCG j,j | (5)

[0103] In formula (5), a is a constant, which can usually be taken as 0.5 and can be determined by empirical value or experimental value.

[0104] Step S212: Calculate the adjustment parameter of category i based on the lambda of categories i and j. The adjustment parameter of category i is calculated based on the following formula (6), thereby obtaining the adjustment direction and adjustment strength.

[0105]

[0106] In formula (6), λ i represents the adjustment parameter for category i, λ i When it is a positive value, it means improving the first ranking value of the information in category i, λ i When it is a negative value, it means lowering the first ranking value of the information in category i. The absolute value of the value indicates the adjustment strength. represents the λ obtained when class i is sorted before class j ij To accumulate, represents the λ obtained when class i is sorted after class j ijFor example, there are four categories in total, and the recommended order is category 1, category 2, category 3, and category 4. When i is category 2, category 2 can swap positions with category 1, category 3, and category 4. Therefore, there are the following sorting sequences:

[0107] Sorting sequence 1, [category 2, category 1, category 3, category 4], corresponds to λ21;

[0108] Sorting sequence 2, [category 1, category 3, category 2, category 4], corresponds to λ22;

[0109] The sorting sequence 3, [category 1, category 3, category 4, category 2], corresponds to λ23.

[0110] What is accumulated are λ21 and λ22. What is accumulated are λ22 and λ23, thus obtaining the adjustment parameter λ2 of category 2.

[0111] In the disclosed embodiment, the adjustment parameter is used as a weight for the first ranking value to adjust the first ranking value. For example, if the adjustment parameter is 0.2 and the first ranking value is b, then the adjusted second ranking value b' = b(1 + 0.2). As a result, the first ranking value is increased by 20%. Similarly, if the adjustment parameter is -0.2, the second ranking value b' = b(1 - 0.2), thereby reducing the first ranking value by 20%.

[0112] After obtaining the second ranking value of each piece of information to be sorted, sorting may be performed based on the second ranking value.

[0113] For example, Figure 3 As shown, it is a schematic diagram of a framework of the information ranking method provided by an embodiment of the present disclosure. In this method, in response to a user's request (request), the request may be, for example, to open an application, or a search request based on a search language. Then, the user's history records are obtained from the database (historyinfo), and the history records include display records and click records of recommended information. Based on the display records, the recommendation ranking (show_rank) of the recommended information is obtained, and based on the click records, the operation ranking (click_rank) of the recommended information is determined. Then, based on the lambdaMart idea, the recommendation ranking and operation ranking are processed to determine the adjustment parameters of each category. As shown in FIG. Figure 3As shown, assuming that the adjustment parameter for the health and wellness category is 0.02, the adjustment parameter for the film and television category is -0.034, the adjustment parameter for the food category is -0.1, and the adjustment parameter for the sports category is 0.12, these adjustment parameters are used as weights to adjust the estimated scores of the information to be sorted based on the request, obtain a recall list (Recall_list), and respond to the user's request based on the recall list. Thus, the embodiment of the present disclosure completes the adjustment of the recall results based on the user's historical recommendation information and realizes the reordering of the recall results.

[0114] In summary, the information sorting method provided by the embodiments of the present disclosure can solve the problem of over-expansion or under-expansion of CF recall results. When a user is recently interested in content in other categories, other categories of content can also be recommended to the user based on the adjustment parameters, thereby solving the problem of information cocoons.

[0115] In the disclosed embodiment, the information to be sorted includes at least one of short videos, music, and web articles to be sorted. Thus, when recommending short videos, music, and web articles to users, the user's interests and hobbies can be fully utilized to make personalized recommendations for the user.

[0116] According to an embodiment of the second aspect of the present disclosure, an information sorting device is also provided, such as Figure 4 As shown, including:

[0117] An acquisition module 401 is configured to acquire information to be sorted and a first sort value corresponding to the information to be sorted;

[0118] An adjustment module 402 is configured to adjust a first ranking value of the information to be sorted based on an adjustment parameter corresponding to the reference recommendation information to obtain a second ranking value of the information to be sorted, where the information to be sorted and the reference recommendation information belong to the same category;

[0119] The sorting module 403 is configured to determine a sorting position of each piece of information to be sorted based on the second sorting value of each piece of information to be sorted.

[0120] In some embodiments, Figure 4 On the basis of Figure 5 As shown, the adjustment module 402 includes:

[0121] An acquisition submodule 501 is configured to acquire, based on the category to which the information to be sorted belongs, adjustment parameters corresponding to the reference recommended information from the adjustment parameters corresponding to the recommended information of each category, wherein the adjustment parameters corresponding to the recommended information of each category are determined based on the user operation data corresponding to the recommended information of each category;

[0122] The adjustment submodule 502 is configured to adjust the first ranking value of the information to be sorted based on the adjustment parameter to obtain a second ranking value of the information to be sorted.

[0123] In some embodiments, where the adjustment parameter includes adjusting the direction and the force, the adjustment submodule 502 is configured to perform at least one of the following:

[0124] In the case where the adjustment direction is increasing, the adjustment strength is used as the increase ratio of the first ranking value to increase the first ranking value to the second ranking value;

[0125] When the adjustment direction is decreasing, the adjustment strength is used as a reduction ratio of the first ranking value to reduce the first ranking value to the second ranking value.

[0126] In some embodiments, as Figure 5 As shown, the information sorting device also includes:

[0127] The recommendation ranking module 406 is used to obtain the initial recommendation ranking of each category of recommendation information and user operation data;

[0128] The number determination module 407 is used to determine the number of times each category of recommended information has been operated based on the user operation data;

[0129] An operation ranking module 408 is used to obtain an operation ranking of each category of recommended information based on the number of times each category of recommended information is operated;

[0130] An adjustment parameter determination module 409 is configured to determine the adjustment parameters corresponding to each category of recommendation information based on the initial recommendation ranking and the operation ranking;

[0131] The recommended information of each category includes reference recommended information that belongs to the same category as the information to be sorted.

[0132] In some embodiments, the recommendation ranking module 406 is configured to obtain an initial recommendation ranking of each category of recommendation information based on the number of times each category of recommendation information is recommended.

[0133] In some embodiments, the adjustment parameter determination module 409 is configured to:

[0134] Based on the initial recommendation ranking, determine the first labeled data of the recommended information in each category;

[0135] Determining second labeled data for recommended information of each category based on the operation ranking;

[0136] Determining a first normalized discounted cumulative gain of an initial recommendation ranking based on the first labeled data and the second labeled data;

[0137] After swapping the ranking positions of the recommendation ranking of one category with the recommendation rankings of other categories in the initial recommendation ranking, the candidate recommendation rankings are obtained;

[0138] Based on each candidate recommendation ranking and the first labeled data, obtaining third labeled data for each candidate recommendation ranking;

[0139] Determine the second labeled data and each third labeled data, and determine the second normalized loss cumulative gain of each candidate recommendation ranking;

[0140] An adjustment parameter corresponding to a category of recommendation information is determined based on the first normalized trade-off cumulative gain and each second normalized trade-off cumulative gain.

[0141] In some embodiments, the information to be sorted includes at least one of short videos, music, and web articles to be sorted.

[0142] In the embodiment of the present disclosure, based on the first ranking value, the first ranking value is further adjusted using the adjustment parameters of the user's reference recommendation information of the same category, thereby adjusting the first ranking value based on the same category information, thereby improving the accuracy of the ranking.

[0143] According to another embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0144] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0145] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0146] Multiple components in the electronic device 600 are connected to the I / O interface 605, including an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0147] The computing unit 601 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the information sorting method described above. In some embodiments, the information sorting method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via the ROM 502 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the information sorting method can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the information sorting method by any other appropriate means (e.g., by means of firmware).

[0148] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0149] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0150] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0151] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0152] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0153] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0154] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0155] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for sorting information, comprising: Obtaining information to be sorted and a first sort value corresponding to the information to be sorted; adjusting the first ranking value of the information to be sorted based on the adjustment parameter corresponding to the reference recommendation information to obtain a second ranking value of the information to be sorted, wherein the information to be sorted and the reference recommendation information belong to the same category; Determining a sorting position of each piece of information to be sorted based on the second sorting value of each piece of information to be sorted; The method further includes determining an adjustment parameter corresponding to the reference recommendation information based on the following method: Obtaining an initial recommendation ranking and user operation data for each category of recommendation information; the initial recommendation ranking includes the ranking position of each category; Determining the number of times each category of recommended information is operated based on the user operation data; Based on the number of times the recommended information of each category is operated, obtaining an operation ranking of the recommended information of each category; Determining first labeled data based on the initial recommendation ranking; Based on the operation ranking, obtaining second labeled data of the recommended information of each category; Determining a first normalized discounted cumulative gain of the initial recommendation ranking based on the first labeled data and the second labeled data; For category i representing the reference recommendation information, swap the position of category i with other categories j in the initial recommendation ranking, and obtain a new candidate recommendation ranking each time the swap is performed, so as to obtain N candidate recommendation rankings; Determining the third labeled data of each of the N candidate recommendation rankings; Calculating the second labeled data and the third labeled data of the N candidate recommendation rankings to obtain N second normalized loss cumulative gains; Determining N weights based on differences between the first normalized discounted cumulative gain and N second normalized discounted cumulative gains; Based on the N weights, an adjustment parameter of the reference recommendation information corresponding to the category i is determined.

2. The method according to claim 1, wherein Adjusting the first ranking value of the information to be sorted based on the adjustment parameter corresponding to the reference recommendation information to obtain the second ranking value of the information to be sorted includes: Based on the category to which the information to be sorted belongs, obtaining the adjustment parameter corresponding to the reference recommendation information from the adjustment parameters corresponding to the recommendation information of each category, wherein the adjustment parameter corresponding to the recommendation information of each category is determined based on the user operation data corresponding to the recommendation information of each category; Based on the adjustment parameter, the first ranking value of the information to be sorted is adjusted to obtain a second ranking value of the information to be sorted.

3. The method according to claim 2, wherein: The adjustment parameters include an adjustment direction and an adjustment strength. Based on the adjustment parameters, the first ranking value of the information to be sorted is adjusted to obtain a second ranking value of the information to be sorted, including at least one of the following: In the case where the adjustment direction is increasing, the adjustment strength is used as the increase ratio of the first ranking value, so as to increase the first ranking value to the second ranking value; When the adjustment direction is decreasing, the adjustment force is used as a reduction ratio of the first ranking value to reduce the first ranking value to the second ranking value.

4. The method according to claim 1, wherein obtaining the initial recommendation ranking of each category of recommendation information comprises: The initial recommendation ranking of the recommendation information of each category is obtained based on the number of times the recommendation information of each category is recommended.

5. The method according to any one of claims 1 to 4, wherein The information to be sorted includes at least one of short videos, music and web articles to be sorted.

6. An information sorting device, comprising: An acquisition module, configured to acquire information to be sorted and a first sort value corresponding to the information to be sorted; an adjustment module, configured to adjust the first ranking value of the information to be sorted based on an adjustment parameter corresponding to the reference recommendation information to obtain a second ranking value of the information to be sorted, wherein the information to be sorted and the reference recommendation information belong to the same category; a sorting module, configured to determine a sorting position of each piece of information to be sorted based on a second sorting value of each piece of information to be sorted; Also includes: A recommendation ranking module is used to obtain the initial recommendation ranking of each category of recommendation information and user operation data; the initial recommendation ranking includes the ranking position of each category; A times determination module, configured to determine the times that the recommended information of each category has been operated based on the user operation data; An operation ranking module, configured to obtain an operation ranking of the recommended information of each category based on the number of times the recommended information of each category is operated; An adjustment parameter determination module, configured to determine first labeled data based on the initial recommendation ranking; Based on the operation ranking, obtaining second labeled data of the recommended information of each category; Determining a first normalized discounted cumulative gain of the initial recommendation ranking based on the first labeled data and the second labeled data; For category i representing the reference recommendation information, swap the position of category i with other categories j in the initial recommendation ranking, and obtain a new candidate recommendation ranking each time the swap is performed, so as to obtain N candidate recommendation rankings; Determining the third labeled data of each of the N candidate recommendation rankings; Calculating the second labeled data and the third labeled data of the N candidate recommendation rankings to obtain N second normalized loss cumulative gains; Determining N weights based on differences between the first normalized discounted cumulative gain and N second normalized discounted cumulative gains; Based on the N weights, an adjustment parameter of the reference recommendation information corresponding to the category i is determined.

7. The device according to claim 6, wherein The adjustment module includes: an acquisition submodule, configured to acquire, based on the category to which the information to be sorted belongs, an adjustment parameter corresponding to the reference recommendation information from the adjustment parameters corresponding to the recommendation information of each category, wherein the adjustment parameter corresponding to the recommendation information of each category is determined based on user operation data corresponding to the recommendation information of each category; The adjustment submodule is configured to adjust the first ranking value of the information to be sorted based on the adjustment parameter to obtain a second ranking value of the information to be sorted.

8. The device according to claim 7, wherein, The adjustment parameters include adjustment direction and adjustment strength, and the adjustment submodule is configured to perform at least one of the following: In the case where the adjustment direction is increasing, the adjustment strength is used as the increase ratio of the first ranking value, so as to increase the first ranking value to the second ranking value; When the adjustment direction is decreasing, the adjustment force is used as a reduction ratio of the first ranking value to reduce the first ranking value to the second ranking value. 9 . The device according to claim 6 , wherein the recommendation ranking module is configured to obtain the initial recommendation ranking of the recommendation information of each category based on the number of times the recommendation information of each category is recommended.

10. The device according to any one of claims 6 to 9, wherein The information to be sorted includes at least one of short videos, music and web articles to be sorted.

11. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.

13. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 5.

Images