Sorting method, device and computer readable storage medium

Abstract

The present invention discloses a sorting method, device and computer-readable storage medium, which relate to the field of data processing. The sorting method includes: calculating a first sorting index for each candidate object according to user operation data and value index data of the candidate object; performing a first sorting on the candidate objects using the first sorting index; obtaining part of the candidate objects in the result of the first sorting as second sorting objects; and re-sorting the second sorting objects using a pre-trained sorting model so as to recommend the second sorting objects to the user in the order of the second sorting results. In an embodiment of the present invention, when a sorting model is used for the second sorting, the sorted objects are not all candidate objects, but part of the candidate objects, thereby reducing the calculation pressure of the sorting model and improving the recommendation efficiency. In addition, the number of objects browsed by a user in a short period of time is limited, so the solution of the present invention does not affect the user's experience.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a sorting method, a device and a computer-readable storage medium. Background Art

[0002] With the continuous development of Internet-based e-commerce technology, the number of products displayed on e-commerce websites has exploded. Timely and accurate identification of users' real-time shopping interests based on their behavioral characteristics can greatly optimize the user's shopping experience, while also bringing click-through advertising revenue to e-commerce websites and significantly increasing shopping transactions.

[0003] At present, the industrial-grade personalized recommendation system is mainly divided into two stages: recall and sorting. In the recall stage, products that users may be interested in can be recalled based on various features. In the sorting stage, the user's historical behavior can be combined into features and then input into a trained large-scale neural network. In this stage, the user's various behaviors will be combined and analyzed, each candidate product will be scored, and the products with high scores will be selected to be recommended to users first. Summary of the invention

[0004] After analyzing the relevant technologies, the inventors found that the sorting stage can sort the recalled products in the recall stage more accurately. However, due to the complexity of the model and the large number of recalled products, the current sorting process is also time-consuming, which reduces the efficiency of the recommendation.

[0005] A technical problem to be solved by the embodiments of the present invention is: how to improve recommendation efficiency.

[0006] According to a first aspect of some embodiments of the present invention, there is provided a sorting method, comprising: calculating a first sorting index for each candidate object based on user operation data and value index data of the candidate object; performing a first sorting on the candidate objects using the first sorting index; obtaining some candidate objects in the results of the first sorting as second sorting objects; and re-performing a second sorting on the second sorting objects using a pre-trained sorting model, so as to recommend the second sorting objects to the user in the order of the second sorting results.

[0007] In some embodiments, a preset number of objects with the highest first sorting index in the results of the first sorting are obtained as second sorting objects.

[0008] In some embodiments, calculating the first ranking index of each candidate object based on the user operation data and value index data of the candidate object includes: using a preset formula to calculate the first ranking index of each candidate object, wherein the independent variable of the preset formula is determined based on the user operation data and the value index data.

[0009] In some embodiments, the first ranking index is calculated using the following formula:

[0010]

[0011] Among them, R1 is the first ranking index of the alternative object, n is the number of clicks on the alternative object, m is the number of exposures of the alternative object, α is the adjustment coefficient, and 0≤α≤1, bid is the advertising bid value of the alternative object, and gmv is the transaction amount of the alternative object.

[0012] In some embodiments, the sorting method further includes: determining the number of second sorting objects according to a preset feedback time and a computing capability of the sorting model.

[0013] In some embodiments, the sorting method further includes: obtaining user operation data from at least one of a user click log and a user order log; and obtaining value index data from at least one of a user order log and an advertising bidding system.

[0014] In some embodiments, the user operation data includes one or more of the user's click data, browsing data, collection data, and purchase data for each candidate object; the value index data includes one or more of the object's advertising bid information and transaction information.

[0015] According to a second aspect of some embodiments of the present invention, there is provided a sorting device, comprising: a first sorting index calculation module, configured to calculate a first sorting index for each candidate object based on user operation data and value index data of the candidate object; a first sorting module, configured to perform a first sorting on the candidate objects using the first sorting index; a second sorting object acquisition module, configured to acquire some candidate objects in the result of the first sorting as second sorting objects; and a second sorting module, configured to re-perform a second sorting on the second sorting objects using a pre-trained sorting model, so as to recommend the second sorting objects to the user in the order of the second sorting results.

[0016] In some embodiments, the second sorting object acquisition module is further configured to acquire a preset number of objects with the highest first sorting index from the results of the first sorting as second sorting objects.

[0017] In some embodiments, the first ranking index calculation module is further configured to calculate the first ranking index of each candidate object using a preset formula, wherein the independent variable of the preset formula is determined based on user operation data and value index data.

[0018] In some embodiments, the first ranking index calculation module is further configured to calculate the first ranking index using the following formula:

[0019]

[0020] Among them, R1 is the first ranking index of the alternative object, n is the number of clicks on the alternative object, m is the number of exposures of the alternative object, α is the adjustment coefficient, and 0≤α≤1, bid is the advertising bid value of the alternative object, and gmv is the transaction amount of the alternative object.

[0021] In some embodiments, the sorting device further includes: a second sorting object quantity determination module configured to determine the quantity of second sorting objects according to a preset feedback time and a computing capability of the sorting model.

[0022] In some embodiments, the sorting device also includes: a data acquisition module, configured to acquire user operation data from at least one of a user click log and a user order log; and acquire value index data from at least one of a user order log and an advertising bidding system.

[0023] According to a third aspect of some embodiments of the present invention, there is provided a sorting device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the aforementioned sorting methods based on instructions stored in the memory.

[0024] According to a fourth aspect of some embodiments of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program implements any one of the aforementioned sorting methods when executed by a processor.

[0025] Some embodiments of the above invention have the following advantages or beneficial effects: when the embodiment of the present invention adopts the sorting model to perform the second sorting, the sorted objects are not all candidate objects, but a part of the candidate objects, thereby reducing the calculation pressure of the sorting model and improving the recommendation efficiency. In addition, the number of objects that the user browses in a short time is limited, so the solution of the present invention does not affect the user's experience.

[0026] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1 Schematic diagram of a flow chart of a sorting method according to some embodiments of the present invention.

[0029] Figure 2 Schematic diagram of a flow chart of a ranking model training method according to some embodiments of the present invention.

[0030] Figure 3 Schematic diagram of a flow chart of a recommended method according to some embodiments of the present invention.

[0031] Figure 4 Schematic diagram of the structure of a sorting device according to some embodiments of the present invention.

[0032] Figure 5 Schematic diagram of application scenarios of the sorting device according to some embodiments of the present invention.

[0033] Figure 6 Schematic diagram of the structure of a sorting device according to some other embodiments of the present invention.

[0034] Figure 7 Schematic diagram of the structure of a sorting device according to some other embodiments of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0037] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0038] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.

[0039] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0040] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0041] Figure 1 FIG. 1 is a flow chart of a sorting method according to some embodiments of the present invention. Figure 1 As shown, the sorting method of this embodiment includes steps S102 to S108.

[0042] In step S102, a first ranking index of each candidate object is calculated according to the user operation data and value index data of the candidate object.

[0043] The candidate objects can be selected according to the specific application scenario. For example, in the user's search scenario, the candidate objects can be objects with the search terms input by the user; in the promotion activity scenario, the candidate objects can be objects participating in the promotion activity; for the recommended position in the user's shopping cart page, the candidate objects can be objects related to the objects in the user's shopping cart; for the recommended position in the user's homepage, the candidate objects can be objects related to the objects that the user has recently browsed and purchased. The object in the present invention can refer to a commodity, or it can refer to other items used for recommendation, such as articles, advertisements, etc.

[0044] The user operation data reflects the user's interest in the candidate object, and the user operation data includes one or more of the user's click data, browsing data, collection data, and purchase data for each candidate object. These data can be quantities or ratios, such as exposure, number of clicks, click-through rate, etc. In some embodiments, the user operation data can be obtained from at least one of the user click log and the user order log.

[0045] In some embodiments, the user operation data may be the user's operation data on the recommended page or layout, for example, the user's operation data on the homepage, shopping cart page, and recommended page of the application or website. Since the user already has a clear intention to click or purchase on the search page, the role of the recommendation is relatively small. Therefore, by obtaining the user's operation data on the recommended page or layout, the accuracy of the recommendation can be further improved.

[0046] The value index data reflects the benefits that the candidate object can bring, which can be historical benefits or future benefits. The value index data can include, for example, one or more of the object's advertising bid information and transaction information. In some embodiments, the value index data can be obtained from at least one of a user order log and an advertising bidding system.

[0047] In step S104, a first sorting index is used to perform a first sorting on the candidate objects.

[0048] In step S106, some candidate objects in the result of the first sorting are obtained as second sorting objects.

[0049] Which candidate objects are specifically selected for the second sorting can be determined according to the specific application scenario. In some embodiments, a preset number of objects with the highest first sorting index in the first sorting result can be obtained as the second sorting objects.

[0050] In step S108, the pre-trained sorting model is used to re-sort the second sorting objects so as to recommend the second sorting objects to the user in the order of the second sorting results. For example, the data of the second sorting objects can be sent to the user's terminal for display.

[0051] The pre-trained ranking model may be, for example, a pre-trained neural network model. In some embodiments, the ranking may be performed based on the object features of the second ranking object and the user features of the user to be recommended. For example, the above features may be input into the ranking model to obtain an output value, and then the second ranking object may be ranked based on the size of the output value.

[0052] In some embodiments, the number of second sorting objects can be determined based on the preset feedback time and the computing power of the sorting model. For example, if the sorting system is required to respond in 400ms, the time taken to sort different numbers of objects in the second sorting can be pre-tested, and the number of objects with a time less than and closest to 400ms can be selected as the number of second sorting objects. In this way, the number of recommended objects can be increased as much as possible while satisfying the user experience.

[0053] By using the method of the above embodiment, when the sorting model is used for the second sorting, the sorted objects are not all candidate objects, but a part of the candidate objects, thereby reducing the calculation pressure of the sorting model and improving the recommendation efficiency. In addition, the number of objects that a user browses in a short time is limited, so the solution of the present invention does not affect the user's experience.

[0054] In some embodiments, the computational complexity of performing the first sorting is less than the computational complexity of performing the second sorting. That is, all candidate objects are first sorted by rough sorting to filter out the commodities to be recommended, and then the commodities to be recommended are finely sorted to make the recommendation results more accurate. For example, a preset formula may be used to calculate the first sorting index of each candidate object, wherein the independent variable of the preset formula is determined according to the user operation data and the value index data.

[0055] Since a formula is used to calculate for rough sorting, namely the first sorting, all candidate objects can be sorted with lower computational complexity, and some of the candidate objects can be sorted with higher computational complexity but more accurate as recommendation results, thereby improving recommendation efficiency.

[0056] A method for calculating the first ranking index is exemplarily introduced below with reference to formula (1).

[0057]

[0058] In formula (1), R1 is the first ranking index of the candidate object, n is the number of clicks on the candidate object, m is the number of exposures of the candidate object, α is the adjustment coefficient, and 0≤α≤1, bid is the advertising bid value of the candidate object, and gmv is the transaction amount of the candidate object. In some embodiments, n, m, and gmv can be values ​​within a preset period of time, for example, they can be the number of clicks, the number of exposures, and the transaction amount in the past month, respectively.

[0059] It reflects the click-through rate of the alternative objects. It reflects the click-through conversion rate of the alternative objects. Can be used to estimate advertising revenue, It can be used to estimate transaction revenue. α can be used to adjust the ratio of advertising revenue to transaction revenue.

[0060] In some embodiments, the sorting model can be pre-trained. Figure 2 An embodiment of the ranking model training method of the present invention is described.

[0061] Figure 2 FIG. 1 is a flow chart of a method for training a sorting model according to some embodiments of the present invention. Figure 2 As shown, the sorting model training method of this embodiment includes steps S202 to S204.

[0062] In step S202, training data is obtained, each piece of training data includes the features of an exposed object and the features of the user involved in the exposed object, and the label value of the training data is determined according to whether the user clicks on the object. For example, the object clicked by the user can be labeled as 1, and the object not clicked can be labeled as 0.

[0063] In step S204, the neural network model is trained using the training data to obtain a ranking model. The output of the ranking model is the probability of the user clicking on the object corresponding to the input data, so that the objects can be ranked according to the output value, so as to preferentially display the objects that the user is more likely to click.

[0064] In some embodiments, a user may generate multiple requests for the same page, and the present invention exemplarily provides two processing methods.

[0065] One implementation method is to re-execute the process in response to each user request. Figure 1 Method of the embodiment: After the user makes a new request, the user's operation after the last request becomes historical data, so the selection of candidate objects and the first ranking index will change, so that the whole process can be re-executed.

[0066] Another implementation method is to pre-divide the candidate objects into multiple batches, perform a second sort on the objects in each batch in response to each user request, and return the sort result. Figure 3 An embodiment of the sorting method of the present invention is described.

[0067] Figure 3 Schematic diagram of a recommended method according to some embodiments of the present invention. Figure 3 As shown, the recommended method of this embodiment includes steps S302 to S312.

[0068] In step S302, in response to a user's request, a first ranking index of each candidate object is calculated according to the user operation data and value index data of the candidate object.

[0069] In step S304, a first sorting index is used to perform a first sorting on the candidate objects to obtain a first sorting result.

[0070] In step S306, the results of the first sorting are divided into multiple batches in sequence.

[0071] In step S308, a pre-trained sorting model is used to re-sort the objects in the nth batch for a second time, where the initial value of n is 1.

[0072] In step S310, the second sorting objects are recommended to the user in the order of the second sorting results.

[0073] In step S312, in response to the user's request again, n+1 is set and the process returns to step S308.

[0074] Therefore, the candidate objects can be roughly sorted at one time, and then finely sorted in batches according to the user's request and the sorting results of the batch are fed back, thereby improving the recommendation efficiency.

[0075] Reference below Figure 4 An embodiment of the sorting device of the present invention is described.

[0076] Figure 4 Schematic diagram of the structure of a sorting device according to some embodiments of the present invention. Figure 4 As shown, the sorting device 40 of this embodiment includes: a first sorting index calculation module 410, configured to calculate a first sorting index for each candidate object based on user operation data and value index data of the candidate object; a first sorting module 420, configured to use the first sorting index to perform a first sort on the candidate objects; a second sorting object acquisition module 430, configured to obtain some candidate objects in the result of the first sorting as second sorting objects; a second sorting module 440, configured to use a pre-trained sorting model to re-perform a second sorting on the second sorting objects, so as to recommend the second sorting objects to the user in the order of the second sorting results.

[0077] In some embodiments, the second sorting object acquisition module 430 is further configured to acquire a preset number of objects with the highest first sorting index from the results of the first sorting as second sorting objects.

[0078] In some embodiments, the first ranking index calculation module 410 is further configured to calculate the first ranking index of each candidate object using a preset formula, wherein the independent variables of the preset formula are determined based on user operation data and value index data.

[0079] In some embodiments, the first ranking index calculation module 410 is further configured to calculate the first ranking index using the following formula:

[0080]

[0081] Among them, R1 is the first ranking index of the alternative object, n is the number of clicks on the alternative object, m is the number of exposures of the alternative object, α is the adjustment coefficient, and 0≤α≤1, bid is the advertising bid value of the alternative object, and gmv is the transaction amount of the alternative object.

[0082] In some embodiments, the sorting device 40 further includes: a second sorting object quantity determination module 450 configured to determine the quantity of the second sorting objects according to a preset feedback time and a computing capability of the sorting model.

[0083] In some embodiments, the sorting device 40 also includes: a data acquisition module 460, which is configured to acquire user operation data from at least one of a user click log and a user order log; and acquire value index data from at least one of a user order log and an advertising bidding system.

[0084] Reference below Figure 5 An application scenario embodiment of the sorting device of the present invention is described.

[0085] Figure 5 Schematic diagram of application scenarios of the sorting device according to some embodiments of the present invention. Figure 5As shown, the sorting device 50 obtains a set of advertising products from the advertising product recall system 51 as candidate products, and reads data from the user click log 52, the advertiser bidding system 53 and the user order log 54 to calculate the first sorting index. After sorting calculation, the sorting device 50 pushes the sorted products to the client 55 for display. The user behavior collected by the client 55 is fed back to the user click log 52 and the user order log 54 for storage.

[0086] Figure 6 Schematic diagram of the structure of a sorting device according to some other embodiments of the present invention. Figure 6 As shown, the sorting device 60 of this embodiment includes: a memory 610 and a processor 620 coupled to the memory 610 , and the processor 620 is configured to execute the sorting method in any one of the aforementioned embodiments based on instructions stored in the memory 610 .

[0087] The memory 610 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0088] Figure 7 Schematic diagram of the structure of a sorting device according to some other embodiments of the present invention. Figure 7 As shown, the sorting device 70 of this embodiment includes: a memory 710 and a processor 720, and may also include an input / output interface 730, a network interface 740, a storage interface 750, etc. These interfaces 730, 740, 750 and the memory 710 and the processor 720 may be connected, for example, via a bus 760. Among them, the input / output interface 730 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 740 provides a connection interface for various networked devices. The storage interface 750 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0089] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the program implements any of the aforementioned sorting methods when executed by a processor.

[0090] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A sorting method, comprising: Calculating a first ranking index of each candidate object according to user operation data and value index data of the candidate object, including: using a preset formula to calculate the first ranking index of each candidate object, wherein the independent variable of the preset formula is determined according to the user operation data and the value index data; Performing a first sorting of the candidate objects using a first sorting index; Acquire some candidate objects in the result of the first sorting as second sorting objects, wherein the number of the second sorting objects is determined according to a preset feedback time and a computing capability of a sorting model; A pre-trained sorting model is used to re-sort the second sorting object so as to recommend the second sorting object to the user in the order of the second sorting result, wherein the sorting model is a neural network model, the input of the sorting model includes the object characteristics of the second sorting object and the user characteristics of the user to be recommended, and the output value of the sorting model is used to sort the second sorting object.

2. The sorting method according to claim 1, wherein: A preset number of objects with the highest first sorting index in the first sorting result are obtained as second sorting objects.

3. The sorting method according to claim 1, wherein: The first ranking index is calculated using the following formula: Among them, R1 is the first ranking index of the alternative object, n is the number of clicks on the alternative object, m is the number of exposures of the alternative object, α is the adjustment coefficient, and 0≤α≤1, bid is the advertising bid value of the alternative object, and gmv is the transaction amount of the alternative object.

4. The sorting method according to claim 1, further comprising: The number of the second sorting objects is determined according to the preset feedback time and the computing capacity of the sorting model.

5. The sorting method according to any one of claims 1 to 4, further comprising: Acquire user operation data from at least one of a user click log and a user order log; The value index data is obtained from at least one of a user order log and an advertisement bidding system.

6. The sorting method according to claim 1, wherein: The user operation data includes one or more of the user's click data, browsing data, collection data, and purchase data for each candidate object; The value index data includes one or more of the object's advertisement bidding information and transaction information.

7. A sorting device, comprising: A first ranking index calculation module is configured to calculate a first ranking index of each candidate object according to user operation data and value index data of the candidate object, including: using a preset formula to calculate the first ranking index of each candidate object, wherein the independent variable of the preset formula is determined according to the user operation data and the value index data; A first sorting module is configured to perform a first sorting on the candidate objects using a first sorting index; A second sorting object acquisition module is configured to acquire some candidate objects from the result of the first sorting as second sorting objects, wherein the number of the second sorting objects is determined according to a preset feedback time and a computing capability of the sorting model; The second sorting module is configured to use a pre-trained sorting model to re-sort the second sorting object so as to recommend the second sorting object to the user in the order of the second sorting result, wherein the sorting model is a neural network model, the input of the sorting model includes object features of the second sorting object and user features of the user to be recommended, and the output value of the sorting model is used to sort the second sorting object.

8. A sorting device, comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the sorting method according to any one of claims 1 to 6 based on instructions stored in the memory.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the sorting method according to any one of claims 1 to 6 is implemented.

Images