Training sample processing method, computing device and storage medium for sorting model

Abstract

The present invention discloses a training sample processing method, computing device and storage medium for a ranking model. The method includes: deleting conflicting samples from a plurality of first positive samples and a plurality of first negative samples to obtain a main page sample library; calculating the display volume ratio of each content type on the home page during the historical delivery process; performing downsampling processing on the plurality of second positive samples based on the display volume ratio to obtain a predetermined number of second positive samples as a sub-page sample library; merging and removing duplicates from the main page sample library and the sub-page sample library to obtain a training sample library for the ranking model.

Description

Technical Field

[0001] The present invention relates to the field of the Internet, and in particular to a training sample processing method, a computing device and a storage medium for a sorting model. Background Art

[0002] With the rapid development of information technology and Internet technology, more and more users obtain online information through various computing devices (such as phone, pad, PC, etc.). In addition, various information resources on the Internet are also growing explosively. People are gradually moving from an era of information scarcity to an era of information overload.

[0003] In this era, both information consumers and information producers have encountered great challenges: as an information consumer, it is very difficult to find the information of interest from a large amount of information; as an information producer, it is also very difficult to make the information they produce stand out and attract the attention of the majority of users.

[0004] In view of this, recommendation systems, especially personalized recommendations, came into being. It can recommend information and products that users are interested in based on their interests and behaviors. In the process of finding personalized content for users from massive data, conventional recommendation systems can be roughly divided into several steps: recall, sorting, supplementary strategies and rearrangement (such as strategies to increase the diversity, real-time, popularity and freshness of sorting results). Among them, the recall and sorting stages will use sorting models generated by machine learning massive data to help screen content that is more suitable for users. This means that better data and sorting models can help recommendation systems bring better user experience and more traffic to network information platforms, thereby generating more revenue.

[0005] In the industry of intelligent recommendation sorting algorithms, there is a popular saying that data determines the upper limit of the recommendation model. This is also the truth gained through the practice of tens of millions of companies in tens of millions of scenarios. In layman's terms, think of the model as a child who is about to receive education, and the massive data is the content to be taught to him, so better data means better education and better results. However, the data of the recommendation system is inseparable from the practical scenario, that is to say, we do not have a universal education method, but we must teach students in accordance with their aptitude, which is also the difficulty of constructing data samples in different recommendation scenarios. This is divided into the feature construction of data samples and the distribution adjustment of data samples. The technology described in this invention is a method for adjusting and optimizing the distribution of training samples of recommendation sorting models in the sorting stage.

[0006] For the adjustment and optimization of training samples, the existing technology usually uses the SMOTE sampling method to upsample the positive samples. Because in the recommendation scenario, the number of positive samples, that is, the content clicked by the user, is very different from that of negative samples, that is, the content with visible exposure but not clicked by the user, which will cause data imbalance, so different sampling methods are generally used to narrow the difference in the number of positive and negative samples and thus optimize the learning effect of the model. This method generates new reasonable positive samples according to the following formula:

[0007] λ*X1+(1-λ)*X2

[0008] Among them, X1 and X2 are two randomly selected positive samples, and λ is a random number that satisfies the Beta distribution.

[0009] Although this can increase the proportion of positive samples while improving the generalization ability as much as possible, it also has some disadvantages:

[0010] First, in the recommendation system of multiple content types (long articles, short articles, videos, travel notes, etc.), since the positive samples are randomly selected, the corresponding features of different types of content will be mixed in a sense, which is obviously unreasonable.

[0011] Second, the content type distribution and category (positive and negative samples) distribution of the training samples will differ from the distribution in the actual production environment. In a multi-type content recommendation system, this will cause positive samples of certain types with large data volumes to be overfitted, resulting in too much content of this type in the model's ranking results, reducing the richness of the ranking or the ranking effects of other types of content. Summary of the invention

[0012] In view of the above problems, the present invention is proposed to provide a training sample processing method, a computing device and a storage medium for a ranking model that overcome the above problems or at least partially solve the above problems.

[0013] According to one aspect of the present invention, a training sample processing method for a ranking model is provided, which is executed in a computing device, the computing device is in communication connection with a data storage device, the data storage device stores a plurality of first positive samples, a plurality of first negative samples and a plurality of second positive samples, the first positive samples are training samples corresponding to content displayed on the homepage and clicked by the user, the first negative samples are training samples corresponding to content displayed on the homepage but not clicked by the user, and the second positive samples are training samples corresponding to content displayed on a subpage and clicked by the user, the method comprising:

[0014] Deleting conflicting samples from the plurality of first positive samples and the plurality of first negative samples to obtain a main page sample library;

[0015] Calculate the proportion of content displayed on the homepage by each content type during the historical delivery process;

[0016] Performing downsampling processing on the plurality of second positive samples based on the display volume ratio to obtain a predetermined number of second positive samples as a sub-page sample library;

[0017] The main page sample library and the sub-page sample library are combined and deduplicated to obtain a training sample library for the ranking model.

[0018] Optionally, in the training sample processing method according to the present invention, the conflicting samples include: a first negative sample having the same content features and user features as any first positive sample, and / or a first negative sample having the same content features and user features as any second positive sample.

[0019] Optionally, according to the training sample processing method of the present invention, before merging and deduplicating the main page sample library and the sub-page sample library, the method further includes: performing oversampling processing on the first negative sample in the main page sample library.

[0020] Optionally, in the training sample processing method according to the present invention, the oversampling processing of the first negative sample in the main page sample library includes: determining a plurality of difficult-to-distinguish negative samples from the first negative sample included in the main page sample library; and oversampling the plurality of difficult-to-distinguish negative samples.

[0021] Optionally, in the training sample processing method according to the present invention, the hard-to-distinguish negative samples are negative samples whose content features match user features.

[0022] Optionally, in the training sample processing method according to the present invention, the oversampling adopts SMOTE sampling.

[0023] Optionally, in the training sample processing method according to the present invention, the predetermined number is a predetermined ratio of the number of positive samples in the main page sample library.

[0024] Optionally, in the training sample processing method according to the present invention, the content type includes at least one of the following: long text, short text, video and travel notes.

[0025] According to another aspect of the present invention, there is provided a computing device, comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be suitable for being executed by the at least one processor, and the program instructions include instructions for executing the above method.

[0026] According to another aspect of the present invention, a readable storage medium storing program instructions is provided. When the program instructions are read and executed by a computing device, the computing device executes the above method.

[0027] According to the sample processing scheme for the recommendation system ranking model of the embodiment of the present invention, it is possible to reasonably sample and fuse the user's samples on different sub-pages according to the visible exposure ratio (display ratio) of each type of content on the recommendation system homepage in the production environment, and solve the problem of conflicting sample distribution between the sub-page and the homepage, so as to improve the ranking effect and thus improve the overall CTR of the recommendation.

[0028] Furthermore, the embodiment of the present invention constructs hard-to-distinguish negative samples based on the characteristic characteristics of the samples, and adopts the method of oversampling the hard-to-distinguish negative samples to increase the recognition ability of the sorting model for the hard-to-distinguish samples.

[0029] Furthermore, the embodiment of the present invention samples the newly added positive and negative samples to ensure that the category proportions of the training samples are similar to the homepage production environment.

[0030] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0032] Figure 1 A schematic diagram of a content recommendation system 100 according to an embodiment of the present invention is shown;

[0033] Figure 2 A schematic diagram of a computing device 200 according to an embodiment of the present invention is shown;

[0034] Figure 3 A flowchart of a method 300 for processing training samples of a ranking model according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0035] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0036] First, some technical terms in the embodiments of the present invention are explained as follows.

[0037] Recommender System: There are many definitions of recommender system, but the most widely accepted concept and definition of recommender system is given by Resnick and Varian in 1997: "It uses e-commerce websites to provide customers with product information and suggestions to help users decide what products to buy, simulating sales staff to help customers complete the purchase process." Recommender system is a tool that automatically connects users and items (also known as content, resources or materials, etc.).

[0038] Personalized recommendation: recommending content that users are interested in based on their interests and behaviors.

[0039] CTR (Click-Through-Rate): Click-through rate is a commonly used term on the Internet, which refers to the click-through rate of online advertisements (or articles), that is, the actual number of clicks on the advertisement (or article) divided by the number of advertisement impressions.

[0040] Visible exposure: the content is displayed to users.

[0041] Content type viewable impression ratio (impression ratio): The ratio of viewable impressions (impressions) of each content type to the viewable impressions (impressions) of all content types.

[0042] Home page: Recommendation system main page.

[0043] The entire site: includes the recommendation system homepage and various sub-pages.

[0044] Recall: Mainly based on some of the user's characteristics, quickly retrieve a small number of items that the user is potentially interested in from a massive item library, and then hand them over to the sorting stage.

[0045] Sorting: Based on the recall results, more features are incorporated and complex models are used to make accurate personalized recommendations. This is a process of selecting the best from the best.

[0046] Data imbalance: The ratio of positive and negative samples input to the model is quite different.

[0047] Over-sampling: In the training of the ranking model of the recommendation system, it refers to the sampling method of increasing the proportion of certain samples in the total samples through different methods.

[0048] Under-sampling: In the training of the ranking model of the recommendation system, it refers to the sampling method of reducing the proportion of certain samples in the total samples through different methods.

[0049] SMOTE sampling: select a specific number of samples from the neighbors of minority class samples and mix them to generate new samples of this class.

[0050] Hard-negative: refers to negative samples whose model inference results are very close to positive sample results.

[0051] Expert sample: refers to a sample that is constructed by incorporating many business insights and can represent certain parts of the sample.

[0052] Overfitting: refers to the model overfitting the characteristics of the training samples and having weak generalization ability.

[0053] Underfitting: refers to the model not having enough training samples to learn its characteristics.

[0054] Beta distribution: refers to a set of continuous probability distributions defined in the interval [0,1], with two positive parameters, called shape parameters, generally represented by α and β. It can be regarded as a probability distribution, which can be understood as a probability distribution of probability.

[0055] Figure 1 FIG. 1 is a schematic diagram of a content recommendation system 100 according to an embodiment of the present invention. Figure 1 As shown, the content recommendation system 100 includes a plurality of user terminals 110 (only one is shown in the figure), a data storage device 120 , a server 130 and a computing device 200 .

[0056] The user terminal 110 is a terminal device used by the user, which can be a personal computer such as a desktop computer or a laptop computer, or a mobile phone, a tablet computer, a multimedia device, a smart wearable device, etc., but not limited thereto. A browser or a web application (webapp) resides in the user terminal 110, and the server 130 on the Internet is accessed through the browser or the web application. In the following, the browser and the web application are collectively referred to as the client, and the server 130 is the server. The server 130 is used to provide services to the user terminal 110, and it can be implemented as an application server, a Web server, etc., for example; it can also be implemented as a desktop computer, a laptop computer, a processor chip, a tablet computer, etc., but not limited thereto.

[0057] According to one embodiment, the server 130 can provide content services to the user terminal 110. When the user opens the client in the user terminal 110, the user terminal 110 will send a network request to the server 130. The server 130 will usually return one or more recommended content to the client, and the client will display the recommended content returned by the server 130 on the home page. This function is usually called homepage recommendation. Among them, one or more recommended content is automatically generated by the server 130. The process of automatically generating recommended content by the server 130 can be roughly divided into two stages, the recall stage and the sorting stage. In the recall stage, a small candidate set (hundreds to thousands) of user interests is selected from the resource pool (content set) based on the user's interests and historical behaviors. The sorting stage performs more accurate calculations on this basis, calculates the click probability for each content, and further selects a small amount of high-quality content (for example, a dozen contents) that the user is most interested in from the candidate set based on the click probability.

[0058] The data storage device 120 may be a relational database such as MySQL, ACCESS, etc., or a non-relational database such as NoSQL, etc.; it may be a local database residing in the server 130, or it may be a distributed database such as HBase, etc., set up in multiple geographical locations. In short, the data storage device 120 is used to store data, and the present invention does not limit the specific deployment and configuration of the data storage device 120. The server 130 may be connected to the data storage device 120 and obtain the data stored in the data storage device 120. For example, the server 130 may directly read the data in the data storage device 120 (when the data storage device 120 is a local database of the computing device 200), or may access the Internet by wired or wireless means, and obtain the data in the data storage device 120 through a data interface.

[0059] In an embodiment of the present invention, the data storage device 120 is suitable for storing content information, including: a resource pool (a content collection consisting of a large amount of content), click information and display information of the content during the delivery process (the number of clicks and displays during the current delivery process, as well as the click information and display information during the historical delivery process). The data storage device 120 is also suitable for storing various strategies and models used in the recall stage and the sorting stage, such as a sorting model.

[0060] Typically, the recommendation process of a content recommendation system is as follows:

[0061] The user terminal 110 sends a network request to the server 130, and the server 130 processes the network request and obtains multiple contents from the content set as a candidate set according to a predetermined recall strategy; in the sorting stage, for each content in the candidate set, the user characteristics and content characteristics are input into the sorting model (i.e., the click-through rate prediction model) for processing, and the user's click probability for the content is output. Based on the click probability, the contents in the candidate set are sorted in descending order of click probability, and a predetermined number of contents are selected as recommended contents, and the determined one or more recommended contents are sent to the client for display.

[0062] Sample construction of the ranking model of the content recommendation system is a problem that personalized recommendation often faces. The embodiment of the present invention needs to solve how to reasonably optimize the sample distribution in the recommendation system that integrates multiple types of content, so as to more comprehensively reflect the user behavior characteristics, thereby improving the effect of the trained ranking model, thereby improving the overall CTR of personalized recommendation, and ultimately improving the user retention rate.

[0063] In an embodiment of the present invention, the data storage device 120 is also suitable for storing a training sample library, which includes a first training sample library and a second training sample library. The first training sample library is generated based on the user's homepage behavior log, and specifically includes a plurality of first training samples, which are divided into two types: first positive samples and first negative samples. The first positive sample is a training sample corresponding to the content displayed on the homepage and clicked by the user, and the first negative sample is a training sample corresponding to the content displayed on the homepage but not clicked by the user.

[0064] The second training sample library is generated based on the user sub-page behavior log. In an embodiment of the present invention, the second training sample library only includes positive samples, referred to as second positive samples. The second positive samples are training samples corresponding to content displayed on the sub-page and clicked by the user.

[0065] Among them, how to construct training samples based on behavior logs is a relatively well-known technology in the field and will not be elaborated here. Those skilled in the art can use one or more existing methods to construct training samples. The sample features of the constructed training samples usually include user features and content features. User features may include one or more of the following features: 1. User attributes, such as gender, age, occupation, region, etc. 2. User behavior, such as browsing time in different time windows, clicks, searches, posts, collections, likes and other behaviors, 3. Interest preferences generated based on behavior, such as car series, tag preferences, etc. 4. Derived statistical indicators based on behavior, such as user CTR, activity, etc.

[0066] Content features may include, for example, one or more of the following features: 1. Attributes of the content itself, such as title, number of words in the text, number of pictures, etc.; 2. Features based on content mining, such as content type (long text, short text, video, travel notes), keywords, emotions, content professionalism, content richness, author influence, etc.; 3. Behaviors assigned to the content by users, such as uv, pv, ctr, collection, like, reply, etc.

[0067] The training sample processing method of the sorting model in the embodiment of the present invention optimizes the sample distribution of the existing training samples in the sample library, mainly targeting at least one of the following problems:

[0068] How to reasonably integrate positive and negative samples of each content type;

[0069] How to reasonably reduce the impact of the data imbalance problem of positive and negative samples;

[0070] How to enhance the effect of the ranking model by adjusting the distribution of samples without overfitting or underfitting.

[0071] The sample processing method of the ranking model according to the embodiment of the present invention may be executed in a computing device. Figure 2 FIG. 2 shows a block diagram of a computing device 200 according to an embodiment of the present invention. Figure 2 As shown, in a basic configuration 202, computing device 200 typically includes a system memory 206 and one or more processors 204. A memory bus 208 may be used for communication between processor 204 and system memory 206.

[0072] Depending on the desired configuration, the processor 204 can be any type of processor, including but not limited to: a microprocessor (μP), a microcontroller (μC), a digital information processor (DSP), or any combination thereof. The processor 204 can include one or more levels of cache such as a primary cache 210 and a secondary cache 212, a processor core 214, and registers 216. An example processor core 214 can include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. An example memory controller 218 can be used with the processor 204, or in some implementations, the memory controller 218 can be an internal part of the processor 204.

[0073] Depending on the desired configuration, the system memory 206 can be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. The system memory 106 may include an operating system 220, one or more applications 222, and program data 224. The application 222 is actually a plurality of program instructions, which are used to instruct the processor 204 to perform corresponding operations. In some embodiments, the application 222 can be arranged to cause the processor 204 to operate using the program data 224 on the operating system.

[0074] Computing device 200 may also include a storage device 232 including a removable storage 236 and a non-removable storage 238 , and a storage interface bus 234 .

[0075] The computing device 200 may also include an interface bus 240 that facilitates communication from various interface devices (e.g., output devices 242, peripheral interfaces 244, and communication devices 246) to the basic configuration 202 via the bus / interface controller 230. Example output devices 242 include a graphics processing unit 248 and an audio processing unit 250. They can be configured to facilitate communication with various external devices such as a display or speakers via one or more A / V ports 252. Example peripheral interfaces 244 may include a serial interface controller 254 and a parallel interface controller 256, which may be configured to facilitate communication with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, touch input devices) or other peripherals (e.g., printers, scanners, etc.) via one or more I / O ports 258. Example communication devices 246 may include a network controller 260, which may be arranged to facilitate communication with one or more other computing devices 262 via a network communication link via one or more communication ports 264.

[0076] A network communication link can be an example of a communication medium. Communication media can generally be embodied as computer-readable instructions, data structures, program modules in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium. A "modulated data signal" can be a signal in which one or more of its data sets or its changes can be carried out in a manner that encodes information in the signal. As non-limiting examples, communication media can include wired media such as a wired network or a dedicated line network, and various wireless media such as sound, radio frequency (RF), microwave, infrared (IR) or other wireless media. The term computer-readable medium used herein can include both storage media and communication media.

[0077] In the computing device 200 according to the present invention, the application 222 includes a plurality of program instructions for executing the training sample processing method 300 of the ranking model.

[0078] Figure 3 A flow chart of a training sample processing method 300 for a ranking model according to an embodiment of the present invention is shown. The method 300 can be executed in a computing device 200, which is in communication with a data storage device 120. The data storage device 120 stores a plurality of first positive samples, a plurality of first negative samples, and a plurality of second positive samples. The first positive samples are training samples corresponding to content displayed on the homepage and clicked by the user, the first negative samples are training samples corresponding to content displayed on the homepage but not clicked by the user, and the second positive samples are training samples corresponding to content displayed on a subpage and clicked by the user. The so-called display means sending the content to the user's client and displaying it to the user on the client's screen, and the so-called click means that the user clicks on the displayed content on the client.

[0079] Because other business types in the entire site are complicated, only samples of user interaction behaviors are recorded on the sub-pages, that is, positive samples with clicks are used as sub-page sample data, which is conducive to improving the accuracy of the trained sorting model.

[0080] like Figure 3 As shown, the method 300 begins with step S302. In step S302, conflicting samples are deleted from a plurality of first positive samples and a plurality of first negative samples to obtain a main page sample library.

[0081] In the embodiment of the present invention, the conflict sample may include:

[0082] 1) A first negative sample having the same content features and user features as any first positive sample.

[0083] That is to say, multiple samples with the same sample features (including content features and user features) but inconsistent labels are filtered out from the homepage sample data, and only positive samples are retained. This means that for the same user, when opening the homepage once, he did not click on a certain content on the homepage, but when opening the homepage another time, he clicked on the content on the homepage. In this case, only click samples are retained. Alternatively, when a user scrolls through a homepage, he may scroll up and down, click on a certain content after seeing it for the first time, and see it again when scrolling down, but does not click on it. In this case, only click samples are retained.

[0084] 2) A first negative sample having the same content features and user features as any second positive sample.

[0085] Some content is displayed on both the home page and the sub-pages. If a user clicks on the content on the sub-page, he may not click on the content on the home page. It is not because he is not interested in the content, but because he has clicked on the sub-page. In this case, the negative samples of the main page corresponding to the content will be filtered out.

[0086] By deleting conflicting samples, the fault tolerance rate of the homepage can be reduced, and the proportion of positive samples on the homepage can be increased. In order to ensure that the ratio of positive and negative samples in the main page sample library is as close as possible to the real environment of the homepage, in some embodiments, the first negative sample in the main page sample library is also oversampled (the number of first negative samples is increased). According to business experience, in the personalized results of the recall, the goal of the sorting model is to select the best from the best, that is, to select the content that is more likely to be clicked from the content that the user is likely to click. Therefore, some negative samples whose partial features are closer to the positive samples (i.e., difficult-to-distinguish negative samples) can be selected for oversampling. Oversampling can use SMOTE sampling, or other types of oversampling algorithms.

[0087] The difficult negative sample may be a negative sample whose content features match the user features. Specifically, for each first negative sample in the main page sample library, the matching degree between the user features and the content features of the negative sample is calculated. If the matching degree is greater than a preset value, the content features are considered to match the user features. For the method for calculating the matching degree between the user features and the content features, those skilled in the art may adopt an existing feature matching method, and the preset value may be determined by those skilled in the art based on experience or experiments.

[0088] In step S304, the proportion of the display volume of each content type on the homepage during the historical delivery process is calculated. The historical delivery process is, for example, the delivery process of the previous day, the previous week, or the previous month, or the delivery process of the previous N days, where N can be reasonably selected by a person skilled in the art. The content types are, for example, long articles, short articles, videos, and travel notes.

[0089] Specifically, if there are n content types, namely content type 1, content type 2, ..., content type n, then based on the homepage user log, the display volume of content of each content type (the sum of the display volumes of all content types of the content on the homepage) can be obtained, which are display quantity 1, display quantity 2, ..., display quantity n, respectively. Sum these n quantities to get the total display volume of content of all content types on the homepage, and then divide the total display volume by display quantity 1, display quantity 2, ..., display quantity n, respectively, to get the corresponding display volume share.

[0090] In step S306, downsampling is performed on the plurality of second positive samples based on the display volume ratio to obtain a predetermined number of second positive samples as a sub-page sample library.

[0091] Specifically, the number of first positive samples in the main page sample library determined in step S302 may be first obtained, and then the number may be multiplied by a predetermined ratio to obtain the predetermined number. The predetermined ratio may be reasonably set by a person skilled in the art, for example, 50%, that is, the number of positive samples in the sub-page sample library is half the number of positive samples in the main page sample library.

[0092] The downsampling process may include:

[0093] For each content type, multiply its corresponding display volume ratio by the predetermined number to obtain the number m of positive samples corresponding to the content type, then obtain all second positive samples corresponding to the content of this content type, perform downsampling on all the obtained second positive samples, obtain m samples from them and add them to the sub-page sample library, after processing all content types, obtain the final sub-page sample library.

[0094] Sampling sub-page samples based on the proportion of home page displays can reasonably control the number of positive samples of sub-pages to ensure that home page behavior samples still dominate in model training, while enriching user click behavior trajectories.

[0095] In step S308, the main page sample library and the sub-page sample library are merged and deduplicated to obtain a training sample library for the ranking model. Deduplication means that only one sample is retained for multiple repeated samples. The ranking model is trained based on the training sample library to obtain a ranking model for the recommendation system. The embodiment of the present invention does not limit the ranking model used, for example, it can be LR, XGBoost, FM, deepFM, etc.

[0096] The various techniques described herein may be implemented in combination with hardware or software, or a combination thereof. Thus, the method and apparatus of the present invention, or some aspects or parts of the method and apparatus of the present invention may be in the form of program codes (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, a USB flash drive, a floppy disk, a CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present invention.

[0097] In the case where the program code is executed on a programmable computer, the computing device generally includes a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store the program code; the processor is configured to execute the method of the present invention according to the instructions in the program code stored in the memory.

[0098] By way of example and not limitation, readable media include readable storage media and communication media. Readable storage media stores information such as computer readable instructions, data structures, program modules or other data. Communication media generally embody computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information transfer medium. Any combination of the above is also included within the scope of readable media.

[0099] In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the examples of the present invention. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to implement the content of the present invention described herein, and the above description of specific languages ​​is for the purpose of disclosing the preferred embodiment of the present invention.

[0100] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0101] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim.

[0102] Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in the devices described in the embodiment, or alternatively may be located in one or more devices different from the devices in the examples. The modules in the foregoing examples may be combined into one module or may be divided into multiple submodules.

[0103] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0104] In addition, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform the functions. Therefore, a processor with necessary instructions for implementing the method or method elements forms a device for implementing the method or method elements. In addition, the elements described herein of the device embodiments are examples of devices for implementing the functions performed by the elements for the purpose of implementing the invention.

[0105] As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe common objects merely indicates that different instances of similar objects are involved, and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner.

[0106] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative and not restrictive with respect to the scope of the present invention, and the scope of the present invention is defined by the appended claims.

Claims

1. A method for processing training samples of a ranking model, executed in a computing device, the computing device being in communication connection with a data storage device, the data storage device storing a plurality of first positive samples, a plurality of first negative samples, and a plurality of second positive samples, the first positive samples being training samples corresponding to content displayed on the homepage and clicked by a user, the first negative samples being training samples corresponding to content displayed on the homepage but not clicked by a user, and the second positive samples being training samples corresponding to content displayed on a subpage and clicked by a user, the method comprising: Deleting conflicting samples from the plurality of first positive samples and the plurality of first negative samples to obtain a main page sample library; Calculate the proportion of content displayed on the homepage by each content type during the historical delivery process; Performing downsampling processing on the plurality of second positive samples based on the display volume ratio to obtain a predetermined number of second positive samples as a sub-page sample library; The main page sample library and the sub-page sample library are combined and deduplicated to obtain a training sample library for the ranking model.

2. The method of claim 1, wherein: The conflict samples include: A first negative sample having the same content features and user features as any first positive sample, and / or a first negative sample having the same content features and user features as any second positive sample.

3. The method according to claim 1 or 2, wherein: Before merging and deduplicating the main page sample library and the sub-page sample library, it also includes: An oversampling process is performed on the first negative sample in the main page sample library.

4. The method of claim 3, wherein: The oversampling process of the first negative sample in the main page sample library includes: Determine a plurality of hard-to-distinguish negative samples from the first negative samples included in the main page sample library; Oversampling is performed on the multiple hard-to-distinguish negative samples.

5. The method of claim 4, wherein: The hard-to-separate negative samples are negative samples whose content features match user features.

6. The method of claim 3, wherein: The oversampling adopts SMOTE sampling.

7. The method of claim 1, wherein: The predetermined number is a predetermined ratio of the number of positive samples in the main page sample library.

8. The method of claim 1, wherein: The content type includes at least one of the following: long text, short text, video and travel notes.

9. A computing device comprising: at least one processor; and A memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, and the program instructions include instructions for executing the method according to any one of claims 1 to 8.

10. A readable storage medium storing program instructions, when the program instructions are read and executed by a computing device, the computing device executes the method according to any one of claims 1 to 8.

Images