Method, device, equipment and storage medium for determining sorting model

Abstract

The present disclosure provides a method, apparatus, device and storage medium for determining a ranking model. It relates to the field of computer technology, and in particular to the fields of data processing, machine learning, intelligent search, and intelligent recommendation. The specific implementation scheme is: obtaining a sample set and a feature set of a target product; determining a candidate ranking model for the target product from a model library; determining a target feature set for the target product based on the feature set, where the target feature set is a set of features used for model parameter adjustment; adjusting the parameters of the candidate ranking model based on the sample set and the target feature set to obtain a target ranking model for the target product, which is used to predict the click-through rate of the target product's resources. According to the technical solution of the present disclosure, a target ranking model that is suitable for each product can be provided, thereby improving the recommendation performance of each product.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the fields of data processing, machine learning, intelligent search, and intelligent recommendation. Background Art

[0002] With the rapid growth of businesses, a large number of scenarios have emerged where recommendation services are needed. To conserve machine resources, a single recommendation platform may be responsible for recommending multiple products. Related technologies use ranking models to rank resources for various products and recommend relevant resources to users based on the ranking. However, due to the varying characteristics of various products, using a single ranking model for ranking results in poor recommendation results. Summary of the Invention

[0003] The present disclosure provides a method, apparatus, device, and storage medium for determining a ranking model.

[0004] According to a first aspect of the present disclosure, a method for determining a ranking model is provided, comprising:

[0005] Obtain sample sets and feature sets of target products;

[0006] Determine candidate ranking models for the target product from the model library;

[0007] determining a target feature set for a target product based on the feature set;

[0008] Based on the sample set and the target feature set, the parameters of the candidate ranking model are adjusted to obtain a target ranking model for the target product, and the target ranking model is used to predict the click-through rate of the resources of the target product.

[0009] According to a second aspect of the present disclosure, a device for determining a ranking model is provided, comprising:

[0010] An acquisition unit, used to acquire a sample set and a feature set of a target product;

[0011] A first determining unit is used to determine a candidate ranking model for a target product from a model library;

[0012] a second determining unit, configured to determine a target feature set for the target product based on the feature set;

[0013] The third determining unit is used to adjust the parameters of the candidate ranking model based on the sample set and the target feature set to obtain a target ranking model for the target product, where the target ranking model is used to predict the click rate of the resources of the target product.

[0014] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided by the first aspect above.

[0018] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method provided by the first aspect above.

[0019] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method provided in the first aspect when executed by a processor.

[0020] According to the technical solution disclosed in the present invention, a target ranking model suitable for each product can be provided, thereby improving the recommendation performance of each product.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a flowchart of a method for determining a ranking model according to an embodiment of the present disclosure;

[0024] Figure 2 is a schematic diagram of an architecture for determining a ranking model according to an embodiment of the present disclosure;

[0025] Figure 3 is a schematic diagram of the architecture of a recommendation system according to an embodiment of the present disclosure;

[0026] Figure 4 is a schematic diagram of a resource recommendation process according to an embodiment of the present disclosure;

[0027] Figure 5 is a structural diagram of a device for determining a ranking model according to an embodiment of the present disclosure;

[0028] Figure 6 is a schematic diagram of a scenario of a customized sorting model according to an embodiment of the present disclosure;

[0029] Figure 7It is a block diagram of an electronic device used to implement the method for determining a ranking model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0031] The terms "first," "second," and "third," etc., in the description, embodiments, claims, and accompanying figures of the present disclosure are used to distinguish similar items and are not necessarily used to describe a particular order or precedence. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions, such as, for example, inclusion of a series of steps or elements. A method, system, product, or apparatus is not necessarily limited to the steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to the process, method, product, or apparatus.

[0032] Before introducing the technical solutions of the embodiments of the present disclosure, the following technical terms that may be used in the present disclosure are further explained:

[0033] Neural Network Intelligence (NNI) is a lightweight yet powerful toolkit that helps users automate feature selection, neural network architecture search, hyperparameter tuning, and model compression. The Tree-Structured Parzen Estimator Approach (TPE) is a method for adjusting hyperparameters in machine learning models.

[0034] Machine learning models generally have two types of parameters. One type can be estimated from data, known as parameters. Another type cannot be estimated from data and must be designed and specified based on experience, known as hyperparameters. Hyperparameters can be set before the learning process begins. In contrast, the values ​​of other parameters are determined through training.

[0035] As described in the background technology section above, to conserve machine resources, a single recommendation platform (also known as a recommendation system) needs to be responsible for recommending multiple products. In related technologies, there are three main recommendation approaches. The first approach aggregates and fuses recall resources for various products and outputs them directly. Specifically, resources recalled from different recall channels are aggregated and filtered through a convergence module (Global Recommend, GR), and then diversity-controlled and directly output. This recommendation approach does not utilize any ranking model and relies entirely on the order in which recall channels recall resources. This lacks rational ranking and results in poor recommendation accuracy. The second approach uses a coarse ranking model to provide estimates for all products. This approach addresses the issue of unranked recall resources to a certain extent, but the coarse ranking results are not very accurate. Using the same coarse ranking model when product relevance is low can lead to poor recommendation results, impacting recommendation performance. The third approach uses a single refined ranking model to provide estimates for all products. This approach ranks each product line using the refined ranking model, which can yield relatively accurate ranking scores. However, using the same refined ranking model can still lead to poor recommendation results, impacting recommendation performance.

[0036] To at least partially address one or more of the aforementioned and other potential issues, the present invention applies NNI technology and concepts to a machine learning ranking model. This allows the ranking model to automatically perform model selection, feature selection, and hyperparameter selection based on the characteristics of sample data after acquiring sample data. Furthermore, the ranking model estimates the click-through rate (CTR) of resources, thereby achieving the effect of improving recommendation performance by customizing the ranking model for different products.

[0037] The present disclosure provides a method for determining a ranking model. Figure 1 It is a flow chart of a method for determining a ranking model according to an embodiment of the present disclosure, and the method for determining a ranking model can be applied to a device for determining a ranking model. The device for determining the ranking model is located in an electronic device, which can be a part of the recommendation system, or can be independent of the recommendation system but can be connected to the recommendation system for communication. The electronic device includes but is not limited to fixed devices and / or mobile devices. For example, fixed devices include but are not limited to servers, and servers can be cloud servers or ordinary servers. For example, mobile devices include but are not limited to: one or more of: mobile phones, tablet computers, and vehicle-mounted terminals. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in a memory. For example Figure 1 As shown, the method for determining the sorting model includes:

[0038] S101: Obtain a sample set and feature set of a target product;

[0039] S102: Determine a candidate ranking model for the target product from the model library;

[0040] S103: Determine a target feature set for the target product based on the feature set;

[0041] S104: Based on the sample set and the target feature set, adjust the parameters of the candidate ranking model to obtain a target ranking model for the target product. The target ranking model is used to predict the click-through rate of the resources of the target product.

[0042] In the disclosed embodiment, the target product is a product for which a target ranking model is to be determined. Generally speaking, the target product is an application (Application, APP for short) product that can provide information to users. Exemplarily, the target product is an application that can provide information services to users. Here, the target products include but are not limited to web APPs, native APPs, and hybrid APPs. For example, the Baidu mobile APP belongs to the web APP category. For example, the photo album and Lemon Love Beauty developed by Baidu are all native APPs. For example, Baidu Maps, Baidu Encyclopedia, etc. belong to hybrid APPs. The above is only an exemplary explanation and is not intended to limit all possible types of target products. It is just not exhaustive here.

[0043] In the disclosed embodiments, a sample set is a set of training samples or historical real samples of a target product. Furthermore, a sample set may include a training set, a validation set, and a test set. The above description is merely illustrative and does not limit all possible sample types in a sample set; however, this is not intended to be exhaustive.

[0044] In the embodiment of the present disclosure, the feature set is a set consisting of the features of the target product. Exemplarily, the feature set includes the features of the object for which the product is intended. For example, the features may be audience group features, which can be further subdivided into features such as user gender, user age group, user geographic location information, and user annual income. As another example, the feature set includes the features of the resource types supported by the product itself. For example, the features include support for text and pictures. For another example, the features include support for pictures but not for videos. The above is only an exemplary explanation and is not intended to limit all possible feature types in the feature set, but it is not intended to be exhaustive here.

[0045] In the disclosed embodiment, the target feature set is a set of valid features selected from the feature set for model parameter adjustment. In some embodiments, the selection can be based on the characteristics of the target product. For example, if the target product's target audience is universal users, the feature of the user's age group in the feature set can be eliminated, and the feature of the user's age group is not a valid feature. In other embodiments, the selection can be based on the characteristics of the sorting model. For example, if sorting model A cannot distinguish the feature of user gender, the feature of user gender in the feature set can be eliminated, and the feature of user gender is not a valid feature. The above is only an illustrative explanation, and is not intended to limit all possible selection methods in the target feature set, but it is not exhaustive here.

[0046] In the embodiment of the present disclosure, the model library includes multiple ranking models. The ranking model is used to predict the user's click-through rate for the resources included in the product, and the ranking model can be further used to rank the predicted click-through rates of each resource. For example, the preset model can be a Regional Convolutional Neural Network (RCNN) model. For another example, the preset model can be a Fully Convolutional Network (FCN) model. The embodiment of the present disclosure does not limit the architecture of the ranking model.

[0047] In embodiments of the present disclosure, the sample set and feature set of a target product may be determined based on information input by a user at a terminal. In some embodiments, the electronic device directly receives the sample set and feature set of the target product from the terminal. In other embodiments, the electronic device receives the sample set and feature set of the target product forwarded by another device. This disclosure does not limit the communication method between the terminal and the electronic device.

[0048] In the disclosed embodiment, adjusting the parameters of the candidate ranking model at least includes adjusting common parameters of the ranking model, namely, neural network parameters.

[0049] In the embodiment of the present disclosure, the target ranking model is a ranking model used to specifically provide prediction services for target products.

[0050] The technical solution of the disclosed embodiment obtains a sample set and feature set for a target product; determines a candidate ranking model for the target product from a model library; determines a target feature set for the target product based on the feature set; and adjusts the parameters of the candidate ranking model based on the sample set and the target feature set to obtain a target ranking model for the target product. In this way, a target ranking model suitable for the target product can be determined, achieving customization of the ranking model. Using the target ranking model suitable for the target product to predict click-through rates for resources of the target product can improve the accuracy of resource click-through rate predictions, thereby helping to improve the recommendation performance of the target product.

[0051] In some embodiments, the above-mentioned method for determining the ranking model may further include: determining the hyperparameters to be adjusted of the candidate ranking model based on the attributes of the candidate ranking model; and adjusting the hyperparameters to be adjusted of the candidate ranking model based on the sample set and the target feature set.

[0052] Here, the attributes of the candidate ranking model include the attribute of the types of hyperparameters included in the candidate ranking model. Different candidate ranking models may include different numbers of hyperparameter types.

[0053] For example, candidate ranking model 1 includes deepfm hyperparameters and fm hyperparameters. Candidate ranking model 2 includes deepfm hyperparameters and ffm hyperparameters. Candidate ranking model 3 includes fm hyperparameters and ffm hyperparameters. Candidate ranking model 4 includes deepfm hyperparameters, fm hyperparameters, and ffm hyperparameters.

[0054] In some embodiments, determining hyperparameters to be adjusted for the candidate ranking model based on attributes of the candidate ranking model includes: determining the hyperparameters to be adjusted for the candidate ranking model based on the number of hyperparameter types included in the candidate ranking model. For example, if the candidate ranking model includes N hyperparameters, then the N hyperparameters included in the candidate ranking model are determined as the hyperparameters to be adjusted.

[0055] In other embodiments, determining the hyperparameters to be adjusted of the candidate ranking model based on the attributes of the candidate ranking model includes: determining the hyperparameters to be adjusted of the candidate ranking model based on the number of types of hyperparameters included in the candidate ranking model and the influencing factors of the hyperparameters. For example, the candidate ranking model includes hyperparameter 1 and hyperparameter 2, and the influencing factor of hyperparameter 1 is 0.99, and the influencing factor of hyperparameter 2 is 0.01; then, in order to improve the generation speed of the target ranking model, only hyperparameter 1 can be determined as the hyperparameter to be adjusted. In order to improve the generation accuracy of the target ranking model, hyperparameter 1 and hyperparameter 2 are determined as the hyperparameters to be adjusted.

[0056] In practical applications, the hyperparameters of the candidate ranking model can be adjusted based on TPE.

[0057] It should be noted that adjusting the hyperparameters of the candidate ranking model can be performed simultaneously with adjusting the parameters of the candidate ranking model. Alternatively, adjusting the hyperparameters of the candidate ranking model can be performed after adjusting the parameters of the candidate ranking model. Alternatively, adjusting the hyperparameters of the candidate ranking model can be performed before adjusting the parameters of the candidate ranking model.

[0058] In this way, when adjusting the parameters of the candidate ranking model, the hyperparameters of the candidate ranking model are also adjusted, which helps to improve the prediction accuracy of the determined target ranking model, thereby helping to improve the recommendation effect of the target ranking model.

[0059] In some embodiments, determining a candidate ranking model for a target product from a model library includes: obtaining model characteristics of each ranking model in the model library; obtaining product characteristics of the target product; and determining a ranking model that matches the model characteristics in the model library with the product characteristics of the target product as a candidate ranking model for the target product.

[0060] In the disclosed embodiments, model characteristics include the characteristics of the scenarios to which the model is applicable. For example, ranking model A is suitable for analyzing images, ranking model B is suitable for analyzing videos, and ranking model C is suitable for analyzing web pages.

[0061] In the disclosed embodiments, product features include the type of product resources. For example, Product A provides image recommendation services for users. Product B provides video recommendation services for users. Product C provides novel recommendation services for users.

[0062] Here, matching means that the model characteristics can at least partially meet the requirements of the product characteristics. In the above example, sorting model a matches product A; sorting model b matches product B; and sorting model c matches product C.

[0063] In some embodiments, ranking models in the model library whose model characteristics match the characteristics of the target product are determined as candidate ranking models for the target product. This includes, if the model characteristics fully cover the product characteristics, determining a ranking model with more model characteristics as a candidate ranking model. For example, if product D can provide users with video, image, and text services, ranking model D will be selected as a candidate ranking model if it can recognize videos, images, text, and music.

[0064] In some embodiments, ranking models in the model library whose model characteristics match the characteristics of a target product are determined as candidate ranking models for the target product. This includes, if the model characteristics do not fully cover the product characteristics, determining the ranking model whose model characteristics cover the most product characteristics as the candidate ranking model. For example, a feature of product E is that it can provide users with video, image, and text services. If ranking model e can recognize video and text, ranking model f can recognize images, and ranking model g can recognize video, then ranking model e is selected as a candidate ranking model for product E.

[0065] In this way, the determined candidate ranking model can be adapted to the characteristics of the target product, which helps to improve the matching degree between the determined target ranking model and the target product, thereby helping to improve the prediction accuracy of the target ranking model.

[0066] In some embodiments, determining a candidate ranking model for a target product from a model library includes: obtaining a target identifier of the ranking model based on received configuration information of the target product; and determining the ranking model identified as the target identifier in the model library as a candidate ranking model for the target product.

[0067] Here, the configuration information includes identification information of the specified sorting model.

[0068] In the embodiment of the present disclosure, the configuration information of the target product can be input by the user through a user interface. The user interface is an interface provided by the recommendation system to the user.

[0069] For example, if the configuration information of the target product includes an identifier h of a specified ranking model, the ranking model identified as h in the model library is determined as a candidate ranking model.

[0070] It should be noted that the configuration information may carry identifiers of multiple sorting models.

[0071] In actual applications, the configuration information may also include a sample set and a feature set of the target product.

[0072] In this way, the candidate ranking model can be determined according to the identification information of the specified ranking model, thereby improving the customization level of determining the target ranking model for the product and better meeting the needs of the product.

[0073] In some embodiments, when there are multiple candidate ranking models, the parameters of the candidate ranking models are adjusted based on the sample set and the target feature set to obtain the target ranking model of the target product, including: adjusting the parameters corresponding to the multiple candidate ranking models based on the sample set and the target feature set to obtain multiple target candidate models; verifying the multiple target candidate models to obtain the first accuracy of the multiple target candidate models; and determining the target candidate model with the highest first accuracy as the target ranking model of the target product.

[0074] In some embodiments, validating multiple target candidate models to obtain first accuracy rates for the multiple target candidate models includes validating the multiple target candidate models using a portion of samples from a sample set to obtain the first accuracy rates for the multiple target candidate models. Here, if the sample set includes a training set, a test set, and a validation set, the portion of samples may be samples from the validation set or samples from the test set.

[0075] In some embodiments, validating the multiple target candidate models to obtain first accuracy rates for the multiple target candidate models includes validating the multiple target candidate models using samples from a non-current sample set to obtain the first accuracy rates for the multiple target candidate models. The non-current sample set may be a historical sample set or a sample set specifically configured to validate the prediction accuracy of the target ranking model.

[0076] Here, taking model M1 as an example, the first accuracy rate r1 = the number of samples in the validation set accurately identified by model M1 / ​​the total number of samples in the validation set. The value of the first accuracy rate can reflect the performance of the target ranking model.

[0077] For example, some samples in the sample set are used to form a validation set, and m target candidate models are verified respectively to obtain the first accuracy r1 of model M1, the first accuracy r2 of model M2, ..., and the first accuracy rm of model Mm; if r1>r2>...>rm, then model M1 is determined as the target ranking model.

[0078] In this way, when there are multiple candidate ranking models, the optimal target ranking model can be selected for the target product, which helps to improve the prediction accuracy of the determined target ranking model, thereby helping to improve the prediction accuracy of the target ranking model.

[0079] In some embodiments, the above-mentioned method for determining the ranking model may further include: in response to detecting a change in the sample set, testing the target ranking model based on the changed sample set to obtain a second accuracy rate of the target ranking model; when the second accuracy rate is greater than a preset threshold, not performing an update operation on the target ranking model; when the second accuracy rate is less than or equal to the preset threshold, updating the parameters of the target ranking model based on the changed sample set.

[0080] Here, the preset threshold value can be set or adjusted according to design requirements such as speed requirements or accuracy requirements.

[0081] In some embodiments, the target ranking model is tested using the changed sample set to obtain a second accuracy of the target ranking model, including: obtaining the target sample that has changed relative to the target sample before the change; and testing the target ranking model based on the target sample to obtain the second accuracy of the target ranking model.

[0082] Here, the target samples include samples that exist in the sample set after the change but do not exist in the sample set before the change.

[0083] In this way, the time required for testing the target sorting model can be saved, thereby helping to quickly determine whether the target sorting model needs to be updated.

[0084] In some embodiments, testing the target ranking model with the modified sample set to obtain a second accuracy of the target ranking model includes: testing the target ranking model with all samples in the modified sample set to obtain the second accuracy of the target ranking model. In this way, the accuracy of the target ranking model can be better determined.

[0085] In this way, by first testing and then determining whether to adjust the target ranking model based on the test results, the speed of determining the target ranking model for the target product can be improved, which in turn helps speed up the provision of services for the target product; at the same time, the target ranking model can be changed in a timely manner according to changes in the sample set, and the target ranking model can be optimized, which helps improve the recommendation effect of the target product.

[0086] In some embodiments, the method for determining the ranking model may further include: in response to detecting a change in the feature set, adjusting the target feature set based on the updated feature set to obtain an updated target feature set. Adjusting parameters of the candidate ranking model based on the sample set and the target feature set to obtain a target ranking model for the target product includes: adjusting parameters of the candidate ranking model based on the sample set and the updated target feature set to obtain a target ranking model for the target product.

[0087] For example, if it is detected that the feature set P is changed to the feature set Q, then the feature set Q is the changed feature set.

[0088] In some embodiments, adjusting the target feature set based on the updated feature set to obtain the updated target feature set includes: obtaining target features that have changed in the updated feature set relative to the feature set before the update; and obtaining the updated target feature set based on the target features.

[0089] Here, the target features include features that exist in the updated feature set but not in the feature set before the update.

[0090] In this way, the determined target feature set can be made more targeted.

[0091] In some embodiments, adjusting the target feature set based on the updated feature set to obtain the updated target feature set includes: determining the updated target feature set based on all features in the updated feature set. In this way, the accuracy of the determined target feature set can be improved.

[0092] In this way, the target ranking model can be changed and optimized in a timely manner according to the changes in the feature set, which helps to improve the recommendation effect of the target product.

[0093] Figure 2 A schematic diagram of the architecture for determining the ranking model is shown, Figure 2As shown, the model library includes multiple ranking models, and the feature set includes user portrait features (user feature), resource features (doc feature), request features (request feature), and serialization features (session feature). User portrait features include but are not limited to gender, age, and education level. Resource features include but are not limited to resource resolution, resource size, and resource author. Request features include but are not limited to device type and device address. Serialization features include but are not limited to historical click records. Hyperparameters include but are not limited to: deepfm hyperparameters, fm hyperparameters, and ffm hyperparameters. Based on NNI technology, candidate ranking models, feature sets, and hyperparameters are selected for the target product from the model library. The parameters in the ranking model are adjusted according to the sample set and feature set, and the optimal target ranking model is finally obtained.

[0094] It should be understood that Figure 2 The schematic diagram of the structure shown is only exemplary and not restrictive, and it is extensible. Those skilled in the art can Figure 2 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0095] Figure 3 The architecture diagram of the recommendation system is shown in Figure 2. Figure 3 As shown in the figure, the recommendation system provides recommendation services for S products. Each product has a corresponding ranking model. Each product can determine the estimated click-through rate of resources through its own ranking model. Then, the aggregation module corresponding to each product aggregates the resources of each product to generate the final recommended resource list and return it to the client.

[0096] When a client detects a user's request to refresh its received information, it first sends a request to the recommendation system's aggregation module, which then sends requests to the downstream recall channels. Each recall channel employs different strategies to recall resources based on their attributes. After a ranking phase involving coarse sorting, fine sorting, and re-ranking, the resources are scored and returned to the aggregation module along with the scores. The aggregation module deduplicates, merges, filters, and sorts the resources recalled by each recall channel to generate a final list of recommended resources and returns it to the client. Based on this list of recommended resources, the client presents resources that match the user's interests. The recommendation system's aggregation module and recall channels are primarily designed and optimized for a single product. Only the ranking phase utilizes highly generalizable techniques such as machine learning and deep learning, enabling it to better integrate the characteristics of different product lines and improve ranking accuracy. Furthermore, automated feature selection and hyperparameter selection for each product line, coupled with shared machine learning, significantly reduces human resources and machine costs.

[0097] It should be understood that Figure 3 The schematic diagram of the structure shown is only exemplary and not restrictive, and it is extensible. Those skilled in the art can Figure 3 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0098] Based on the target ranking model determined for the target product, the recommendation system can make resource recommendations, such as Figure 4 As shown, the recommended process may include:

[0099] S401: Obtain a resource recommendation request for a target product sent by a client;

[0100] S402: Recalling the target product's resources through the target product's recall channel;

[0101] S403: Predicting the click rate of each resource recalled by the recall channel using the target ranking model of the target product, and ranking each resource;

[0102] S404: Aggregate the rankings of the resources to obtain a recommended resource list, which is a list of resources to be displayed;

[0103] S405: Return the recommended resource list to the client.

[0104] Here, the target product may correspond to multiple recall channels, and different recall channels may be used to recall different types of resources of the target product.

[0105] In this way, by adopting a target ranking model that is suitable for the target product and predicting the click rate of the target product's resources, the probability of the target product's resources being accessed can be increased.

[0106] The method for determining the ranking model provided by the present disclosure can be applied to information flow (Feed) recommendations. The recommendation system selects the most appropriate ranking model to access different product services, allowing each product line to share machine resources and solve the problem of online ranking models, thereby improving the accuracy of resource estimation clicks, saving a large amount of machine resources, reducing development costs, and improving efficiency.

[0107] The embodiment of the present disclosure provides a device for determining a ranking model, such as Figure 5As shown, the video processing device may include: an acquisition unit 501, used to acquire a sample set and a feature set of a target product; a first determination unit 502, used to determine a candidate ranking model for the target product from a model library; a second determination unit 503, used to determine a target feature set for the target product based on the feature set; a third determination unit 504, used to adjust the parameters of the candidate ranking model based on the sample set and the target feature set to obtain a target ranking model for the target product, and the target ranking model is used to predict the click-through rate of the resources of the target product.

[0108] In some embodiments, the device for determining the ranking model may also include: a fourth determination unit 505 (not shown in the figure), used to determine the hyperparameters to be adjusted of the candidate ranking model based on the attributes of the candidate ranking model; a fifth determination unit 506 (not shown in the figure), used to adjust the hyperparameters to be adjusted of the candidate ranking model based on the sample set and the target feature set.

[0109] In some embodiments, the first determination unit 502 includes: a first acquisition subunit, used to obtain the model characteristics of each ranking model in the model library; a second acquisition subunit, used to obtain the product characteristics of the target product; and a first determination subunit, used to determine the ranking model that matches the model characteristics in the model library with the product characteristics of the target product as a candidate ranking model for the target product.

[0110] In some embodiments, the first determination unit 502 includes: a third acquisition subunit, used to obtain the target identifier of the sorting model based on the received configuration information of the target product; and a second determination subunit, used to determine the sorting model identified as the target identifier in the model library as a candidate sorting model for the target product.

[0111] In some embodiments, the second determination unit 503 includes: an adjustment subunit for adjusting the parameters corresponding to the multiple candidate ranking models based on the sample set and the target feature set when there are multiple candidate ranking models, so as to obtain multiple target candidate models; a third determination subunit for verifying the multiple target candidate models to obtain the first accuracy of the multiple target candidate models; and a fourth determination subunit for determining the target candidate model with the highest first accuracy as the target ranking model of the target product.

[0112] In some embodiments, the device for determining the ranking model may also include: a verification unit 507 (not shown in the figure), which is used to test the target ranking model with the changed sample set in response to detecting a change in the sample set, and obtain a second accuracy rate of the target ranking model; a control unit 508 (not shown in the figure), which is used to not perform an update operation on the target ranking model when the second accuracy rate is greater than a preset threshold; and update the parameters of the target ranking model based on the changed sample set when the second accuracy rate is less than or equal to the preset threshold.

[0113] In some embodiments, the second determination unit 503 is further used to adjust the target feature set based on the updated feature set in response to detecting a change in the feature set, so as to obtain an updated target feature set; the third determination unit 504 is further used to adjust the parameters of the candidate ranking model based on the sample set and the updated target feature set, so as to obtain a target ranking model for the target product.

[0114] Those skilled in the art should understand that the functions of each processing module in the device for determining the sorting model of the embodiment of the present disclosure can be understood with reference to the relevant description of the method for determining the sorting model described above. Each processing module in the device for determining the sorting model of the embodiment of the present disclosure can be implemented by an analog circuit that implements the functions described in the embodiment of the present disclosure, or can be implemented by running software that executes the functions described in the embodiment of the present disclosure on an electronic device.

[0115] The ranking model determination device of the disclosed embodiments can determine a target ranking model suitable for a target product, thus achieving customization of the ranking model. Using a target ranking model suitable for the target product to predict click-through rates for resources related to the target product can improve the accuracy of resource click-through rate predictions, thereby contributing to improved recommendation performance for the target product.

[0116] Figure 6 A schematic diagram of the scenario for determining the target ranking model is shown. Figure 6 As can be seen, electronic devices, such as cloud servers, receive requests from various terminals to customize a target ranking model for a product. Based on the request, they determine a sample set and feature set for the product, select a candidate ranking model from a model library, determine a target feature set for each product, and automatically adjust the parameters of the candidate ranking model, including both common and hyperparameters. Finally, they determine a target ranking model for each product. Furthermore, a message indicating successful customization of the target ranking model can be returned to the terminal. Furthermore, a correspondence between the product and the target ranking model can be established, and upon detecting changes in the product's sample set or feature set, the target ranking model can be updated. In this way, a ranking model can be customized for each product. Using a target ranking model tailored to the product to predict click-through rates for its resources can improve the accuracy of resource click-through rate predictions, thereby contributing to improved recommendation performance for the target product.

[0117] Here are a few scenarios. For example, after product R is developed, relevant information about product R, such as the sample set and feature set, is sent to an electronic device, which then determines a ranking model suitable for product R. For another example, if the electronic device has already determined a ranking model suitable for product K, and a new function is added to product K, the feature set and sample set corresponding to the newly added function are sent to the electronic device, which then updates the original target ranking model to make it more suitable for product K with the newly added function.

[0118] It should be understood that Figure 6 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 6 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0119] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0120] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0121] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0122] like Figure 7As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0123] Various components in device 700 are connected to I / O interface 705, including an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, optical disk, etc.; and a communication unit 709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0124] The computing unit 701 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the method for determining the ranking model. For example, in some embodiments, the method for determining the ranking model can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the method for determining the ranking model described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the method for determining the ranking model in any other appropriate manner (for example, by means of firmware).

[0125] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0126] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0127] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0128] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0129] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0130] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0131] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0132] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for determining a ranking model, comprising: Obtain sample sets and feature sets of target products; Determining a candidate ranking model for the target product from a model library; Determining a target feature set for the target product based on the feature set; the target feature set is a feature set related to the features of the target product; wherein the features of the target product include resource type features or audience group features; Based on the sample set and the target feature set, adjusting the parameters of the candidate ranking model to obtain a target ranking model for the target product, wherein the target ranking model is used to predict the click-through rate of the resources of the target product; The method further comprises: Determining, based on the attributes of the candidate ranking model, hyperparameters to be adjusted of the candidate ranking model; wherein, determining, based on the attributes of the candidate ranking model, the hyperparameters to be adjusted of the candidate ranking model comprises: determining, based on the number of types of hyperparameters included in the candidate ranking model and influencing factors of the hyperparameters, the hyperparameters to be adjusted of the candidate ranking model; Based on the sample set and the target feature set, the hyperparameter to be adjusted of the candidate ranking model is adjusted.

2. The method according to claim 1, wherein Determining a candidate ranking model for the target product from a model library includes: Obtaining model characteristics of each ranking model in the model library; Obtaining product features of the target product; The ranking models whose model characteristics in the model library match the product characteristics of the target product are determined as candidate ranking models for the target product.

3. The method according to claim 1, wherein Determining a candidate ranking model for the target product from a model library includes: Obtaining a target identifier for a sorting model based on the received configuration information of the target product; The ranking model identified as the target identifier in the model library is determined as the candidate ranking model for the target product.

4. The method according to claim 1, wherein In the case where there are multiple candidate ranking models, adjusting the parameters of the candidate ranking models based on the sample set and the target feature set to obtain the target ranking model of the target product includes: Based on the sample set and the target feature set, adjusting the parameters corresponding to the plurality of candidate ranking models respectively to obtain a plurality of target candidate models; Verifying the multiple target candidate models to obtain first accuracy rates of the multiple target candidate models; The target candidate model with the highest first accuracy is determined as the target ranking model for the target product.

5. The method according to claim 1, further comprising: In response to detecting that the sample set has changed, testing the target ranking model using the changed sample set to obtain a second accuracy rate of the target ranking model; When the second accuracy rate is greater than a preset threshold, not performing an update operation on the target ranking model; When the second accuracy is less than or equal to the preset threshold, the parameters of the target ranking model are updated based on the changed sample set.

6. The method according to claim 1, further comprising: In response to detecting that the feature set has changed, adjusting the target feature set based on the updated feature set to obtain an updated target feature set; The adjusting the parameters of the candidate ranking model based on the sample set and the target feature set to obtain the target ranking model of the target product includes: Based on the sample set and the updated target feature set, the parameters of the candidate ranking model are adjusted to obtain the target ranking model of the target product.

7. A device for determining a ranking model, comprising: An acquisition unit, used to acquire a sample set and a feature set of a target product; A first determining unit, configured to determine a candidate ranking model for the target product from a model library; a second determining unit, configured to determine a target feature set for the target product based on the feature set; the target feature set is a feature set related to features of the target product; wherein the features of the target product include resource type features or audience group features; a third determining unit, configured to adjust parameters of the candidate ranking model based on the sample set and the target feature set to obtain a target ranking model for the target product, wherein the target ranking model is used to predict a click-through rate of resources of the target product; a fourth determining unit, configured to determine, based on the attributes of the candidate ranking model, the hyperparameters to be adjusted of the candidate ranking model; the fourth determining unit is specifically configured to determine, based on the number of types of hyperparameters included in the candidate ranking model and the influencing factors of the hyperparameters, the hyperparameters to be adjusted of the candidate ranking model; A fifth determining unit is configured to adjust the hyperparameter to be adjusted of the candidate ranking model based on the sample set and the target feature set.

8. The device according to claim 7, wherein The first determining unit includes: A first acquisition subunit is used to acquire the model characteristics of each ranking model in the model library; A second acquisition subunit is used to acquire product characteristics of the target product; The first determining subunit is configured to determine a ranking model whose model characteristics in the model library match the product characteristics of the target product as a candidate ranking model for the target product.

9. The device according to claim 7, wherein The first determining unit includes: A third acquisition subunit is configured to acquire a target identifier of a sorting model according to the received configuration information of the target product; The second determining subunit is configured to determine the ranking model identified as the target identifier in the model library as a candidate ranking model for the target product.

10. The device according to claim 7, wherein The second determining unit includes: an adjusting subunit, configured to adjust parameters corresponding to the plurality of candidate ranking models respectively based on the sample set and the target feature set, to obtain a plurality of target candidate models when there are a plurality of candidate ranking models; a third determining subunit, configured to verify the plurality of target candidate models to obtain first accuracy rates of the plurality of target candidate models; The fourth determining subunit is configured to determine the target candidate model with the highest first accuracy as the target ranking model for the target product.

11. The apparatus according to claim 7, further comprising: a verification unit, configured to, in response to detecting a change in the sample set, test the target ranking model using the changed sample set to obtain a second accuracy rate of the target ranking model; a control unit, configured to not perform an update operation on the target ranking model when the second accuracy rate is greater than a preset threshold; When the second accuracy is less than or equal to the preset threshold, the parameters of the target ranking model are updated based on the changed sample set.

12. The device according to claim 7, wherein The second determining unit is further configured to, in response to detecting that the feature set has changed, adjust the target feature set based on the updated feature set to obtain an updated target feature set; The third determining unit is further configured to adjust the parameters of the candidate ranking model based on the sample set and the updated target feature set to obtain the target ranking model for the target product.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images