Mixed content selection method and device, equipment and medium

Abstract

The embodiment of the invention provides a mixed content selection method and device, equipment and a medium. The method comprises the following steps: acquiring a plurality of materials belonging to at least two material types and a model pre-estimation score corresponding to each material; for each material type, selecting a preset candidate number of materials as candidate materials according to the model pre-estimation score of each material under the material type; performing unified calibration processing on the model pre-estimated score corresponding to each candidate material to obtain a calibration score corresponding to each candidate material; and according to the calibration scores of all the candidate materials, selecting a preset target number of candidate materials with the highest calibration score to form mixed content.

Description

Technical Field

[0001] The present invention relates to the field of recommendation system model calibration, and in particular to a mixed content selection method, device, equipment and medium. Background Art

[0002] As an information filtering system, the recommendation system calculates and predicts user interests based on the user's historical behavior and social relationships, and then filters through massive amounts of information to provide users with more accurate information; among them, social recommendations are based on user social networks, strengthen relationships between users, and make more accurate social content recommendations. Its purpose is to use user social relationships to explore content that users are interested in about the people they follow. Therefore, there will be many types of materials in this type of recommendation system, such as follower materials, follower behavior materials, follower interest materials, etc.

[0003] Calibration technology is a branch of machine learning. Due to the deviations in model sample sampling and model training, the final prediction results of the model deviate from the actual situation. Currently, common calibration technologies in the industry include PlattCalibration algorithm and Isotonic Regression algorithm.

[0004] In the process of implementing the present invention, the applicant found that due to factors such as model differences, the estimated scores were not comparable, and there was a significant deviation between the estimated scores and the true values, resulting in poor content mixing effects, which became one of the technical problems that needed to be urgently solved in the prior art. Summary of the invention

[0005] The embodiments of the present invention provide a mixed content selection method, device, equipment and medium, which solve the problem that the estimated scores are not comparable due to factors such as model differences, and there is a significant deviation between the estimated scores and the true values, resulting in poor content mixing effect.

[0006] To achieve the above-mentioned purpose, in a first aspect, an embodiment of the present invention provides a method for selecting mixed content, comprising:

[0007] Acquire multiple materials belonging to at least two material types and a model estimated score corresponding to each material; the model estimated score represents the predicted popularity of the material; the model estimated score is predicted by an estimated score model corresponding to the material type; different material types correspond to estimated score models in different business scenarios, the estimated score model is trained based on the real score of the material in the corresponding business scenario, and the real score represents the real popularity of the material;

[0008] For each material type, a preset number of candidate materials are selected as candidate materials according to the model estimated score of each material under the material type;

[0009] The model estimation score corresponding to each candidate material is uniformly calibrated to obtain a calibration score corresponding to each candidate material; the calibration score represents the calibration popularity of the material obtained after uniform calibration for different material types;

[0010] According to the calibration scores of all candidate materials, a preset target number of candidate materials with the highest calibration scores are selected to form the mixed arrangement content.

[0011] In a second aspect, an embodiment of the present invention provides a mixed content selection device, characterized by comprising:

[0012] A model estimation score acquisition unit is used to acquire a plurality of materials belonging to at least two material types and a model estimation score corresponding to each material; the model estimation score represents the predicted popularity of the material; the model estimation score is predicted by an estimation score model corresponding to the material type; different material types correspond to estimation score models in different business scenarios, the estimation score model is trained based on the real score of the material in the corresponding business scenario, and the real score represents the real popularity of the material;

[0013] A candidate material selection unit, for selecting a preset number of materials as candidate materials for each material type according to the model estimated score of each material under the material type;

[0014] A calibration unit, used to uniformly calibrate the model estimated score corresponding to each candidate material to obtain a calibration score corresponding to each candidate material; the calibration score represents the calibration popularity of the material obtained after uniform calibration for different material types;

[0015] The mixing unit is used to select a preset target number of candidate materials with the highest calibration scores to form mixed content according to the calibration scores of all candidate materials.

[0016] In a third aspect, an embodiment of the present invention provides a computer device, comprising at least one host, wherein the at least one host comprises a processor adapted to execute various instructions; and

[0017] A storage device is suitable for storing a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing any of the methods described above.

[0018] In a fourth aspect, an embodiment of the present invention provides a storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing any of the methods described above.

[0019] The above technical solution has the following beneficial effects: in view of the fact that different material types correspond to estimated score models in different business scenarios, a model calibration system is constructed for the model estimated scores obtained by different estimated score models. Different types of materials can be predicted through independent estimated score models to improve their respective accuracy. At the same time, score conversion based on a unified model calibration system can provide a more reasonable and efficient value ranking basis and improve the overall ranking accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 is a flow chart of a method for selecting mixed content according to one embodiment of the present invention;

[0022] Figure 2 is an architectural diagram of a mixed content selection device according to one embodiment of the present invention;

[0023] Figure 3 This is a data flow diagram of a multi-type content shuffling method based on model calibration according to one embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of the relationship between the estimated scores and the true values ​​after grouping by the estimated scores according to one embodiment of the present invention;

[0025] Figure 5 is a schematic diagram of the relationship between the estimated score and the true value after adjusting the monotonicity according to one embodiment of the present invention;

[0026] Figure 6 is a schematic diagram of a mapping relationship between an estimated score and a true value according to one embodiment of the present invention;

[0027] Figure 7 It is a schematic diagram of a sorting process according to a mapping relationship according to one embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The inventors analyzed the prior art and found that the main problems of the prior art are: in the current mixed sorting system of multiple types of materials, the commonly used solution is to build a unified sorting model for different types of materials for comparison and output, and use a unified model to make estimates for different types of materials, which cannot capture the differences between different types of materials and cannot be refined; other methods are to use different models to make estimates for different types of materials, and then compare them. Due to the differences in samples and models between models, the estimated scores are often not comparable; some methods are based on the original estimated scores of models obtained from different models. There is a deviation between the predicted value and the true value, and the materials cannot be accurately sorted; some calibration technologies perform estimation and calibration for a single model, which cannot handle the situation of multiple models for multiple types of materials.

[0030] In view of the shortcomings of the prior art, the inventors proposed a mixed content selection method, device, equipment and medium. In a multi-type content mixed recommendation system based on social networks, multiple estimated score models corresponding to multi-type materials are uniformly calibrated, and the original model estimated scores are combined with the calibrated scores to complete the precise mixing of multiple materials, greatly improving the overall mixing accuracy and efficiency.

[0031] First, as Figure 1 As shown, an embodiment of the present invention provides a method for selecting mixed content, including:

[0032] Step S10, obtaining a plurality of materials belonging to at least two material types and a model estimated score corresponding to each material; the model estimated score represents the predicted popularity of the material; the model estimated score is predicted by an estimated score model corresponding to the material type; different material types correspond to estimated score models under different business scenarios, the estimated score models are trained based on real scores under corresponding business scenarios, and the real scores represent the real popularity of the material;

[0033] Step S11, for each material type, selecting a preset number of candidate materials as candidate materials according to the model estimated score of each material under the material type;

[0034] Step S12, uniformly calibrating the model estimated score corresponding to each candidate material to obtain a calibration score corresponding to each candidate material; the calibration score represents the calibrated popularity of the material obtained after uniform calibration for different material types;

[0035] Step S13, according to the calibration scores of all candidate materials, a preset target number of candidate materials with the highest calibration scores are selected to form the mixed arrangement content.

[0036] In some embodiments, each material type customizes the estimated score model for the corresponding business scenario, and the estimated score models corresponding to different material types are independent of each other. For example, the estimated score models corresponding to different material types can be independently developed by different development groups, and can be obtained by training with the real scores generated during the provision of business services in different business scenarios. During the training period, the real score is used as the target value in the training data, and the loss function is calculated together with the model prediction value, and the model parameters of the corresponding estimated score model are updated by the gradient descent method. For example, each material type, used in a corresponding business scenario, corresponds to an estimated score model. The real score can be directly from the log file of the corresponding business scenario, or determined according to the frequency of user interaction operations on the material recorded in the log file of the corresponding business scenario. Each material type has a different estimated score model for different business scenarios, so that the prediction accuracy of each material type can be improved; but because the focus of each business scenario may be different, the interactive operation behavior of the user on the material is different, which accordingly leads to the incomparability of the real scores obtained in each business scenario, and the model estimated scores of the estimated score models obtained by using the incomparable real scores as the training target values ​​are also incomparable. When performing content shuffling services, directly performing content shuffling based on the incomparable model estimated scores output by each estimated score model will reduce the shuffling effect. It is necessary to uniformly calibrate the model estimated scores of each material to obtain comparable calibrated scores in order to perform more accurate shuffling. Material types include but are not limited to concerned materials, community materials, and non-concerned materials. For each material type, a preset candidate number of materials are selected as candidate materials according to the model estimated scores of each material under the material type. Specifically, for each material type, a preset candidate number of materials with the highest model estimated scores can be selected as candidate materials according to the model estimated scores of each material under the material type; each estimated score model is trained for the corresponding material type in the corresponding application business scenario, so that each estimated score model can make more accurate predictions for the materials of the corresponding material type. By selecting candidate materials based on the model estimated scores of each material under the material type, materials that are more suitable for pushing to users under the material type can be more accurately screened out, so as to achieve the purpose of accurate initial screening. The estimated score models under the application business scenarios corresponding to different material types have different model structures, or have the same model structure but different model parameters. As a result, the model estimated scores of materials of different material types are not comparable. If materials are sorted and selected directly based on the model estimated scores of materials of different material types, the mixed sorting effect will be reduced. By uniformly calibrating the model estimated scores of materials of each material type in the mixed sorting business scenario, the calibration score of each material is obtained, and the calibration scores of the materials are comparable. Then, based on the calibration scores of all candidate materials, the preset target number of candidate materials with the highest calibration scores are selected to form the mixed sorting content.

[0037] The embodiments of the present invention have the following technical effects: in view of the fact that different material types correspond to estimation score models in different business scenarios, a model calibration system is constructed for the model estimation scores obtained by different estimation score models. Different types of materials can be predicted through independent estimation score models to improve their respective accuracy. At the same time, score conversion based on a unified model calibration system can provide a more reasonable and efficient value ranking basis and improve the overall ranking accuracy.

[0038] Furthermore, the model estimation score corresponding to each candidate material is uniformly calibrated to obtain the calibration score corresponding to each candidate material, including:

[0039] For each candidate material, the model estimated score of the candidate material is mapped to the calibration score of the candidate material according to a pre-constructed estimated calibration mapping relationship corresponding to the material type of the candidate material.

[0040] In some embodiments, by analyzing the relationship between the model estimated score and the calibration score of the same material in historical data, an estimated-calibration mapping relationship can be pre-constructed. During business services, the model estimated score of the candidate material is mapped to the calibration score of the candidate material through the pre-constructed estimated-calibration mapping relationship.

[0041] The embodiments of the present invention have the following technical effects: once the estimated calibration mapping relationship is established, it can be directly used during business services. The estimated calibration mapping relationship can map the model estimated score of the candidate material to the calibration score of the candidate material through simple calculations. Compared with complex neural networks or expert systems with multiple condition judgments, it has higher execution efficiency, reduces the occupancy of server computing resources, and improves the server response rate.

[0042] Furthermore, the method for constructing the estimated calibration mapping relationship includes:

[0043] Acquire a material set for constructing an estimated calibration mapping relationship, wherein a plurality of sample materials in the material set belong to at least two material types;

[0044] For each sample material in the material set, obtain the model estimated score and the actual score of the sample material;

[0045] For each material type, according to the model estimated scores and actual scores of all sample materials under the material type, a target mapping relationship from the model estimated scores to the actual scores is fitted based on a preset mapping relationship structure, and the fitted target mapping relationship is used as the estimated calibration mapping relationship corresponding to the material type.

[0046] The estimated calibration mapping relationship corresponding to each material type is used to map the model estimated score of the material of the material type into a calibration score that can be compared among all material types.

[0047] In some embodiments, the model estimated score of each sample material in the material set comes from the corresponding estimated score model, and the model estimated scores obtained by each estimated score model are not comparable. Specifically, the model estimated score of the material of the corresponding material type can be obtained from the logs in each business scenario. The real score of each sample material in the material set can come from the same business scenario. For example, the materials accessed in the past period of time in the offline log file can be obtained in the same business scenario (such as a mixed push scenario or a relationship flow scenario), and the access situation of the corresponding material in the log includes but is not limited to click-through rate, viewing time, forwarding rate, etc. to comprehensively calculate the real score of the corresponding material. It is also possible to collect the real scores of the accessed materials in the recent period of time online in real time under the same business scenario. For each material type, the estimated calibration mapping relationship corresponding to the material type is fitted based on the same mapping relationship structure. Since the real scores used in the fitting are from user behaviors in the same business scenario, the real scores obtained are also comparable. Based on the comparable real scores and the same mapping relationship structure, an estimated calibration mapping relationship is fitted, which can map the model estimated scores output by various incomparable estimated score models into comparable calibration scores.

[0048] The embodiments of the present invention have the following technical effects: a unified model calibration system is constructed for model estimation scores of different estimation score models for different types of materials. Different types of materials can be predicted through independent estimation score models to improve their respective accuracies. At the same time, based on the unified model calibration system, a more reasonable and efficient sorting basis can be provided to improve the overall sorting accuracy. The embodiments of the present invention integrate multiple information sources, reduce the impact of deviations in each link, and improve the overall sorting system performance.

[0049] Furthermore, for each material type, according to the model estimated scores and the actual scores of all sample materials under the material type, a target mapping relationship from the model estimated score to the actual score is obtained based on a preset mapping relationship structure fitting, including:

[0050] For all sample materials under any material type, all sample materials are sorted according to the model estimated scores, and are divided into buckets at equal intervals to obtain a bucket sequence corresponding to the material type consisting of multiple buckets;

[0051] For each bucket in the bucket sequence corresponding to the material type, calculate the average model estimated score, the average true score, the bucket left boundary estimated score, and the bucket right boundary estimated score of all sample materials in the bucket;

[0052] For all the buckets in the bucket sequence corresponding to the material type, an order-preserving regression method is used to traverse all the buckets and merge the buckets to obtain an updated bucket sequence corresponding to the material type, so that the average true score of all the buckets in the updated bucket sequence has the same monotonicity as the average model estimated score;

[0053] According to the monotonicity of the average model estimated scores and the average true scores of multiple buckets in the updated bucket sequence corresponding to the material type, based on the preset mapping relationship structure, the parameter values ​​in the mapping relationship structure are fitted and determined to obtain the target mapping relationship.

[0054] In some embodiments, materials are sorted and divided into buckets, each bucket having an average estimated score and an average true score, and by sorting and merging the buckets, the average estimated score and the average true score of the updated bucket sequence corresponding to each material type are adjusted to the same monotonicity, thereby facilitating the construction of a simple linear mapping relationship.

[0055] Furthermore, for all the buckets in the bucket sequence corresponding to the material type, the order-preserving regression method is used to traverse all the buckets and merge the buckets to obtain an updated bucket sequence corresponding to the material type, so that the average true score of all the buckets in the updated bucket sequence has the same monotonicity as the average model estimated score, including:

[0056] When the monotonicity is selected as monotonically increasing, all buckets in the updated bucket sequence are traversed in turn in a loop. If the average true score of the i+1th bucket is less than the average true score of the i-th bucket, the i+1th bucket and the i-th bucket are merged to obtain a new i-th bucket, and the average true score, average model estimated score, bucket left boundary estimated score and bucket right boundary estimated score of the new i-th bucket are updated;

[0057] When the monotonicity is selected as monotonically decreasing, all buckets in the updated bucket sequence are traversed in turn in a loop. If the average true score of the i+1th bucket is greater than the average true score of the i-th bucket, the i+1th bucket and the i-th bucket are merged to obtain a new i-th bucket, and the average true score, average model estimated score, bucket left boundary estimated score and bucket right boundary estimated score of the new i-th bucket are updated.

[0058] In some embodiments, the monotonicity of the bucket sequence can be adjusted to be monotonically increasing or monotonically decreasing, as long as the average estimated score and the average true score of the bucket sequence are adjusted to the same monotonicity, so as to facilitate the establishment of a simple linear mapping relationship.

[0059] Furthermore, the monotonicity of the average model estimated scores and the average true scores of the multiple buckets in the updated bucket sequence corresponding to the material type is determined based on the preset mapping relationship structure by fitting each parameter value in the mapping relationship structure to obtain the target mapping relationship, including:

[0060] For each bucket in the updated bucket sequence corresponding to the material type, obtain a bucket left boundary estimated score and a bucket right boundary estimated score of the bucket;

[0061] Based on the monotonicity and the preset mapping relationship structure, determine a segmented mapping relationship from the average model estimated score of the bucket to the average true score within the range of the bucket left boundary estimated score and the bucket right boundary estimated score of the bucket;

[0062] The target mapping relationship is formed by segmented mapping relationships within the range of the estimated bucket left boundary scores and the estimated bucket right boundary scores of all buckets in the updated bucket sequence corresponding to the material type.

[0063] In some embodiments, a preset mapping relationship structure provides a mapping relationship formula with pending parameters, and the pending parameters need to be determined by fitting. Based on the preset mapping relationship structure, the segmented mapping relationship corresponding to the bucket is fitted within the range of the bucket left boundary estimated score and the bucket right boundary estimated score of each bucket. The value range of the model estimated score in the segmented mapping relationship corresponding to the bucket is within the range of the bucket left boundary estimated score and the bucket right boundary estimated score. The segmented mapping relationships corresponding to each bucket in the bucket sequence are combined together to form a target mapping relationship. The embodiment of the present invention can establish segmented mapping relationships for each bucket in the range of the bucket left boundary estimated score and the bucket right boundary estimated score, respectively, and can achieve higher calculation accuracy at a small calculation cost.

[0064] Furthermore, the preset mapping relationship structure is a linear mapping relationship structure.

[0065] Second, as Figure 2 As shown, an embodiment of the present invention provides a mixed content selection device, including:

[0066] The model estimation score acquisition unit 200 is used to acquire a plurality of materials belonging to at least two material types and a model estimation score corresponding to each material; the model estimation score represents the predicted popularity of the material; the model estimation score is predicted by an estimation score model corresponding to the material type; different material types correspond to estimation score models in different business scenarios, the estimation score models are trained based on real scores in the corresponding business scenarios, and the real scores represent the real popularity of the material;

[0067] A candidate material selection unit 201 is used to select a preset number of materials as candidate materials for each material type according to the model estimated score of each material under the material type;

[0068] A calibration unit 202 is used to perform a unified calibration process on the model estimation score corresponding to each candidate material to obtain a calibration score corresponding to each candidate material; the calibration score represents the calibration popularity of the material obtained after unified calibration for different material types;

[0069] The mixing unit 203 is used to select a preset target number of candidate materials with the highest calibration scores to form a mixed content according to the calibration scores of all candidate materials.

[0070] Furthermore, the calibration unit is specifically used for:

[0071] For each candidate material, the model estimated score of the candidate material is mapped to the calibration score of the candidate material according to a pre-constructed estimated calibration mapping relationship corresponding to the material type of the candidate material.

[0072] Furthermore, the device further comprises: an estimated calibration mapping relationship construction unit;

[0073] The estimated calibration mapping relationship construction unit includes:

[0074] A training material set acquisition module, used to acquire a material set for building an estimated calibration mapping relationship, wherein a plurality of sample materials in the material set belong to at least two material types;

[0075] A training material score acquisition module is used to obtain the model estimated score and the actual score of each sample material in the material set;

[0076] A mapping relationship fitting module is used to obtain, for each material type, a target mapping relationship from the model estimated score to the actual score based on the model estimated score and the actual score of all sample materials under the material type and based on a preset mapping relationship structure fitting, and use the fitted target mapping relationship as the estimated calibration mapping relationship corresponding to the material type.

[0077] Furthermore, the mapping relationship fitting module includes:

[0078] The material sorting and bucketing module is used to sort all sample materials under any material type according to the model estimated score, and perform equidistant bucketing to obtain a bucket sequence corresponding to the material type consisting of multiple buckets;

[0079] A bucket score determination module, for calculating, for each bucket in the bucket sequence corresponding to the material type, an average model estimated score, an average true score, a bucket left boundary estimated score, and a bucket right boundary estimated score of all sample materials in the bucket;

[0080] A bucket monotonicity adjustment module is used to traverse all buckets in the bucket sequence corresponding to the material type using an order-preserving regression method and merge the buckets to obtain an updated bucket sequence corresponding to the material type, so that the average true score of all buckets in the updated bucket sequence has the same monotonicity as the average model estimated score;

[0081] A mapping relationship determination module is used to fit and determine the parameter values ​​in the mapping relationship structure based on the preset mapping relationship structure according to the monotonicity of the average model estimated scores and the average true scores of multiple buckets in the updated bucket sequence corresponding to the material type, so as to obtain the target mapping relationship.

[0082] Furthermore, the bucket monotonicity adjustment module is configured as follows:

[0083] When the monotonicity is selected as monotonically increasing, all buckets in the updated bucket sequence are traversed in turn in a loop. If the average true score of the i+1th bucket is less than the average true score of the i-th bucket, the i+1th bucket and the i-th bucket are merged to obtain a new i-th bucket, and the average true score, average model estimated score, bucket left boundary estimated score and bucket right boundary estimated score of the new i-th bucket are updated;

[0084] When the monotonicity is selected as monotonically decreasing, all buckets in the updated bucket sequence are traversed in turn in a loop. If the average true score of the i+1th bucket is greater than the average true score of the i-th bucket, the i+1th bucket and the i-th bucket are merged to obtain a new i-th bucket, and the average true score, average model estimated score, bucket left boundary estimated score and bucket right boundary estimated score of the new i-th bucket are updated.

[0085] Furthermore, the mapping relationship determination module includes:

[0086] A bucket boundary score determination module, for obtaining, for each bucket in the updated bucket sequence corresponding to the material type, a bucket left boundary estimated score and a bucket right boundary estimated score of the bucket;

[0087] A segmented mapping module, for determining a segmented mapping relationship from an average model estimated score of the bucket to an average true score within a range of a bucket left boundary estimated score and a bucket right boundary estimated score of the bucket based on the monotonicity and the preset mapping relationship structure;

[0088] The segmented mapping integration module is used to form the target mapping relationship from the segmented mapping relationships within the range of the estimated bucket left boundary scores and the estimated bucket right boundary scores of all buckets in the updated bucket sequence corresponding to the material type.

[0089] Furthermore, the preset mapping relationship structure is a linear mapping relationship structure.

[0090] The embodiments of the present invention are device embodiments corresponding one-to-one to the aforementioned mixed content selection method. The embodiments of the present invention can be understood based on the aforementioned embodiments of the mixed content selection method based on model calibration, and will not be described in detail here.

[0091] In a third aspect, an embodiment of the present invention provides a computer device, comprising at least one host, wherein the at least one host comprises a processor adapted to execute various instructions; and

[0092] A storage device is suitable for storing a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing any of the methods described above.

[0093] In a fourth aspect, an embodiment of the present invention provides a storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing any of the methods described above.

[0094] The above technical solution of the embodiment of the present invention is described in detail below in conjunction with specific application examples. For technical details not introduced during the implementation process, please refer to the relevant description in the previous text.

[0095] In a mixed recommendation system for multi-type content, each type of material has a corresponding independent model to estimate the original estimated score. The estimated scores of different models for different types of materials need to be calibrated uniformly in a unified scenario in order to make a more accurate and reasonable comparison and improve distribution efficiency. Figure 3 As shown, the input is different types of materials (such as Figure 3 materials of concern, community materials and non-materials of concern, etc.), through the corresponding independent preset scoring models (such as Figure 3 The concerned material model, community material model and non-concerned material model in the model are used for online reasoning to obtain the model estimation score. After the model estimation scores are sorted, the third preset number (topN) of materials of various material types are taken to enter the online model calibration link for model calibration. The calibration scores after model calibration are compared and inserted by position to obtain the final sorting result.

[0096] The main steps are as follows:

[0097] 1. Bucketing: For the estimated sub-models corresponding to each material type, the material exposure records and model estimated scores corresponding to each estimated sub-model in the offline statistical exposure records are as follows: Figure 4As shown, according to the model estimated score (in Figure 4 The corresponding horizontal axis in the figure) is sorted and divided into equidistant buckets to ensure that the number of items recorded in each bucket is the same, for example, there are C items in each bucket; after the buckets are divided, the average real score of the items in the bucket is calculated, such as the average value of the interaction rate, click rate, etc., that is, Figure 4 The corresponding ordinate in (the average real score of the material, i.e. the actual quality score). For bucket i, the average model estimated score in the bucket is obtained. Average true rating Bucket left boundary value Bucket right boundary value Among them, PCTR represents the model estimated score of the material in the barrel; PCTR i represents the average model estimated score of the i-th bucket; CTR represents the actual score of the materials in the bucket; CTR i represents the average true score of bucket i; C i Represents the number of materials in the i-th bucket; L i Represents the minimum value of the model estimated score of the material in the i-th bucket; R i Represents the maximum value of the model estimated score for the material in the i-th bucket.

[0098] 2. Bucket order-preserving regression: Figure 5 As shown in the figure, for the N buckets sorted by the model estimated scores, the rank-preserving regression method is used to convert them into a sequence of buckets with ordered true scores. The specific process is as follows: traverse from left to right, for bucket i+1, if its corresponding average true score is less than the average true score corresponding to bucket i, that is, CTR i+1 <CTR i , it is considered to be in reverse order, and bucket i+1 is merged with bucket i. The specific merging process is as follows:

[0099] Bucket left boundary L new =L i , the right boundary of the barrel R new =R i+1 , C new =C i+1 +C i Among them, L new The left boundary of the bucket after the i+1th and ith buckets are merged; R new The right boundary of the bucket after merging the i+1th and ith buckets; C new is the number of materials in the barrel after the i+1th and ith barrels are combined;

[0100] Average estimated value of the merged buckets Average true score of the merged buckets

[0101] Repeat the above steps until the merged buckets meet the real score order, and end this round of merging. The number of buckets after order-preserving regression is N new ≤N, that is, the order-preserving regression may reduce the number of original buckets.

[0102] 3. Linear interpolation: such as Figure 6 As shown, the essence of model calibration is to find a mapping to convert the original model estimated score PCTR of the material into a calibration score ctr that reflects the real score in a unified scenario. After the order-preserving regression link, the real score CTR in the bucket is linearly related to the original model estimated score PCTR in the bucket, that is, as the original model estimated score PCTR increases, the corresponding real score CTR also increases. Therefore, a linear formula is used to simulate the mapping function from the original estimated score to the calibration score that reflects the real score in a unified scenario. The specific method is: for bucket i, that is, the model estimated score is at the left boundary L of bucket i i With right boundary R i Within the range, parameters Parameter b = CTR i -a*PCTR i , the final mapping function is ctr=a*pctr+b, where ctr is the calibration score of the material after unified calibration, and pctr is the model estimated score of the material.

[0103] Different materials are trained through different samples and models, so it is necessary to build an independent mapping system for each model. After the construction is completed, every time a user requests, different materials are inferred online through the corresponding estimated score model to obtain the original model estimated score. After mapping by the mapping system, the calibrated score that reflects the real score after unified calibration is obtained. Because the distribution effect in the Weibo relationship flow scenario is different from that in the recommendation flow, in the Weibo relationship flow scenario, in order to ensure effectiveness, the click-through rate and interaction rate of the materials have serious long-tail distribution, that is, there are many materials with high effectiveness, resulting in low interaction rate and click-through rate. In the Weibo recommendation flow scenario, the distributed materials are The materials have high interaction rates and click-through rates, so the estimated scores of the same material (sample) in the two scenarios are different; the embodiment of the present invention takes the efficiency of the material in the relationship flow scenario as the standard, and uniformly maps the original model estimated scores according to the standard of the relationship flow, that is, mapping based on a unified scenario to obtain comparable calibration scores that reflect the real scores in the unified scenario. The data collection, calculation, and establishment of the mapping system of the embodiment of the present invention are all based on a unified scenario, that is, the data collection, calculation, and establishment are all completed in the relationship flow scenario based on the user's attention relationship for content distribution. Therefore, the mapped calibration scores are comparable, such as Figure 7 As shown, the calibration points are sorted to obtain the final output sequence.

[0104] Since the model calibration is based on a unified scenario, the calibration points obtained after mapping are measured under a unified value system and are comparable. Therefore, the most logical approach for the sorting framework is to use bitwise filtering, that is, for each position, compare the top 1 of various current materials, select the material with the highest calibration score and put it into the current position until all positions are filled with corresponding materials. The materials already placed in the position sequence will be removed from the corresponding material type sequence.

[0105] The embodiment of the present invention has the following technical effects: a more reasonable and efficient comparison is made for multiple types of content, and a better sorting result is achieved. Compared with the use of a unified model for sorting different types of materials, the embodiment of the present invention uses independent models for training and online estimation of different types of materials, which can improve the prediction accuracy of each material; compared with the use of different models for estimation and then comparison and sorting of different types of materials, the embodiment of the present invention is based on the unified scene for model calibration conversion, and the different model estimation scores are converted into comparable real scene calibration scores, and the calibration score sorting system is used to perform efficient sorting and distribution. Compared with directly using the model estimation score for sorting and comparison, the calibration score obtained after model calibration (reflecting the real score in the unified scene) can better represent the benefits that the material can bring. In the embodiment of the present invention, a method combining model estimation score with real score is used to cover the entire sorting process: first, after obtaining the model estimation score, all materials are sorted according to the model estimation score and the third preset number of materials with the highest model estimation score are taken for subsequent model calibration process. After the model calibration, the sorting system based on the calibration score reflecting the real score in the unified scene is used for position insertion to obtain the final sorting result. In this way, on the one hand, the advantages of the two scores can be combined to reduce the deviation, and on the other hand, the amount of calculation can be reduced to improve the system sorting performance. Compared with the existing single model calibration technology, the method in the embodiment of the present invention constructs a model calibration system based on unified calibration for different models, and constructs different calibration systems for different models by counting offline data in the same scene. After mapping, comparable calibration scores that reflect the real score in the unified scene are obtained, which greatly increases the interpretability and rationality of the model and sorting results. The embodiment of the present invention constructs a unified model calibration system for different models of different types of materials. Different types of materials can improve their respective accuracy by predicting through independent models. At the same time, based on the score conversion system in the same scene, a more reasonable and efficient sorting basis can be provided to improve the overall sorting accuracy. By combining the original estimated score of the model with the calibration score that reflects the real score in the unified scene obtained after model calibration, a variety of information sources are integrated to reduce the influence of deviations in each link and improve the overall sorting system performance. By offline statistics of data records in the same scene, independent mapping systems can be constructed for different models, and comparable calibration scores that reflect the real score in the unified scene are obtained after processing through a unified mapping architecture, which improves the accuracy and interpretability of the sorting results.

[0106] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0107] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0108] The disclosed embodiments are described above to enable any person skilled in the art to implement or use the present invention. Various modifications of these embodiments are obvious to those skilled in the art, and the general principles defined herein may also be applied to other embodiments without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0109] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including". In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

[0110] Those skilled in the art may also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present invention.

[0111] The various illustrative logic blocks or units described in the embodiments of the present invention can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of the above. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0112] The steps of the method or algorithm described in the embodiments of the present invention can be directly embedded in hardware, a software module executed by a processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a user terminal. Optionally, the processor and the storage medium can also be arranged in different components in the user terminal.

[0113] In one or more exemplary designs, the above functions described in the embodiments of the present invention can be implemented in hardware, software, firmware or any combination of the three. If implemented in software, these functions can be stored on a computer-readable medium, or transmitted in the form of one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by any general or special computer. For example, such computer-readable media can include but are not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program codes in the form of instructions or data structures and other forms that can be read by general or special computers, or general or special processors. In addition, any connection can be appropriately defined as a computer-readable medium, for example, if the software is transmitted from a website site, server or other remote resource through a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disk and disc include compact disk, laser disk, optical disk, DVD, floppy disk and blue-ray disk. Disks usually copy data magnetically, while discs usually copy data optically with lasers. The above combination can also be included in computer readable media.

[0114] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for selecting mixed content, characterized in that: include: Acquire multiple materials belonging to at least two material types and a model estimated score corresponding to each material; the model estimated score represents the predicted popularity of the material; the model estimated score is predicted by an estimated score model corresponding to the material type; different material types correspond to estimated score models in different business scenarios, the estimated score model is trained based on the real score of the material in the corresponding business scenario, and the real score represents the real popularity of the material; For each material type, a preset number of candidate materials are selected as candidate materials according to the model estimated score of each material under the material type; The model estimation score corresponding to each candidate material is uniformly calibrated to obtain a calibration score corresponding to each candidate material; the calibration score represents the calibration popularity of the material obtained after uniform calibration for different material types; According to the calibration scores of all candidate materials, a preset target number of candidate materials with the highest calibration scores are selected to form the mixed arrangement content.

2. The method for selecting mixed content according to claim 1, characterized in that: The model estimated score corresponding to each candidate material is uniformly calibrated to obtain the calibration score corresponding to each candidate material, including: For each candidate material, the model estimated score of the candidate material is mapped to the calibration score of the candidate material according to a pre-constructed estimated calibration mapping relationship corresponding to the material type of the candidate material.

3. The method for selecting mixed content according to claim 1, wherein: The method for constructing the estimated calibration mapping relationship includes: Acquire a material set for constructing an estimated calibration mapping relationship, wherein a plurality of sample materials in the material set belong to at least two material types; For each sample material in the material set, obtain the model estimated score and the actual score of the sample material; For each material type, according to the model estimated scores and actual scores of all sample materials under the material type, a target mapping relationship from the model estimated scores to the actual scores is fitted based on a preset mapping relationship structure, and the fitted target mapping relationship is used as the estimated calibration mapping relationship corresponding to the material type.

4. The method for selecting mixed content according to claim 3, characterized in that: For each material type, according to the model estimated scores and the actual scores of all sample materials under the material type, a target mapping relationship from the model estimated score to the actual score is obtained based on a preset mapping relationship structure fitting, including: For all sample materials under any material type, all sample materials are sorted according to the model estimated scores, and are divided into buckets at equal intervals to obtain a bucket sequence corresponding to the material type consisting of multiple buckets; For each bucket in the bucket sequence corresponding to the material type, calculate the average model estimated score, the average true score, the bucket left boundary estimated score, and the bucket right boundary estimated score of all sample materials in the bucket; For all the buckets in the bucket sequence corresponding to the material type, an order-preserving regression method is used to traverse all the buckets and merge the buckets to obtain an updated bucket sequence corresponding to the material type, so that the average true score of all the buckets in the updated bucket sequence has the same monotonicity as the average model estimated score; According to the monotonicity of the average model estimated scores and the average true scores of multiple buckets in the updated bucket sequence corresponding to the material type, based on the preset mapping relationship structure, the parameter values ​​in the mapping relationship structure are fitted and determined to obtain the target mapping relationship.

5. The method for selecting mixed content according to claim 4, characterized in that: The method of traversing all the buckets in the bucket sequence corresponding to the material type and merging the buckets by using the order-preserving regression method to obtain an updated bucket sequence corresponding to the material type, so that the average true score of all the buckets in the updated bucket sequence has the same monotonicity as the average model estimated score, includes: When the monotonicity is selected as monotonically increasing, all buckets in the updated bucket sequence are traversed in turn in a loop. If the average true score of the i+1th bucket is less than the average true score of the i-th bucket, the i+1th bucket and the i-th bucket are merged to obtain a new i-th bucket, and the average true score, average model estimated score, bucket left boundary estimated score and bucket right boundary estimated score of the new i-th bucket are updated; When the monotonicity is selected as monotonically decreasing, all buckets in the updated bucket sequence are traversed in turn in a loop. If the average true score of the i+1th bucket is greater than the average true score of the i-th bucket, the i+1th bucket and the i-th bucket are merged to obtain a new i-th bucket, and the average true score, average model estimated score, bucket left boundary estimated score and bucket right boundary estimated score of the new i-th bucket are updated.

6. The method for selecting mixed content according to claim 4, characterized in that: The method of fitting and determining the parameter values ​​in the mapping relationship structure based on the monotonicity of the average model estimated scores and the average true scores of the multiple buckets in the updated bucket sequence corresponding to the material type and the preset mapping relationship structure to obtain the target mapping relationship includes: For each bucket in the updated bucket sequence corresponding to the material type, obtain a bucket left boundary estimated score and a bucket right boundary estimated score of the bucket; Based on the monotonicity and the preset mapping relationship structure, determine a segmented mapping relationship from the average model estimated score of the bucket to the average true score within the range of the bucket left boundary estimated score and the bucket right boundary estimated score of the bucket; The target mapping relationship is formed by segmented mapping relationships within the range of the estimated bucket left boundary scores and the estimated bucket right boundary scores of all buckets in the updated bucket sequence corresponding to the material type.

7. The method for selecting mixed content according to claim 3, characterized in that: The preset mapping relationship structure is a linear mapping relationship structure.

8. A mixed content selection device, characterized in that: include: A model estimation score acquisition unit is used to acquire a plurality of materials belonging to at least two material types and a model estimation score corresponding to each material; the model estimation score represents the predicted popularity of the material; the model estimation score is predicted by an estimation score model corresponding to the material type; different material types correspond to estimation score models in different business scenarios, the estimation score model is trained based on the real score of the material in the corresponding business scenario, and the real score represents the real popularity of the material; A candidate material selection unit, for selecting a preset number of materials as candidate materials for each material type according to the model estimated score of each material under the material type; A calibration unit, used to perform a unified calibration process on the model estimated score corresponding to each candidate material to obtain a calibration score corresponding to each candidate material; The calibration score represents the calibration popularity of the material obtained after uniform calibration for different material types; The mixing unit is used to select a preset target number of candidate materials with the highest calibration scores to form mixed content according to the calibration scores of all candidate materials.

9. A computer device, characterized in that: The device comprises at least one host, wherein the at least one host comprises a processor adapted to execute various instructions; and A storage device, adapted to store a plurality of instructions, wherein the instructions are adapted to be loaded by a processor and execute the method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: A plurality of instructions are stored therein, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 7.

Images