User interest quality evaluation method and device, storage medium and electronic equipment
By collecting historical and real-world behavioral data in user interest assessment, and generating and comparing the normalized depreciation cumulative gain of interest points and recommended content sequences, the high cost and low efficiency of interest point assessment in existing technologies are solved, enabling high-frequency, low-cost interest model verification and recommendation system optimization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING QIYI CENTURY SCI & TECH CO LTD
- Filing Date
- 2026-02-05
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies for evaluating the quality of user interest points suffer from high evaluation costs, long cycles, complex cross-team collaboration, and the inability to verify the effects in a timely manner after the interest model is updated. This results in a significant gap between interest points and recommendation results, making it difficult to achieve rapid and low-cost evaluation.
By collecting historical behavioral data of users within the first time window, interest point sequences are generated using old and new interest models, and then converted into recommended content sequences through a preset recall mechanism. In the second time window, real behavioral data is collected, and reference content sequences are generated based on viewing time. The normalized depreciation cumulative gain of the recommended content sequences is calculated and compared to determine the merits of the new interest model.
It enables high-frequency, low-cost correlation verification between interest models and user behavior, improves the accuracy of recommendation systems and the efficiency of user profile iteration, and solves the problem of difficulty in directly associating interest points.
Smart Images
Figure CN122019883A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of multimedia intelligent processing technology, and in particular to a method, apparatus, storage medium, and electronic device for evaluating the quality of user interests. Background Technology
[0002] With the widespread application of personalized recommendation systems, user interests play a central role in generating recommendation results. For example, in film and television platforms, user preferences for celebrities, genres, or themes are typically represented by user profiles. However, existing technologies still have significant shortcomings in evaluating the quality of interest points. Current methods mainly rely on online A / B experiments or manual sampling evaluations to verify the accuracy of interest points. While these methods can reflect the recommendation effect to some extent, they suffer from high evaluation costs, long cycles, and complex cross-team collaboration. Furthermore, offline evaluation methods are usually based solely on interest data generated by the model or historical behavioral features, lacking a direct correlation with actual user behavior and failing to promptly verify the effect of updated interest points. Since interest data is an intermediate representation, it is difficult to directly map it to user viewing behavior, resulting in a significant gap between interest points and recommendation results, making rapid and low-cost evaluation of interest point quality difficult. At the same time, after updating the interest model, a considerable amount of time is often required to wait for actual feedback before judging the effect, which further limits the efficiency of recommendation algorithm iteration and user profile updates. Summary of the Invention
[0003] This application provides a method, apparatus, storage medium, and electronic device for evaluating the quality of user interests, in order to solve the technical problem that it is difficult to directly correlate points of interest with user behavior and that the evaluation cost is high.
[0004] Firstly, this application provides a method for evaluating the quality of user interests, comprising: collecting historical behavioral data of users within a first time window, and generating an old interest point sequence and a new interest point sequence based on the historical behavioral data using an old interest model and a new interest model, respectively; converting the old interest point sequence and the new interest point sequence into a first recommended content sequence and a second recommended content sequence predicting the user's interest preferences through a preset recall mechanism; collecting the user's actual behavioral data within a second time window, and quantifying the actual behavioral data based on viewing duration to generate a reference content sequence reflecting the user's actual interest preferences, wherein the second time window is located after the first time window in time; calculating the normalized cumulative gain corresponding to the first recommended content sequence and the second recommended content sequence based on the reference content sequence, and performing a comparative analysis to determine the superiority or inferiority of the new interest model relative to the old interest model.
[0005] Secondly, this application provides a device for evaluating the quality of user interests, comprising: a first generation module, used to collect historical behavioral data of users within a first time window, and generate an old interest point sequence and a new interest point sequence based on the historical behavioral data using an old interest model and a new interest model, respectively; a conversion module, used to convert the old interest point sequence and the new interest point sequence into a first recommended content sequence and a second recommended content sequence predicting the user's interest preferences through a preset recall mechanism; a second generation module, used to collect the actual behavioral data of users within a second time window, and quantify the actual behavioral data based on viewing time to generate a reference content sequence reflecting the user's actual interest preferences, wherein the second time window is located after the first time window in time; and a comparison module, used to calculate the normalized cumulative gain corresponding to the first recommended content sequence and the second recommended content sequence based on the reference content sequence, and perform comparative analysis to determine the superiority or inferiority of the new interest model relative to the old interest model.
[0006] As an optional example, the first generation module includes: a first generation unit, used to generate an old set of interest points and a new set of interest points based on the historical behavior data using the old interest model and the new interest model, respectively; and a first sorting unit, used to sort the old set of interest points and the new set of interest points according to a preset interest point sorting rule, and select the first few interest points of equal quantity from them as the old interest point sequence and the new interest point sequence, respectively.
[0007] As an optional example, the above conversion module includes: a first acquisition unit, used to invoke the above-preset recall mechanism for each interest point in the above-mentioned old interest point sequence and the above-mentioned new interest point sequence, to acquire a first candidate content set and a second candidate content set associated with the interest point respectively; and a second sorting unit, used to sort the above-mentioned first candidate content set and the above-mentioned second candidate content set according to a preset content sorting rule, and to select the first number of contents with the same quantity from them as the above-mentioned first recommended content sequence and the above-mentioned second recommended content sequence respectively.
[0008] As an optional example, the second generation module includes: a second acquisition unit, used to acquire the viewing duration corresponding to each content in the real behavior data; a third sorting unit, used to sort each content in the real behavior data in descending order based on the viewing duration to obtain a first content sequence; a mapping unit, used to map the viewing duration of each content in the first content sequence to a relevance weight value to form a second content sequence with relevance weight values; a second generation unit, used to select the first few contents from each recall result obtained from the multiple recall results triggered by all interest points based on the preset recall mechanism, and merge them to form a third candidate content set; and a fourth sorting unit, used to perform an intersection operation on the second content sequence and the third candidate content set, and sort them in reverse order according to the relevance weight values to generate the reference content sequence.
[0009] As an optional example, the second generation unit includes: an acquisition subunit, used to trigger the preset recall mechanism by using each point of interest in the preset recall mechanism as a recall keyword, to obtain a fourth candidate content set associated with each point of interest; a sorting subunit, used to sort each fourth candidate content set according to its relevance to its point of interest, and select the top few contents to form a corresponding fifth candidate content set; and a merging subunit, used to merge and deduplicate the fifth candidate content sets corresponding to each point of interest to form the third candidate content set.
[0010] As an optional example, the comparison module includes: a first calculation unit, used to calculate the ideal cumulative loss gain based on the relevance weight value and sorting position of each content in the reference content sequence; a second calculation unit, used to match the content in the first recommended content sequence with the reference content sequence respectively, and calculate the cumulative loss gain of the first recommended content sequence based on the position of the matched content in the first recommended content sequence and its corresponding relevance weight value; a third calculation unit, used to match the content in the second recommended content sequence with the reference content sequence respectively, and calculate the cumulative loss gain of the second recommended content sequence based on the position of the matched content in the second recommended content sequence and its corresponding relevance weight value; and a fourth calculation unit, used to normalize the cumulative loss gain of the first recommended content sequence and the second recommended content sequence with the ideal cumulative loss gain to obtain the corresponding normalized cumulative loss gain.
[0011] As an optional example, the comparison module includes: a first determining unit, configured to determine that the new interest model is superior to the old interest model when the normalized cumulative gain of the first recommended content sequence is less than the normalized cumulative gain of the second recommended content sequence; a second determining unit, configured to determine that the old interest model is superior to the new interest model when the normalized cumulative gain of the first recommended content sequence is greater than the normalized cumulative gain of the second recommended content sequence; and a third determining unit, configured to determine that the old interest model and the new interest model are of equal quality when the normalized cumulative gain of the first recommended content sequence is equal to the normalized cumulative gain of the second recommended content sequence.
[0012] Thirdly, this application provides a storage medium storing a computer program, wherein the computer program is executed by a processor to perform the aforementioned user interest quality evaluation method.
[0013] Fourthly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the aforementioned user interest quality evaluation method through the computer program.
[0014] The technical solutions provided in this application have the following advantages compared with the prior art: This application employs the following methods: collecting historical user behavior data within a first time window; generating old interest point sequences and new interest point sequences based on the historical behavior data using an old interest model and a new interest model, respectively; converting the old interest point sequences and new interest point sequences into a first recommended content sequence and a second recommended content sequence predicting the user's interest preferences using a preset recall mechanism; collecting the user's actual behavior data within a second time window; quantifying the actual behavior data based on viewing duration; and generating a reference content sequence reflecting the user's actual interest preferences, wherein the second time window is located after the first time window; and based on the reference... The method involves calculating the normalized cumulative gain (NCG) of the first and second recommended content sequences, respectively, and comparing them to determine the superiority of the new interest model over the old one. This method collects historical user behavior data within the first time window, generates interest point sequences using both the old and new interest models, and maps them to recommended content sequences through a pre-defined recall mechanism. Simultaneously, it collects real user behavior data within the second time window, quantifies it according to viewing duration and order to generate reference content sequences, calculates the NCG of the recommended content sequences, and compares them to evaluate the merits of the new and old interest models. This achieves high-frequency, low-cost correlation verification between interest models and real user behavior, effectively improving the accuracy of the recommendation system and the efficiency of user profile iteration. It also solves the technical problem of difficulty in directly correlating interest points with user behavior and the high cost of evaluation. Attached Figure Description
[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.
[0018] Figure 1 This is a flowchart of an optional method for evaluating the quality of user interests according to an embodiment of this application; Figure 2This is a flowchart illustrating the specific implementation of an optional user interest quality assessment method according to an embodiment of this application. Figure 3 This is a schematic diagram of an optional user interest quality assessment device according to an embodiment of this application; Figure 4 This is a schematic diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0020] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.
[0021] According to a first aspect of the embodiments of this application, a method for evaluating the quality of user interests is provided, optionally, as follows: Figure 1 As shown, the above method includes: S102, collect the user's historical behavior data within the first time window, and generate the old interest point sequence and the new interest point sequence based on the historical behavior data through the old interest model and the new interest model respectively; S104, through a preset recall mechanism, convert the old interest point sequence and the new interest point sequence into the first recommended content sequence and the second recommended content sequence for predicting user interest preferences, respectively; S106, Collect the user's real behavior data within the second time window, quantify the real behavior data according to the viewing duration, and generate a reference content sequence that reflects the user's real interests and preferences. The second time window is located after the first time window in time. S108. Based on the reference content sequence, calculate the normalized cumulative gain corresponding to the first recommended content sequence and the second recommended content sequence respectively, and conduct a comparative analysis to determine the superiority or inferiority of the new interest model over the old interest model.
[0022] Optionally, this embodiment provides a method for evaluating user interest quality, aiming to overcome the gap between traditional interest models and actual user behavior, achieve high-frequency, low-cost interest quality evaluation, and establish a multi-dimensional verification system, thereby providing technical support for the optimization of personalized recommendation systems. The specific implementation process is as follows: Figure 2 As shown, firstly, historical behavior data of users within a first time window is collected. This time window serves as the input data for generating the user interest model. After acquiring the historical behavior data, the old interest model and the new interest model are used to analyze and process the historical behavior data, generating corresponding old and new user interest point sequences. To ensure the accuracy and comparability of the evaluation, for each interest point sequence generated by the interest model, the top N interest points can be further selected as candidate sequences for subsequent analysis based on their importance or activity, thus avoiding interference from long-tail interests in the evaluation results.
[0023] Subsequently, through a pre-defined recall mechanism, the aforementioned old and new interest point sequences are mapped to a set of content that users can actually access. Specifically, each interest point serves as a recall keyword to trigger a recall channel, retrieving a list of candidate content associated with that interest point from the same target content library. Each candidate content list is then sorted based on its relevance score to the interest point, and a predetermined number of top-ranked content items are selected. All candidate content corresponding to all interest points is merged and deduplicated to obtain a first recommended content sequence and a second recommended content sequence, used to predict user interests and preferences.
[0024] After constructing the interest prediction sequence, real user behavior data is collected within a second time window. This second time window, located after the first time window, is used to verify the accuracy of the predicted interests. The collected real behavior data is quantified, mapping the user's viewing time for content to interest relevance weights, and applying decay processing based on the user's viewing order to generate a reference content sequence reflecting the user's true interest preferences. Furthermore, the reference content sequence is intersected with the entire set of recallable content to filter out content that has both been actually viewed and is in the candidate recommendation list. This content is then sorted according to its relevance weights to form the final reference sequence for evaluation. The entire set of recallable content is formed by selecting the top few items from each of the multiple recall results obtained from the preset recall mechanism triggered by all interest points in the target content library, and merging them together.
[0025] After constructing the reference sequence, the Normalized Discounted Cumulative Gain (NDCG) is calculated for both the first and second recommended content sequences. The calculation process involves first generating the iDCG (Indexed Cumulative Gain) index under ideal ranking based on the reference content sequence. Then, the actual DCG is calculated based on the matching relationship and corresponding weights of the recommended sequences within the reference sequence. Finally, the NDCG index is obtained by normalizing the DCG and iDCG. By comparing the NDCG values of the first and second recommended content sequences, the superiority of the new interest model over the old interest model in depicting users' true interest preferences can be determined.
[0026] Optionally, this embodiment achieves accurate offline evaluation of the interest model through time-dynamic window design, interest point pruning, recall mapping, real behavior quantification, and normalized loss cumulative gain calculation. It has the advantages of high frequency, low cost, and quantifiability, and can obtain effective feedback on the interest model in a short time. At the same time, it takes into account both the abstractness of interest points and the specificity of recommended content, solving the disconnect between interest data and user behavior.
[0027] As an optional example, generating old interest point sequences and new interest point sequences based on historical behavioral data using old interest models and new interest models respectively includes: Based on historical behavior data, the old interest model and the new interest model generate sets of old interest points and sets of new interest points, respectively. According to the preset interest point sorting rules, the old interest point set and the new interest point set are sorted respectively, and the top several interest points with the same number are selected as the old interest point sequence and the new interest point sequence respectively.
[0028] Optionally, in this embodiment, firstly, historical behavioral data of the user within a preset time window is collected. This historical behavioral data may include the user's browsing history, click behavior, viewing time, etc., to characterize basic information about the user's interests and preferences. Subsequently, the historical behavioral data is analyzed and processed using both old and new interest models. Based on the model algorithms, a set of old and new interest points corresponding to the user is generated. Each set contains multiple interest points, and each interest point corresponds to the user's preference intensity on a certain interest dimension. To ensure the comparability and accuracy of subsequent evaluations, the generated interest point sets are further sorted using preset interest point sorting rules. These sorting rules can be set based on the activity level of the interest points, the frequency of user behavior, or the interest weights predicted by the model. After sorting, the top few interest points of equal quantity are selected from each interest point set to form the final sequence of old and new interest points. This sequence retains the user's most representative interest information while filtering out low-activity or potentially noisy interest points to reduce interference with the interest evaluation results. In this way, the interest point sequences generated by the old interest model and the new interest model can remain consistent in terms of quantity and representativeness, providing a reliable data foundation for subsequent generation of recommended content sequences based on recall mechanisms, quantitative evaluation of user behavior, and comparative analysis of interest models.
[0029] As an optional example, by using a pre-defined recall mechanism, the old and new interest point sequences are respectively converted into a first recommended content sequence and a second recommended content sequence to predict user interest preferences, including: For each point of interest in the old and new interest sequences, a preset recall mechanism is invoked to obtain the first and second candidate content sets associated with that point of interest, respectively. According to the preset content sorting rules, the first candidate content set and the second candidate content set are sorted respectively, and the first few contents with the same number are selected as the first recommended content sequence and the second recommended content sequence respectively.
[0030] Optionally, in this embodiment, firstly, for each interest point in the old and new interest point sequences, a preset recall mechanism is invoked to retrieve a set of candidate content associated with that interest point from the same target content library. The recall mechanism can be based on collaborative filtering, content similarity, deep learning prediction models, or a combination of multiple recommendation algorithms to map interest points to recommendable content. After obtaining the candidate content set, the candidate content set corresponding to each interest point is sorted. The sorting rules can be set according to the relevance score between the content and the interest point, the activity level of the content, the user's historical preferences, or a preset recommendation strategy to ensure that content more in line with the user's interests is ranked higher. After sorting, the top few items of equal quantity are selected from each candidate content set to form the final first and second recommended content sequences. These sequences are used to predict the content that the user may prefer in future time windows. The recommended content sequences generated by the old and new interest point sequences maintain consistency in quantity, sorting rules, and candidate content selection, thereby ensuring the fairness and accuracy of subsequent comparisons of interest model performance. In addition, to improve the coverage and diversity of recommended content, the candidate content sets corresponding to different points of interest can be deduplicated and merged to form a complete set of recommended content that can be used for evaluation.
[0031] As an optional example, quantifying real-world behavioral data based on viewing duration to generate a sequence of reference content reflecting users' true interests and preferences includes: Obtain the viewing duration corresponding to each piece of content in the real behavioral data; Based on viewing time, the content in the real behavior data is sorted in descending order to obtain the first content sequence; The viewing time of each content in the first content sequence is mapped to a relevance weight value to form a second content sequence with relevance weight values; From the multiple recall results obtained by the preset recall mechanism triggered by all points of interest based on the preset recall mechanism, select the top few contents from each recall result and merge them to form a third candidate content set. The intersection of the second content sequence and the third candidate content set is performed, and the sequence is sorted in reverse order according to the relevance weight values to generate a reference content sequence.
[0032] Optionally, in this embodiment, firstly, the user's actual behavior data within a second time window is acquired, and the viewing duration corresponding to each piece of content is extracted. The viewing duration can be used to measure the intensity of the user's interest in the content. Subsequently, the content in the actual behavior data is sorted in descending order according to the viewing duration to obtain a preliminary sorted first content sequence, which reflects the user's actual attention to each piece of content. Next, the viewing duration of each piece of content in the first content sequence is mapped to a relevance weight value to form a second content sequence with a relevance weight value. This relevance weight is used for subsequent matching and evaluation with the recommended content sequence.
[0033] Simultaneously, the entire set of recallable content is identified as the third candidate content set, thus constructing a complete set of candidate content for evaluation. To ensure the relevance and fairness of the evaluation, the intersection of the second content sequence and the third candidate content set is calculated to filter out content that has been genuinely viewed and is in the recall set, excluding content that has not been reached by the recommendation system or is irrelevant. Subsequently, the intersection content is sorted in reverse order according to its relevance weight in the second content sequence to generate the final reference content sequence. This sequence not only reflects the strength and preference order of users' true interests but also maintains consistency with the content space accessible by the recommendation system.
[0034] Optionally, this embodiment can transform abstract user behavior data into a quantitative reference sequence that can be used for interest model evaluation, providing a reliable basis for calculating the normalized depreciation cumulative gain (NDCG) of the recommended content sequence. This achieves direct alignment between interest model prediction results and actual user behavior, making interest quality evaluation more accurate and quantifiable.
[0035] As an optional example, from the multiple recall results obtained by the preset recall mechanism triggered by all points of interest based on the preset recall mechanism, the top few items from each recall result are selected and merged to form a third candidate content set, including: Using each point of interest in the preset recall mechanism as a recall keyword, the preset recall mechanism is triggered to obtain a fourth set of candidate content associated with each point of interest. For each set of fourth candidate content, sort them according to their relevance to their points of interest, and select the top few content items to form the corresponding set of fifth candidate content. The fifth candidate content sets corresponding to each point of interest are merged and deduplicated to form the third candidate content set.
[0036] Optionally, in this embodiment, firstly, each interest point in the preset recall mechanism is used as a recall keyword to trigger the preset recall mechanism, thereby obtaining a fourth set of candidate content associated with each interest point from the target content library. The recall mechanism may include methods based on collaborative filtering, content similarity calculation, deep learning model prediction, or a combination of multiple recommendation algorithms to map user interest points to actual recommendable content, ensuring the relevance and coverage of the recalled content.
[0037] Subsequently, for each point of interest, the fourth candidate content set is sorted according to the relevance between the content and the point of interest. The sorting rules can be set by combining the matching score between the content and the point of interest, the weight of users' historical behavior preferences, and the activity level or recommendation strategy of the content itself. After sorting, the top few contents from each fourth candidate content set are selected to form the corresponding fifth candidate content set, to ensure that representative content for each point of interest is fully preserved, while avoiding interference from low-relevance or long-tail content in the evaluation results.
[0038] Finally, the fifth candidate content sets corresponding to each interest point are merged, and the merged content is deduplicated to generate the final third candidate content set. This set includes not only the most representative candidate content for user interest points but also covers the complete content space that can be used for interest model evaluation, providing a reliable data foundation for subsequent intersection operations of reference content sequences and quality evaluation of recommendation sequences.
[0039] As an optional example, based on the reference content sequence, the normalized cumulative gain of the first recommended content sequence and the second recommended content sequence are calculated respectively, including: Based on the relevance weight values and sorting positions of each content in the reference content sequence, the ideal cumulative gain of loss is calculated. The content in the first recommended content sequence is matched with the reference content sequence, and the cumulative gain of the first recommended content sequence is calculated based on the position of the matched content in the first recommended content sequence and its corresponding relevance weight value. The content in the second recommended content sequence is matched with the reference content sequence, and the cumulative gain of the second recommended content sequence is calculated based on the position of the matched content in the second recommended content sequence and its corresponding relevance weight value. The cumulative loss gain and the ideal cumulative loss gain of the first recommended content sequence and the second recommended content sequence are normalized respectively to obtain the corresponding normalized cumulative loss gain.
[0040] Optionally, in this embodiment, firstly, the ideal depreciation cumulative gain (iDCG) is calculated based on the relevance weight values of each content in the reference content sequence and their ranking position in the sequence. The ideal depreciation cumulative gain reflects the cumulative interest value when the recommended content perfectly matches the user's true interests under the optimal ranking, providing a standard for subsequent normalization processing.
[0041] Subsequently, each piece of content in the first recommended content sequence is matched with the reference content sequence to determine its position within the reference sequence and its corresponding relevance weight. Based on the ranking position and relevance weight of the matched content in the recommended sequence, the discounted cumulative gain (DCG) of the first recommended content sequence is calculated using the discounted cumulative gain formula. Similarly, the content in the second recommended content sequence is matched with the reference content sequence, and its discounted cumulative gain is calculated based on its position and corresponding relevance weight.
[0042] After calculating the cumulative loss gain, the cumulative loss gains of the first and second recommended content sequences are normalized to the ideal cumulative loss gain to obtain the corresponding Normalized Cumulative Loss Gain (NDCG). Normalization eliminates the influence of different recommendation sequence lengths and content quantities on the evaluation results, allowing for a fair comparison of recommended content sequences generated by different interest models. By comparing the NDCG values of the first and second recommended content sequences, the accuracy and merits of the new and old interest models in predicting users' true interest preferences can be quantitatively evaluated.
[0043] As an optional example, a comparative analysis to determine the merits of the new interest model relative to the old interest model includes: If the normalized cumulative gain of the first recommended content sequence is less than the normalized cumulative gain of the second recommended content sequence, the new interest model is determined to be superior to the old interest model. If the normalized cumulative gain of the first recommended content sequence is greater than the normalized cumulative gain of the second recommended content sequence, the old interest model is determined to be superior to the new interest model. If the normalized cumulative gain of the first recommended content sequence is equal to the normalized cumulative gain of the second recommended content sequence, then the old interest model and the new interest model are determined to be of equal quality.
[0044] Optionally, in this embodiment, after calculating the NDCG of the first and second recommended content sequences, the normalized depreciation cumulative gain of the two sequences is directly compared. If the NDCG of the first recommended content sequence is less than the NDCG of the second recommended content sequence, it indicates that the recommended content sequence generated by the new interest model matches the user's actual interest preferences better than the old interest model, thus determining that the new interest model is superior to the old interest model. Conversely, if the NDCG of the first recommended content sequence is greater than the NDCG of the second recommended content sequence, it indicates that the old interest model performs better in predicting user interests than the new interest model, thus determining that the old interest model is superior to the new interest model.
[0045] When the NDCG values are equal, it indicates that the recommended content sequences generated by the old and new interest models do not differ significantly in their matching degree with users' actual interests and preferences. Therefore, it can be concluded that the two models are of equal quality. This comparative analysis not only quantifies the effectiveness of interest models but also clarifies the relative merits of different models in real-world recommendation scenarios, providing a scientific basis for subsequent model iteration and optimization.
[0046] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.
[0047] According to another aspect of the embodiments of this application, a device for evaluating the quality of user interests is also provided, such as... Figure 3 As shown, it includes: The first generation module 302 is used to collect historical behavior data of users within the first time window, and generate old interest point sequences and new interest point sequences based on the historical behavior data through old interest model and new interest model respectively. The conversion module 304 is used to convert the old interest point sequence and the new interest point sequence into a first recommended content sequence and a second recommended content sequence that predicts the user's interest preferences, respectively, through a preset recall mechanism. The second generation module 306 is used to collect the user's real behavior data within the second time window, quantify the real behavior data according to the viewing duration, and generate a reference content sequence that reflects the user's real interests and preferences. The second time window is located after the first time window in time. The comparison module 308 is used to calculate the normalized cumulative gain of the first recommended content sequence and the second recommended content sequence according to the reference content sequence, and to perform a comparative analysis to determine the superiority or inferiority of the new interest model over the old interest model.
[0048] It should be noted that the first generation module 302 in this embodiment can be used to execute step S102 in this application embodiment, the conversion module 304 in this embodiment can be used to execute step S104 in this application embodiment, the second generation module 306 in this embodiment can be used to execute step S106 in this application embodiment, and the comparison module 308 in this embodiment can be used to execute step S108 in this application embodiment.
[0049] As an optional example, the first generation module includes: The first generation unit is used to generate sets of old interest points and sets of new interest points based on historical behavior data, respectively, using old interest models and new interest models. The first sorting unit is used to sort the old interest point set and the new interest point set according to the preset interest point sorting rules, and select the first few interest points of equal quantity from them as the old interest point sequence and the new interest point sequence, respectively.
[0050] As an optional example, the conversion module includes: The first acquisition unit is used to invoke a preset recall mechanism for each interest point in the old interest point sequence and the new interest point sequence to obtain the first candidate content set and the second candidate content set associated with the interest point, respectively. The second sorting unit is used to sort the first candidate content set and the second candidate content set according to the preset content sorting rules, and select the first few contents with the same number from them as the first recommended content sequence and the second recommended content sequence.
[0051] As an optional example, the second generation module includes: The second acquisition unit is used to acquire the viewing duration corresponding to each content in the real behavior data; The third sorting unit is used to sort the content in the real behavior data in descending order based on the viewing time to obtain the first content sequence; The mapping unit is used to map the viewing time of each content in the first content sequence to a relevance weight value, forming a second content sequence with relevance weight values; The second generation unit is used to select the top few contents from each recall result obtained from the multiple recall results triggered by the preset recall mechanism based on all points of interest, and merge them to form a third candidate content set. The fourth sorting unit is used to perform an intersection operation on the second content sequence and the third candidate content set, and sort them in reverse order according to the relevance weight value to generate a reference content sequence.
[0052] As an optional example, the second generation unit includes: The sub-unit is used to trigger the preset recall mechanism by using each point of interest in the preset recall mechanism as a recall keyword, so as to obtain the fourth set of candidate content associated with each point of interest. The sorting subunit is used to sort each fourth candidate content set according to its relevance to its points of interest, and select the top few contents to form the corresponding fifth candidate content set. The merging sub-unit is used to merge and deduplicate the fifth candidate content sets corresponding to each point of interest to form the third candidate content set.
[0053] As an optional example, the comparison modules include: The first calculation unit is used to calculate the ideal cumulative gain of loss based on the relevance weight value and sorting position of each content in the reference content sequence. The second calculation unit is used to match the contents in the first recommended content sequence with the reference content sequence respectively, and calculate the cumulative loss gain of the first recommended content sequence based on the position of the matched content in the first recommended content sequence and its corresponding relevance weight value. The third calculation unit is used to match the contents in the second recommended content sequence with the reference content sequence respectively, and calculate the cumulative gain of the second recommended content sequence based on the position of the matched content in the second recommended content sequence and its corresponding relevance weight value. The fourth calculation unit is used to normalize the cumulative loss gain and the ideal cumulative loss gain of the first recommended content sequence and the second recommended content sequence, respectively, to obtain the corresponding normalized cumulative loss gain.
[0054] As an optional example, the comparison modules include: The first determining unit is used to determine that the new interest model is superior to the old interest model when the normalized cumulative gain of the first recommended content sequence is less than the normalized cumulative gain of the second recommended content sequence. The second determining unit is used to determine that the old interest model is superior to the new interest model when the normalized cumulative gain of the first recommended content sequence is greater than the normalized cumulative gain of the second recommended content sequence. The third determining unit is used to determine that the old interest model and the new interest model are equally good or bad when the normalized cumulative gain of the first recommended content sequence is equal to the normalized cumulative gain of the second recommended content sequence.
[0055] For other examples of this embodiment, please refer to the examples above, which will not be repeated here.
[0056] Figure 4 This is a schematic diagram of an optional electronic device according to an embodiment of this application, such as... Figure 4 As shown, it includes a processor 402, a communication interface 404, a memory 406, and a communication bus 408. The processor 402, communication interface 404, and memory 406 communicate with each other via the communication bus 408. Memory 406 is used to store computer programs; When processor 402 executes a computer program stored in memory 406, it performs the following steps: Collect users' historical behavior data within the first time window, and generate old interest point sequences and new interest point sequences based on the historical behavior data using old interest models and new interest models respectively; By using a pre-set recall mechanism, the old interest point sequence and the new interest point sequence are respectively converted into the first recommended content sequence and the second recommended content sequence to predict user interest preferences; Collect real behavioral data of users within the second time window, quantify the real behavioral data according to the viewing time, and generate a reference content sequence that reflects the user's real interests and preferences. The second time window is located after the first time window in time. Based on the reference content sequence, the normalized cumulative gain of the first and second recommended content sequences is calculated and compared to determine the merits of the new interest model relative to the old interest model.
[0057] Optionally, in this embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic devices and other devices.
[0058] The memory may include RAM, or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0059] As an example, the memory 406 described above may include, but is not limited to, the first generation module 302, the conversion module 304, the second generation module 306, and the comparison module 308 from the user interest quality evaluation device described above. Furthermore, it may include, but is not limited to, other module units from the user interest quality evaluation device described above, which will not be elaborated upon in this example.
[0060] The processor mentioned above can be a general-purpose processor, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0061] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.
[0062] Those skilled in the art will understand that Figure 4 The structure shown is for illustrative purposes only. The device used to implement the above-described method for evaluating the quality of user interest can be a terminal device, such as a smartphone (e.g., an Android phone, an iOS phone), a tablet computer, a PDA, a mobile internet device (MID), a PAD, or other terminal devices. Figure 4 This does not limit the structure of the aforementioned electronic devices. For example, the electronic device may also include components that are more... Figure 4 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 4 The different configurations shown.
[0063] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.
[0064] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, which, when executed by a processor, performs the steps in the above-described method for evaluating the quality of user interests.
[0065] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0066] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0067] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.
[0068] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0069] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.
[0070] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0071] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0072] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A method for evaluating the quality of user interest, characterized in that, include: Collect users' historical behavior data within the first time window, and generate old interest point sequences and new interest point sequences based on the historical behavior data using old interest models and new interest models, respectively. By using a preset recall mechanism, the old interest point sequence and the new interest point sequence are respectively converted into a first recommended content sequence and a second recommended content sequence to predict the user's interest preferences; Collect the user's real behavior data within the second time window, quantify the real behavior data according to the viewing duration, and generate a reference content sequence that reflects the user's real interests and preferences, wherein the second time window is located after the first time window in time; Based on the reference content sequence, the normalized cumulative gain of the first recommended content sequence and the second recommended content sequence is calculated respectively, and a comparative analysis is performed to determine the superiority or inferiority of the new interest model over the old interest model.
2. The method according to claim 1, characterized in that, Based on the historical behavior data, the old interest model and the new interest model are used to generate old interest point sequences and new interest point sequences, respectively, including: Based on the historical behavior data, the old interest model and the new interest model respectively generate an old interest point set and a new interest point set; According to the preset interest point sorting rules, the old interest point set and the new interest point set are sorted respectively, and the first few interest points with the same number are selected as the old interest point sequence and the new interest point sequence respectively.
3. The method according to claim 1, characterized in that, By using a preset recall mechanism, the old interest point sequence and the new interest point sequence are respectively converted into a first recommended content sequence and a second recommended content sequence for predicting the user's interest preferences, including: For each point of interest in the old and new interest sequences, the preset recall mechanism is invoked to obtain the first and second candidate content sets associated with that point of interest, respectively. According to the preset content sorting rules, the first candidate content set and the second candidate content set are sorted respectively, and the first few items with the same number are selected as the first recommended content sequence and the second recommended content sequence respectively.
4. The method according to claim 1, characterized in that, The real behavioral data is quantified based on viewing time to generate a reference content sequence reflecting the user's true interests and preferences, including: Obtain the viewing duration corresponding to each content in the real behavior data; Based on viewing duration, the content in the real behavior data is sorted in descending order to obtain the first content sequence; The viewing time of each content in the first content sequence is mapped to a relevance weight value to form a second content sequence with relevance weight values; From the multiple recall results obtained by the preset recall mechanism triggered by all points of interest based on the preset recall mechanism, select the top several contents from each recall result and merge them to form a third candidate content set. The second content sequence and the third candidate content set are intersected, and then sorted in reverse order according to the relevance weight values to generate the reference content sequence.
5. The method according to claim 4, characterized in that, From the multiple recall results obtained by the preset recall mechanism triggered by all points of interest, the top few items from each recall result are selected and merged to form a third candidate content set, including: Using each point of interest in the preset recall mechanism as a recall keyword, the preset recall mechanism is triggered to obtain a fourth set of candidate content associated with each point of interest. For each set of fourth candidate content, sort them according to their relevance to their points of interest, and select the top few content items to form the corresponding set of fifth candidate content. The fifth candidate content sets corresponding to each point of interest are merged and deduplicated to form the third candidate content set.
6. The method according to claim 4, characterized in that, Based on the reference content sequence, the normalized cumulative gain of the first recommended content sequence and the second recommended content sequence are calculated respectively, including: Based on the relevance weight values and sorting positions of each content in the reference content sequence, the ideal cumulative gain of loss is calculated. The content in the first recommended content sequence is matched with the reference content sequence respectively, and the cumulative gain of the first recommended content sequence is calculated based on the position of the matched content in the first recommended content sequence and its corresponding relevance weight value. The content in the second recommended content sequence is matched with the reference content sequence respectively, and the cumulative gain of the second recommended content sequence is calculated based on the position of the matched content in the second recommended content sequence and its corresponding relevance weight value. The cumulative loss gain of the first recommended content sequence and the second recommended content sequence are normalized to the ideal cumulative loss gain to obtain the corresponding normalized cumulative loss gain.
7. The method according to any one of claims 1 to 6, characterized in that, A comparative analysis is conducted to determine the advantages and disadvantages of the new interest model compared to the old interest model, including: If the normalized cumulative gain of the first recommended content sequence is less than the normalized cumulative gain of the second recommended content sequence, then the new interest model is determined to be superior to the old interest model. If the normalized cumulative gain of the first recommended content sequence is greater than the normalized cumulative gain of the second recommended content sequence, then the old interest model is determined to be superior to the new interest model. If the normalized cumulative gain of the first recommended content sequence is equal to the normalized cumulative gain of the second recommended content sequence, then the old interest model and the new interest model are determined to be of equal quality.
8. A device for evaluating the quality of user interest, characterized in that, include: The first generation module is used to collect historical behavior data of users within the first time window, and generate old interest point sequences and new interest point sequences based on the historical behavior data through old interest model and new interest model respectively; The conversion module is used to convert the old interest point sequence and the new interest point sequence into a first recommended content sequence and a second recommended content sequence for predicting the user's interest preferences, respectively, through a preset recall mechanism. The second generation module is used to collect the user's real behavior data within the second time window, quantify the real behavior data according to the viewing duration, and generate a reference content sequence that reflects the user's real interests and preferences, wherein the second time window is located after the first time window in time. The comparison module is used to calculate the normalized cumulative gain of the first recommended content sequence and the second recommended content sequence according to the reference content sequence, and to perform comparative analysis to determine the superiority or inferiority of the new interest model relative to the old interest model.
9. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the method described in any one of claims 1 to 7.
10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 7 through the computer program.