News recommendation system evaluation method and device, electronic equipment and storage medium

Abstract

The invention provides a news recommendation system evaluation method and device, electronic equipment and a storage medium, and belongs to the technical field of system evaluation, and the method comprises the steps: simulating a user to send a browsing request to a target news recommendation system; obtaining push data returned by the target news recommendation system; and performing evaluation from five dimensions of information cocoon house evaluation, diversity evaluation, emotion cocoon house evaluation, objectivity evaluation and intention evaluation based on the pushed data, and outputting an evaluation result. According to the news recommendation system evaluation method provided by the invention, the technical problem that the news recommendation system evaluation method in the related technology cannot accurately and comprehensively evaluate the influence of the news recommendation system on the user is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of system evaluation, and in particular to a news recommendation system evaluation method, device, electronic device and storage medium. Background Art

[0002] The news recommendation system is an information filtering system that recommends news that users may be interested in by analyzing their behaviors, preferences, historical interactions, and the characteristics of the news content itself. However, users who use news recommendation systems often fall into partial information blind spots, that is, they can only receive information with homogeneous content and cannot access news headlines and news reports from opposing positions, which in turn exacerbates the popular bias and polarization effect of the recommendation system, resulting in challenges to the objectivity of the recommendation system.

[0003] The evaluation of news recommendation systems in related technologies mainly focuses on improving user satisfaction or improving user experience. The optimization goal of news recommendation systems is to increase customer retention rate. Therefore, it is inevitable to ignore the potential cocoon effect and the impact of news recommendation systems on users themselves.

[0004] It can be seen that the news recommendation system evaluation method in the related art has the technical problem of being unable to accurately and comprehensively evaluate the impact of the news recommendation system on users. Summary of the invention

[0005] The present invention provides a news recommendation system evaluation method, device, electronic device and storage medium, which are used to solve the technical problem that the news recommendation system evaluation method in the related art cannot accurately and comprehensively evaluate the impact of the news recommendation system on users.

[0006] The present invention provides a news recommendation system evaluation method, comprising the following steps: simulating a user sending a browsing request to a target news recommendation system; obtaining push data returned by the target news recommendation system; performing evaluation from five dimensions of information cocoon evaluation, diversity evaluation, emotional cocoon evaluation, objectivity evaluation and intention evaluation based on the push data, and outputting an evaluation result; wherein the information cocoon evaluation is used to evaluate the concentration of subject categories of the push information of the target news recommendation system, the diversity evaluation is used to evaluate the diversification of subject categories of the push information of the target news recommendation system, the emotional cocoon evaluation is used to evaluate the concentration of emotional tendencies of the push information of the target news recommendation system, the objectivity evaluation is used to evaluate whether the push information of the target news recommendation system is affected by economic interests and personal emotions, and the intention evaluation is used to evaluate the impact of the push information of the target news recommendation system on user behavior.

[0007] According to a news recommendation system evaluation method provided by the present invention, the simulated user sends a browsing request to the target news recommendation system, including: based on a target virtual agent model, the simulated user sends a browsing request to the target news recommendation system, wherein the target virtual agent model is a trained model for simulating user behavior of a real user group.

[0008] According to a news recommendation system evaluation method provided by the present invention, the target virtual agent model is obtained through the following steps: constructing an initial virtual agent model based on prior domain knowledge; calculating the difference in reading topic probability distribution between the virtual user and the real user of the initial virtual agent model based on the KL divergence, and adjusting the parameters of the initial virtual agent model based on the difference; using modified conditional entropy to evaluate the regularity of the behavior time series of the virtual user, and adjusting the parameters of the initial virtual agent model based on the evaluation result; obtaining real environment data during the interaction between the real user and the target news recommendation system; adjusting the parameters of the initial virtual agent model based on the real environment data to obtain the target virtual agent model.

[0009] According to a news recommendation system evaluation method provided by the present invention, the parameters of the initial virtual agent model are adjusted based on the real environment data to obtain the target virtual agent model, including: extracting keywords in the real environment data based on the TF-IDF method to obtain a keyword set; updating the keyword database of the initial virtual agent model according to the keyword set based on the freshness effect; determining the real topic category according to the real environment data, and updating the parameters of the initial virtual agent model based on the real topic category to obtain the target virtual agent model.

[0010] According to a news recommendation system evaluation method provided by the present invention, the prior domain knowledge includes user behavior feature data, topic preference distribution data and login environment feature data, and the construction of an initial virtual agent model based on the prior domain knowledge includes: initializing a behavior feature model based on the user behavior feature data, wherein the behavior feature model is used to simulate the time when the virtual user uses the target news recommendation system; initializing a topic feature model based on the topic preference distribution data, wherein the topic feature model is used to simulate the topic preference when the virtual user uses the target news recommendation system; initializing an environment feature model based on the login environment feature data, wherein the environment feature model is used to simulate the login environment of the virtual user, and the login environment includes the device value for logging into the target news recommendation system; and merging the behavior feature model, the topic feature model and the environment feature model to obtain the initial virtual agent model.

[0011] According to a news recommendation system evaluation method provided by the present invention, the push data includes at least one push article.

[0012] The present invention also provides a news recommendation system evaluation device, comprising the following modules: a simulation module, used to simulate a user sending a browsing request to a target news recommendation system; an acquisition module, used to acquire push data returned by the target news recommendation system; an evaluation module, used to evaluate based on the push data from five dimensions: information cocoon evaluation, diversity evaluation, emotional cocoon evaluation, objectivity evaluation and intention evaluation, and output an evaluation result; wherein, the information cocoon evaluation is used to evaluate the concentration of subject categories of the push information of the target news recommendation system, the diversity evaluation is used to evaluate the diversification of subject categories of the push information of the target news recommendation system, the emotional cocoon evaluation is used to evaluate the concentration of emotional tendencies of the push information of the target news recommendation system, the objectivity evaluation is used to evaluate whether the push information of the target news recommendation system is affected by economic interests and personal emotions, and the intention evaluation is used to evaluate the impact of the push information of the target news recommendation system on user behavior.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the news recommendation system evaluation method described in any one of the above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the news recommendation system evaluation method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the news recommendation system evaluation method described in any one of the above is implemented.

[0016] The news recommendation system evaluation method, device, electronic device and storage medium provided by the present invention simulate a user sending a browsing request to a target news recommendation system; obtain push data returned by the target news recommendation system; evaluate from five dimensions of information cocoon evaluation, diversity evaluation, emotional cocoon evaluation, objectivity evaluation and intention evaluation based on the push data, and output the evaluation results; by evaluating the push data returned by the target news recommendation system based on multiple dimensions from the perspective of the information receiver, it is possible to accurately and comprehensively evaluate the impact of the news recommendation system on the user, thereby solving the technical problem that the news recommendation system evaluation method in the related art cannot accurately and comprehensively evaluate the impact of the news recommendation system on the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the news recommendation system evaluation method provided by the present invention.

[0019] Figure 2 It is a structural diagram of the news recommendation system evaluation device provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] It should be noted that in the description of the present invention, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0023] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" means at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0024] The news recommendation system is an information filtering system that recommends news that may be of interest to users by analyzing their behaviors, preferences, historical interactions, and the characteristics of the news content itself. However, users who use news recommendation systems often fall into partial information blind spots, that is, they can only receive information with homogeneous content and cannot access news headlines and news reports from opposing positions, which in turn exacerbates the popular bias and polarization of the recommendation system, resulting in challenges to the objectivity of the recommendation system. In addition, the passive "push" mode between users and news recommendation systems limits users' choices in reading information content, and users are easily influenced by the intention of "push" messages, resulting in users being very active in responding to false information, but passive in reflecting on the authenticity of the information.

[0025] The evaluation of news recommendation systems in related technologies mainly focuses on improving user satisfaction or user experience, and the optimization goal of news recommendation systems is to increase customer retention. Therefore, it is inevitable to ignore the potential cocoon effect and the impact of news recommendation systems on users themselves. It can be seen that the evaluation methods of news recommendation systems in related technologies have the technical problem of being unable to accurately and comprehensively evaluate the impact of news recommendation systems on users.

[0026] In order to at least solve some of the above problems, the following Figure 1-Figure 3 The present invention describes a news recommendation system evaluation method, device, electronic device and storage medium.

[0027] The news recommendation system evaluation method provided in this embodiment can be applied to the scenario of evaluating the news recommendation system.

[0028] The news recommendation system evaluation method in this embodiment can be executed by a server. The server simulates a user sending a browsing request to the target news recommendation system. After obtaining the push data returned by the target news recommendation system, the server evaluates the target news recommendation system from multiple dimensions based on the push data and outputs the evaluation results.

[0029] Figure 1 is a flow chart of the news recommendation system evaluation method provided by the present invention, such as Figure 1 As shown, including but not limited to the following steps: Step 101, simulating a user sending a browsing request to a target news recommendation system.

[0030] In this embodiment, the server can create a simulation environment in which a virtual user imitates the behavior of a real user and sends a browsing request to the target news recommendation system; the browsing request may include information such as the user's interest preferences, historical behavior, geographic location, etc., so that the recommendation system can provide personalized news content based on this information.

[0031] Specifically, the server may be provided with a virtual agent model, through which a virtual user is created and the virtual user imitates the behavior of a real user; here, a virtual agent is a software program that can simulate human behavior or perform specific tasks.

[0032] Step 102: Obtain push data returned by the target news recommendation system.

[0033] Optionally, the pushed data includes at least one pushed article.

[0034] After sending a browsing request, the simulated user will receive a response from the target news recommendation system, that is, push data; the push data may include at least one pushed article, and the number of pushed articles is generally multiple, which are selected by the target news recommendation system based on the virtual user's browsing request and recommendation algorithm.

[0035] Step 103, based on the pushed data, an evaluation is performed from five dimensions: information cocoon evaluation, diversity evaluation, emotional cocoon evaluation, objectivity evaluation, and intention evaluation, and the evaluation results are output; Among them, the information cocoon evaluation is used to evaluate the concentration of the subject categories of the push information of the target news recommendation system, the diversity evaluation is used to evaluate the diversification of the subject categories of the push information of the target news recommendation system, the emotional cocoon evaluation is used to evaluate the concentration of the emotional tendency of the push information of the target news recommendation system, the objectivity evaluation is used to evaluate whether the push information of the target news recommendation system is affected by economic interests and personal emotions, and the intention evaluation is used to evaluate the impact of the push information of the target news recommendation system on user behavior.

[0036] Information cocoon evaluation is used to evaluate the concentration of the subject categories of the push information of the target news recommendation system, that is, to analyze the subject categories of the push data and evaluate the subject concentration of the push data. If the pushed articles in the push data are too concentrated in terms of subject categories, users may receive a single piece of information, forming an information cocoon.

[0037] Specifically, if a user reads a large number of articles belonging to the same subject category, then the risk of falling into an information cocoon is relatively high; conversely, if the user's reading interests are relatively dispersed in the subject category, then the risk of falling into an information cocoon is relatively low. In this embodiment, this indicator is calculated based on the user's online time (or user's clicks) and the concentration of topics covered in the recommended news, as shown in formula (1). The recommendation system For users In the The information cocoon coefficient of the day is : (1) in, It is a recommender The number of recommended topics, Recommended to users Articles in the topic The proportion of ;when When , the information cocoon coefficient is equal to 1, indicating a high degree of convergence.

[0038] The convergence factor refers to the number of topic categories during the observation period (weekly / every 500 hits). The number of days / clicks required for the recommended articles to reach the highest point within a period will be used to recommend the system The convergence factor with respect to the time interval is expressed as , which is calculated as shown in formula (2), the convergence factor of the number of clicks can be calculated in a similar way: (2) in, It is Days belong to the theme The proportion of articles is the highest point, , Is the starting point.

[0039] Furthermore, diversity evaluation is used to evaluate the diversity of the subject categories of the push information of the target news recommendation system, that is, to evaluate the diversity of the subject categories of the push data. High diversity means that the recommendation system can provide a variety of types of news to help users broaden their horizons.

[0040] Specifically, diversity is used to measure the extent to which articles in the target news recommendation system belong to different topic categories. As shown in formula (3), the "entropy" metric is selected as a representative metric for "diversity" calculation because it reflects more "richness" and "uniformity" than "coverage": (3) in, Recommended to users The article in The proportion of content in each subject category, with higher values ​​indicating more diverse content.

[0041] Optionally, the separation degree is used to measure the degree of difference between different categories, that is, the degree to which their respective articles are dispersed in terms of sentiment. The specific calculation is shown in formula (4), which reflects the degree to which the recommended information in each topic category is different in terms of attitude: (4) in, Indicates of the categories The emotional value of the article, Indicates the number of articles.

[0042] Optionally, the difference is used to measure whether the article is more authoritative than other recommended articles. It can be measured by the number of followers of the author in online news media. The specific calculation is shown in formula (5): (5) in, Recommended to users Articles The number of followers of the author, It represents the number of articles. Indicates the average.

[0043] Furthermore, the emotional cocoon evaluation is used to evaluate the concentration of the emotional tendency of the pushed information of the target news recommendation system, that is, to analyze the emotional tendency of the pushed data and evaluate the degree of emotional polarization. If the emotional tendency of the pushed articles is too concentrated, it may strengthen the user's existing emotions and form an emotional cocoon.

[0044] Specifically, the emotional cocoon is used to reflect the concentration of emotions in the recommended news in polarized user groups, which will For users In the The emotional cocoon coefficient of the day is expressed as , calculated as shown in formula (6): (6) in, represents the number of sentiment categories 1, Indicates that it belongs to the category The proportion of articles , y1 and y2 are freely settable parameters; .

[0045] Furthermore, objectivity evaluation is used to evaluate whether the push information of the target news recommendation system is affected by economic interests and personal emotions, that is, to check whether the push data is affected by economic interests and personal emotions, and to evaluate its objectivity. Objective news recommendations should be based on facts rather than prejudice or commercial interests.

[0046] Specifically, objectivity includes "commercial orientation", "head account orientation" and "emotional neutrality"; commercial orientation refers to the proportion of commercial advertising articles, and the specific calculation is shown in formula (7): (7) in, Indicates that the news recommender recommends to the user The total number of articles Recommended to users The number of commercials.

[0047] The top account orientation refers to the proportion of articles from top accounts. The specific calculation is shown in formula (8).

[0048] (8) in, Indicates the number of articles written by top users, i.e. users with ≥ 100,000 followers.

[0049] Sentiment neutrality refers to the proportion of articles with “neutral” sentiment. The specific calculation is shown in formula (9).

[0050] (9) in, Indicates the number of articles with neutral sentiment.

[0051] Furthermore, intention evaluation is used to evaluate the impact of the push information of the target news recommendation system on user behavior, that is, to analyze the impact of push data on user behavior and evaluate whether the recommendation system can guide users to perform expected behaviors, such as reading, sharing, and commenting.

[0052] Specifically, intentionality includes audiences’ perceived self-relevance, perceived persuasiveness, and perceived difficulty.

[0053] Audience’s perception of self-relevance: According to Maslow’s hierarchy of needs theory, information closely related to the bottom two levels, namely safety needs and physiological needs, will attract the greatest attention of the audience. The subject categories belonging to these two levels include natural disasters, social unrest, price fluctuations, birth, death, medical insurance, etc. The calculation is shown in formula (10): (10) in, represents the number of articles on topics such as natural disasters, social unrest, price fluctuations, births, deaths, and health care, while Indicates the total number of recommended articles.

[0054] Audience-perceived persuasiveness: Articles with provocative and exaggerated sentences, some of which are common in rumors, will make the audience feel highly persuaded and have a viral effect in information dissemination. The calculation is shown in formula (11): (11) in, Indicates the number of articles with provocative sentences in their titles.

[0055] Difficulty perceived by the audience: Specialized terms in a specific subject area increase the difficulty of understanding and require additional effort, which is unlikely to happen to users who use news aggregation applications at leisure. Such content is likely to receive less attention and have a weaker intentional impact on the information recipient. Therefore, the proportion of terms in the text is calculated as shown in formula (12): (12) in, Indicates the number of articles whose title and first paragraph contain the term.

[0056] It can be understood that the output evaluation results may include scores for each dimension, overall performance evaluation, and improvement suggestions. These evaluation results are crucial for understanding the effectiveness of the target news recommendation system, identifying potential problems, and guiding the optimization of the target news recommendation system. Through the evaluation method of this embodiment, it can be ensured that the recommendation system can provide high-quality and diverse news content while avoiding the formation of problems such as information cocoons and emotional cocoons.

[0057] In this example, an impact assessment framework for a news recommendation system is proposed, which consists of five structures: diversity, objectivity, information cocoon, emotional cocoon, and intentionality. The assessment is performed in a "black box" manner, based on empirical interaction data between the news recommendation system and individual users; in order to achieve individual-level assessment, a lightweight agent-based simulation is proposed to simulate and collect interaction data between users and the news recommendation system in complex real-world scenarios.

[0058] Specifically, information cocoon measures the degree of concentration of the topic distribution of articles recommended to users by the recommendation system as the user's online time increases; diversity is reflected in the diversification of the topic groups in the recommended news for each individual user; objectivity measures whether the user's perception is distorted by economic interests and personal feelings, such as business orientation, emotional neutrality, etc.; emotional cocoon measures the concentration of emotions reflected in the recommended news related to the user's polarized groups; intentionality measures the system's ability to influence the user's decision to use the system, the user's agreement to use the system, the user's retention, and the ability to introduce the system to his / her friends.

[0059] The final evaluation results are shown in the following table:

[0060] Through the embodiments provided by the present application, a user is simulated to send a browsing request to a target news recommendation system; push data returned by the target news recommendation system is obtained; based on the push data, an evaluation is performed from five dimensions: information cocoon evaluation, diversity evaluation, emotional cocoon evaluation, objectivity evaluation, and intention evaluation, and an evaluation result is output; wherein, the information cocoon evaluation is used to evaluate the concentration of the subject categories of the push information of the target news recommendation system, the diversity evaluation is used to evaluate the diversification of the subject categories of the push information of the target news recommendation system, the emotional cocoon evaluation is used to evaluate the concentration of the emotional tendencies of the push information of the target news recommendation system, the objectivity evaluation is used to evaluate whether the push information of the target news recommendation system is affected by economic interests and personal emotions, and the intention evaluation is used to evaluate the impact of the push information of the target news recommendation system on user behavior; the news recommendation system evaluation method in the related art is solved, and there is a technical problem that the impact of the news recommendation system on users cannot be accurately and comprehensively evaluated, thereby improving the accuracy of the evaluation of the news recommendation system.

[0061] As an optional solution, simulate a user sending a browsing request to the target news recommendation system, including: S11, simulating a user to send a browsing request to a target news recommendation system based on a target virtual agent model, wherein the target virtual agent model is a trained model for simulating user behavior of a real user group.

[0062] In this embodiment, the server simulates the user sending a browsing request to the target news recommendation system, which is implemented based on the target virtual agent model. The target virtual agent model is set in the server, and the target virtual agent model is a trained model for simulating user behavior of a real user group.

[0063] A virtual agent model is an artificial intelligence model that is designed to simulate the behavior and decision-making process of real users. This model is usually used in various application scenarios, such as simulating user interactions, testing software, analyzing user behavior, optimizing recommendation systems, etc. In this embodiment, the target virtual agent model can simulate and predict user behavior without involving real users, providing support for the evaluation of news recommendation systems.

[0064] Through this embodiment, based on the target virtual agent model, simulating the user to send a browsing request to the target news recommendation system, it is possible to simulate and predict user behavior without involving real users, thereby improving the reliability of the news recommendation system evaluation method of this embodiment.

[0065] As an optional solution, the target virtual agent model is obtained by the following steps: S21, constructing the initial virtual agent model based on prior domain knowledge; S22, calculating the difference in reading topic probability distribution between the virtual user and the real user of the initial virtual agent model based on the KL divergence, and adjusting the parameters of the initial virtual agent model based on the difference; S23, using the modified conditional entropy to evaluate the regularity of the behavior time series of the virtual user, and adjusting the parameters of the initial virtual agent model based on the evaluation results; S24, obtaining real environment data during the interaction between real users and the target news recommendation system; S25, adjusting the parameters of the initial virtual agent model based on the real environment data to obtain a target virtual agent model.

[0066] In this embodiment, prior domain knowledge refers to the knowledge accumulated by people in a specific field based on past experience, research and learning. An initial virtual agent model is constructed based on the prior domain knowledge. Here, the initial virtual agent model can simulate the process of user interaction with the target news recommendation system. Steps S22-S25 are the process of training the initial virtual agent model and adjusting the parameters of the initial virtual agent model.

[0067] Furthermore, the difference in reading topic probability distribution between the virtual user and the real user of the initial virtual agent model is calculated based on the KL (Kullback-Leibler) divergence, and the parameters of the initial virtual agent model are adjusted based on the difference; the KL divergence is used to measure the difference between two probability distributions and is a measure of divergence or difference; S22 verifies the interest compatibility of the real user and the virtual user.

[0068] Specifically, KL divergence is used to calculate the real user and virtual users The difference between the probability distributions of the topics read (once a week), the specific calculation is shown in formula (13), and a smaller KL divergence value indicates a closer interest distribution: (13) in, Represents a subject index, represents the cross entropy, Represents the entropy of real users.

[0069] Further, the modified conditional entropy is used to evaluate the regularity of the behavior timing of the virtual user, and the parameters of the initial virtual agent model are adjusted based on the evaluation results; here, S23 is based on the behavior timing compatibility verification of the real user and the virtual user.

[0070] Specifically, human behavior is usually more complex and irregular than machine behavior. Therefore, as a measure of process complexity, entropy is usually used to distinguish human behavior from machine behavior. In order to verify the temporal compatibility of real users and virtual users, the modified conditional entropy is used for verification. The specific calculation is shown in formula (14): (14) in, Indicates a conditional item. The right part is the correction term. Represents the sequence of online clicks within a time period, with the lower Values ​​indicate periodic or regular timing of tweet behavior, indicating signs of automation, while higher values ​​mean irregular behavior, indicating human involvement.

[0071] Furthermore, real-world data is obtained during the interaction between real users and the target news recommendation system. Here, it is necessary to collect recommendation information from real users each time they click on the target news recommendation system for evaluation and analysis. For each topic, the distribution of topic keywords may change within a certain period of time, so the target virtual agent model needs to be adjusted accordingly.

[0072] Furthermore, the parameters of the initial virtual agent model are adjusted based on the real environment data to obtain the target virtual agent model; optionally, the parameters of the initial virtual agent model may be updated based on changes in topic keywords in the real environment data.

[0073] Through this embodiment, the efficiency of parameter adjustment of the initial virtual agent model can be effectively improved, thereby improving the reliability and stability of the news recommendation system evaluation method of this embodiment.

[0074] As an optional solution, the parameters of the initial virtual agent model are adjusted based on the real environment data to obtain the target virtual agent model, including: S31, extracting keywords from real environment data based on the TF-IDF method to obtain a keyword set; S32, updating the keyword database of the initial virtual agent model according to the keyword set based on the freshness effect; S33, determining the real subject category according to the real environment data, and updating the parameters of the initial virtual agent model based on the real subject category to obtain the target virtual agent model.

[0075] In this embodiment, TF-IDF (Term Frequency-Inverse Document Frequency) is a weight calculation method widely used in information retrieval and text mining, and is used to evaluate the importance of a word in a document set or corpus. By extracting keywords from real environment data based on the TF-IDF method, an accurate keyword set can be obtained.

[0076] Specifically, Indicates that each article is in the time interval The top numbers extracted using the TF-IDF method A set of keywords (optional, ).

[0077] Furthermore, based on the freshness effect, the keyword database of the initial virtual agent model is updated according to the keyword set; specifically, considering the freshness effect, the keyword set (You can Set to once a week) is divided into the most recent past keyword collection (keywords of articles read in the past 7 days) and further past keyword collections (Keywords of articles read more than 7 days ago), ; Among them, for The adjustments to the keyword database are as follows: ; For keywords ; For keywords .

[0078] Furthermore, the real subject category is determined according to the real environment data, and the parameters of the initial virtual agent model are updated based on the real subject category to obtain the target virtual agent model.

[0079] Specifically, since the interests of users may change over time, the real topic categories can be determined according to the real environment data, and the parameters of the initial virtual agent model can be updated based on the real topic categories; first, a convolutional neural network (CNN) can be applied to learn and predict which topics an article belongs to; then, for each user, the topics are sorted according to the number of their articles, and the top n topic categories are obtained; if the top n topic categories are different from the set of topic categories defined in the initial virtual agent model, the set of topic categories of the initial virtual agent model is updated, that is, the parameters of the initial virtual agent model are updated based on the real topic categories to obtain the target virtual agent model.

[0080] Through this embodiment, the accuracy of parameter adjustment of the initial virtual agent model can be effectively improved, thereby improving the reliability of the news recommendation system evaluation method of this embodiment.

[0081] As an optional solution, the prior domain knowledge includes user behavior feature data, topic preference distribution data, and login environment feature data. The initial virtual agent model is constructed based on the prior domain knowledge, including: S41, initializing a behavior feature model based on the user behavior feature data, wherein the behavior feature model is used to simulate the time that a virtual user uses the target news recommendation system; S42, initializing a topic feature model based on the topic preference distribution data, wherein the topic feature model is used to simulate the topic preference of a virtual user when using the target news recommendation system; S43, initializing an environment feature model based on the login environment feature data, wherein the environment feature model is used to simulate the login environment of the virtual user, and the login environment includes the device value of the login target news recommendation system; S44, the behavior feature model, the topic feature model and the environment feature model are combined to obtain an initial virtual agent model.

[0082] In this embodiment, a behavior characteristic model is initialized based on user behavior characteristic data, wherein the behavior characteristic model is used to simulate the time that a virtual user uses the target news recommendation system; the purpose of this step is to create a model to simulate the usage time of a virtual user on the target news recommendation system; specifically, user behavior data can be collected, including but not limited to the user's login time, browsing time, click frequency, reading time, etc.; these data are used to train a behavior characteristic model that can predict or simulate the user's usage time pattern; this model can be a time series analysis, a regression model or other suitable machine learning model, and a behavior characteristic model is obtained after the training is completed.

[0083] Furthermore, a topic feature model is initialized based on the topic preference distribution data, wherein the topic feature model is used to simulate the topic preference of a virtual user when using the target news recommendation system; the purpose of this step is to create a model to simulate the topic preference of a virtual user on the target news recommendation system; specifically, the user's historical reading records are analyzed to extract the user's topic preference, such as through TF-IDF, topic models and other methods; these preference data are used to train a topic feature model that can represent the user's interest distribution; this model can be a classifier, a clustering algorithm or other machine learning model that can process preference data, and a topic feature model is obtained after the training is completed.

[0084] Furthermore, an environment feature model is initialized based on the login environment feature data, wherein the environment feature model is used to simulate the login environment of the virtual user, and the login environment includes the device value of the target news recommendation system for logging in; the purpose of this step is to create a model to simulate the login environment of the virtual user, especially the device value; specifically, feature data about the user login environment is collected, such as device type, operating system, browser version, network speed, etc.; these data are used to train an environment feature model that can reflect the impact of different login environments on user behavior; this model can be a regression model after feature engineering or other suitable environment modeling methods, and the environment feature model is obtained after training.

[0085] Furthermore, the behavior feature model, the topic feature model and the environment feature model are combined to obtain an initial virtual agent model; here, multiple feature models are combined to create a comprehensive initial virtual agent model to more comprehensively simulate real users; the outputs of the behavior feature model, the topic feature model and the environment feature model are integrated, specifically, this can be achieved through model fusion techniques such as stacking, weighted averaging or other integrated learning methods; the obtained initial virtual agent model will be used to simulate the comprehensive behavior of users in the news recommendation system.

[0086] Optionally, the prior domain knowledge can be expressed as ,in is the initial theme feature, is a collection of themes, is a set of keywords. , is the user's theme preference; Behavioral characteristics, i.e., the interaction frequency (number of clicks) per 100 articles, the interaction time per day, and the online time period; is a feature of the environment; the problem can be defined as In this paper, we simulate the online behavior of users on new aggregation and recommendation applications.

[0087] Based on the prior knowledge of the user group (young people in tertiary cities), the assumptions of the simulation method for N users are as follows: each user belongs to a certain user group; during the observation period (1-3 months), the composition of the user group is relatively stable; during the observation period (1-3 months), the interest distribution of users is relatively stable.

[0088] Furthermore, an initial virtual agent model is constructed based on user behavior based on prior domain knowledge. The specific steps are as follows: 1) Login behavior simulation based on behavioral characteristics. For each user First, based on prior domain knowledge Get random online time at the time interval specified in the template and random login times ;when When, it means in the time period Continuous activity within When, it indicates continuous activity. Not exceeding the time period Get the number of hours of online time for each segment Random online time within let Indicates time period The length of time that users will be in the segmented time period Log in within , start time no later than — .

[0089] 2) News reading behavior simulation based on initial topic features. First, obtain the simulated interaction frequency The simulator distributes according to the subject preference And keyword topic distribution characteristics Determines whether the avatar clicks to read the news; if the article belongs to (keyword set of the preferred topic), then the article will be displayed in the segmented online time period with the simulated click frequency is clicked, the recommended page is simulated to slide until Finish.

[0090] 3) Login environment simulation based on environmental characteristics. First, mobile phone brands are classified into four categories: "economic type (≤ RMB 1,500)", "ordinary type (RMB 1,500-5,000)", "flagship type (≥ RMB 5,000)", and "special type (limited edition and hunger marketing)"; to model mobile phone brand characteristics, including social media usage distribution and sales distribution of mobile phone brands within each category; Can be directly Derived.

[0091] Through this embodiment, an initial virtual agent model can be accurately constructed based on prior domain knowledge, thereby improving the reliability and stability of the news recommendation system evaluation method of this embodiment.

[0092] Figure 2 is a structural diagram of the news recommendation system evaluation device provided by the present invention, such as Figure 2 As shown, including but not limited to the following modules: Simulation module 201, used to simulate a user sending a browsing request to a target news recommendation system; The acquisition module 202 is used to acquire the push data returned by the target news recommendation system; Evaluation module 203, used to evaluate the push data from five dimensions: information cocoon evaluation, diversity evaluation, emotional cocoon evaluation, objectivity evaluation, and intention evaluation, and output the evaluation results; Among them, the information cocoon evaluation is used to evaluate the concentration of the subject categories of the push information of the target news recommendation system, the diversity evaluation is used to evaluate the diversification of the subject categories of the push information of the target news recommendation system, the emotional cocoon evaluation is used to evaluate the concentration of the emotional tendency of the push information of the target news recommendation system, the objectivity evaluation is used to evaluate whether the push information of the target news recommendation system is affected by economic interests and personal emotions, and the intention evaluation is used to evaluate the impact of the push information of the target news recommendation system on user behavior.

[0093] Through the embodiments of the present application, a user is simulated to send a browsing request to a target news recommendation system; push data returned by the target news recommendation system is obtained; based on the push data, an evaluation is performed from five dimensions: information cocoon evaluation, diversity evaluation, emotional cocoon evaluation, objectivity evaluation, and intention evaluation, and an evaluation result is output; wherein, the information cocoon evaluation is used to evaluate the concentration of the subject categories of the push information of the target news recommendation system, the diversity evaluation is used to evaluate the diversification of the subject categories of the push information of the target news recommendation system, the emotional cocoon evaluation is used to evaluate the concentration of the emotional tendencies of the push information of the target news recommendation system, the objectivity evaluation is used to evaluate whether the push information of the target news recommendation system is affected by economic interests and personal emotions, and the intention evaluation is used to evaluate the impact of the push information of the target news recommendation system on user behavior; the news recommendation system evaluation method in the related art is solved, and there is a technical problem that the impact of the news recommendation system on users cannot be accurately and comprehensively evaluated, thereby improving the accuracy of the evaluation of the news recommendation system.

[0094] It should be noted that the news recommendation system evaluation device provided by the present invention can execute the news recommendation system evaluation method of any of the above embodiments during specific operation, which will not be described in detail in this embodiment.

[0095] Figure 3 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 complete mutual communication through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the news recommendation system evaluation method, which includes: simulating a user to send a browsing request to the target news recommendation system; obtaining the push data returned by the target news recommendation system; based on the push data, evaluating from five dimensions of information cocoon evaluation, diversity evaluation, emotional cocoon evaluation, objectivity evaluation and intention evaluation, and outputting the evaluation result; wherein, the information cocoon evaluation is used to evaluate the concentration of the subject categories of the push information of the target news recommendation system, the diversity evaluation is used to evaluate the diversification of the subject categories of the push information of the target news recommendation system, the emotional cocoon evaluation is used to evaluate the concentration of the emotional tendency of the push information of the target news recommendation system, the objectivity evaluation is used to evaluate whether the push information of the target news recommendation system is affected by economic interests and personal emotions, and the intention evaluation is used to evaluate the impact of the push information of the target news recommendation system on user behavior.

[0096] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0097] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the news recommendation system evaluation method provided in the above-mentioned embodiments, and the method includes: simulating a user sending a browsing request to a target news recommendation system; obtaining push data returned by the target news recommendation system; based on the push data, performing an evaluation from five dimensions: information cocoon evaluation, diversity evaluation, emotional cocoon evaluation, objectivity evaluation, and intention evaluation, and outputting an evaluation result; wherein, the information cocoon evaluation is used to evaluate the concentration of subject categories of the push information of the target news recommendation system, the diversity evaluation is used to evaluate the diversification of subject categories of the push information of the target news recommendation system, the emotional cocoon evaluation is used to evaluate the concentration of emotional tendencies of the push information of the target news recommendation system, the objectivity evaluation is used to evaluate whether the push information of the target news recommendation system is affected by economic interests and personal emotions, and the intention evaluation is used to evaluate the impact of the push information of the target news recommendation system on user behavior.

[0098] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the news recommendation system evaluation method provided in the above-mentioned embodiments, the method comprising: simulating a user sending a browsing request to a target news recommendation system; obtaining push data returned by the target news recommendation system; performing an evaluation based on the push data from five dimensions, namely, information cocoon evaluation, diversity evaluation, emotional cocoon evaluation, objectivity evaluation and intention evaluation, and outputting an evaluation result; wherein, the information cocoon evaluation is used to evaluate the concentration of subject categories of the push information of the target news recommendation system, the diversity evaluation is used to evaluate the diversification of subject categories of the push information of the target news recommendation system, the emotional cocoon evaluation is used to evaluate the concentration of emotional tendencies of the push information of the target news recommendation system, the objectivity evaluation is used to evaluate whether the push information of the target news recommendation system is affected by economic interests and personal emotions, and the intention evaluation is used to evaluate the impact of the push information of the target news recommendation system on user behavior.

[0099] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0100] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A news recommendation system evaluation method, characterized in that: include: Simulate users to send browsing requests to the target news recommendation system; Obtaining the push data returned by the target news recommendation system; Based on the push data, the evaluation is conducted from five dimensions: information cocoon evaluation, diversity evaluation, emotional cocoon evaluation, objectivity evaluation, and intention evaluation, and the evaluation results are output; Among them, the information cocoon evaluation is used to evaluate the concentration of the subject categories of the push information of the target news recommendation system, the diversity evaluation is used to evaluate the diversification of the subject categories of the push information of the target news recommendation system, the emotional cocoon evaluation is used to evaluate the concentration of the emotional tendency of the push information of the target news recommendation system, the objectivity evaluation is used to evaluate whether the push information of the target news recommendation system is affected by economic interests and personal emotions, and the intentionality evaluation is used to evaluate the impact of the push information of the target news recommendation system on user behavior.

2. The news recommendation system evaluation method according to claim 1, characterized in that: The simulated user sends a browsing request to the target news recommendation system, including: The target virtual agent model is used to simulate a user sending a browsing request to the target news recommendation system, wherein the target virtual agent model is a trained model for simulating user behavior of a real user group.

3. The news recommendation system evaluation method according to claim 2, characterized in that: The target virtual agent model is obtained by the following steps: Construct an initial virtual agent model based on prior domain knowledge; Calculating the difference in reading topic probability distribution between the virtual user and the real user of the initial virtual agent model based on KL divergence, and adjusting the parameters of the initial virtual agent model based on the difference; Using the modified conditional entropy to evaluate the regularity of the behavior time series of the virtual user, and adjusting the parameters of the initial virtual agent model based on the evaluation results; Acquire real environment data during the interaction between the real user and the target news recommendation system; The parameters of the initial virtual agent model are adjusted based on the real environment data to obtain the target virtual agent model.

4. The news recommendation system evaluation method according to claim 3, characterized in that: The adjusting the parameters of the initial virtual agent model based on the real environment data to obtain the target virtual agent model includes: Extract keywords from the real environment data based on the TF-IDF method to obtain a keyword set; updating a keyword database of the initial virtual agent model according to the keyword set based on a freshness effect; A real subject category is determined according to the real environment data, and parameters of the initial virtual agent model are updated based on the real subject category to obtain the target virtual agent model.

5. The news recommendation system evaluation method according to claim 3, characterized in that: The prior domain knowledge includes user behavior feature data, topic preference distribution data, and login environment feature data. The initial virtual agent model is constructed based on the prior domain knowledge, including: Initializing a behavior characteristic model based on the user behavior characteristic data, wherein the behavior characteristic model is used to simulate the time when the virtual user uses the target news recommendation system; Initializing a topic feature model based on the topic preference distribution data, wherein the topic feature model is used to simulate the topic preference of the virtual user when using the target news recommendation system; Initializing an environment feature model based on the login environment feature data, wherein the environment feature model is used to simulate the login environment of the virtual user, and the login environment includes the device value of logging into the target news recommendation system; The initial virtual agent model is obtained by combining the behavior feature model, the subject feature model and the environment feature model.

6. The news recommendation system evaluation method according to any one of claims 1 to 5, characterized in that: The pushed data includes at least one pushed article.

7. A news recommendation system evaluation device, characterized in that: include: A simulation module is used to simulate a user sending a browsing request to a target news recommendation system; An acquisition module, used to acquire the push data returned by the target news recommendation system; An evaluation module, used to evaluate the push data from five dimensions: information cocoon evaluation, diversity evaluation, emotional cocoon evaluation, objectivity evaluation, and intention evaluation, and output evaluation results; Among them, the information cocoon evaluation is used to evaluate the concentration of the subject categories of the push information of the target news recommendation system, the diversity evaluation is used to evaluate the diversification of the subject categories of the push information of the target news recommendation system, the emotional cocoon evaluation is used to evaluate the concentration of the emotional tendency of the push information of the target news recommendation system, the objectivity evaluation is used to evaluate whether the push information of the target news recommendation system is affected by economic interests and personal emotions, and the intentionality evaluation is used to evaluate the impact of the push information of the target news recommendation system on user behavior.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the news recommendation system evaluation method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the news recommendation system evaluation method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the news recommendation system evaluation method according to any one of claims 1 to 6 is implemented.

Images