Evaluation method, device, electronic device and computer-readable storage medium

Abstract

The present application discloses an evaluation method, device, electronic device, and computer-readable storage medium, the method comprising: determining the probability value of each user in the first user set participating in an event to be evaluated based on first data corresponding to each user in the first user set; determining the weight value of each user in the first user set in evaluating the event to be evaluated based on the probability value of each user in the first user set participating in the event to be evaluated and second data corresponding to each user in the first user set; determining the evaluation result of the event to be evaluated based on the weight value of each user in the first user set in evaluating the event to be evaluated, the second data corresponding to each user in the first user set, and the third data corresponding to each user in the first user set. The method uses the weight value of the user in evaluating the event to be evaluated as a factor in calculating the evaluation result, avoiding the problem of inaccurate evaluation results caused by individual differences between users.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an evaluation method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the rapid development of computer technology, online business applications such as online shopping and online entertainment have gradually replaced offline activities and become a new trend. Operators can increase the revenue of online business applications through refined operations. For example, in gaming applications, game developers can increase additional revenue by recommending gift packs to players. However, not every refined operations project will have a positive impact on business applications. For example, if a refined operations project increases business application revenue for a period of time, but user churn is excessive, revenue will inevitably decrease in subsequent periods. Therefore, it is essential to evaluate the effectiveness of refined operations projects, and user retention is a key indicator for evaluating the effectiveness of refined operations projects.

[0003] Existing technologies primarily evaluate refined operations through AB experiments. This approach, on the one hand, dilutes the impact of refined operations on user retention due to the limited number of users participating in the experimental group. On the other hand, significant individual differences among users make it impossible to directly compare users within the experimental group. Consequently, existing refined operations evaluation methods are unscientific and lack accurate results. Summary of the Invention

[0004] The present application provides an evaluation method, device, electronic device and computer-readable storage medium to solve the problems of unscientific evaluation methods and inaccurate evaluation results in existing refined operation evaluation methods.

[0005] The present application provides an evaluation method, which includes:

[0006] Determining, based on first data corresponding to each user in the first user set, a probability value of each user in the first user set participating in the event to be evaluated, wherein the first data is data generated by each user in the first user set within a first preset time period for at least one influencing factor that affects the user's participation in the event to be evaluated;

[0007] Determining a weight value of each user in the first user set in evaluating the event to be evaluated based on the probability value of each user in the first user set participating in the event to be evaluated and second data corresponding to each user in the first user set, wherein the second data is data indicating whether each user in the first user set participated in the event to be evaluated within the first preset time period;

[0008] An evaluation result of the event to be evaluated is determined based on a weight value of each user in the first user set in evaluating the event to be evaluated, the second data corresponding to each user in the first user set, and the third data corresponding to each user in the first user set, wherein the third data is data indicating whether each user in the first user set has churned within a second preset time period, the second preset time period is a time period adjacent to the first preset time period, and the first preset time period is earlier than the second preset time period.

[0009] The embodiment of the present application further provides an evaluation device, the device comprising: a probability value determination unit, a weight value determination unit, and an evaluation result determination unit;

[0010] The probability value determination unit is configured to determine, based on first data corresponding to each user in the first user set, a probability value of each user in the first user set participating in the event to be evaluated, wherein the first data is data generated by each user in the first user set within a first preset time period for at least one influencing factor that affects the user's participation in the event to be evaluated;

[0011] The weight value determining unit is configured to determine a weight value of each user in the first user set in evaluating the event to be evaluated based on the probability value of each user in the first user set participating in the event to be evaluated and second data corresponding to each user in the first user set, wherein the second data is data indicating whether each user in the first user set participated in the event to be evaluated within the first preset time period;

[0012] The evaluation result determination unit is used to determine the evaluation result of the event to be evaluated based on the weight value of each user in the first user set in the evaluation of the event to be evaluated, the second data corresponding to each user in the first user set, and the third data corresponding to each user in the first user set, wherein the third data is data indicating whether each user in the first user set has been lost within a second preset time period, the second preset time period is a time period adjacent to the first preset time period, and the first preset time period is earlier than the second preset time period.

[0013] An embodiment of the present application further provides an electronic device, comprising: a memory and a processor;

[0014] The memory is used to store one or more computer instructions;

[0015] The processor is configured to execute the one or more computer instructions to implement the above method.

[0016] An embodiment of the present application also provides a computer-readable storage medium on which one or more computer instructions are stored, and the instructions are executed by a processor to implement the above method.

[0017] Compared with the prior art, the evaluation method provided by this application calculates the probability value of each user in the user set participating in the event to be evaluated, and then obtains the weight value of each user in the evaluation of the event to be evaluated through the probability value of each user participating in the event to be evaluated and the true value (first data) of each user participating in the event to be evaluated. This weight value is used as a factor in the subsequent evaluation process, avoiding the problem of inaccurate evaluation results caused by individual differences between users. The evaluation method provided by this application introduces the concept of user weight value, which dilutes the individual differences between users in the user set and improves the accuracy of the evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is an application system diagram of an evaluation method provided in an embodiment of the present application;

[0019] Figure 2 is a flowchart of the evaluation method provided in the first embodiment of the present application;

[0020] Figure 3 This is a flowchart of obtaining the probability value of a user participating in an event to be evaluated, provided by the first embodiment of the present application;

[0021] Figure 4 is a flowchart of the evaluation method provided in the second embodiment of the present application;

[0022] Figure 5 is a schematic structural diagram of an evaluation device provided in a third embodiment of the present application;

[0023] Figure 6 It is a structural diagram of an electronic device provided in the fourth embodiment of the present application. DETAILED DESCRIPTION

[0024] The following description sets forth many specific details to facilitate a thorough understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of the present application. Therefore, the present application is not limited to the specific implementations disclosed below.

[0025] With the rapid development of computer technology, online business applications such as online shopping and online entertainment have gradually replaced offline activities and become a new trend. Operators can increase the revenue of online business applications through refined operations. Such refined operations can include updating and improving the business applications themselves, or integrating other application projects into business applications. For example, optimizing game scenes to attract more players to participate in the game, or injecting coupons into online shopping applications to attract more consumers to purchase.

[0026] Among online business applications, online games are a rapidly growing sector. Currently, online games are becoming increasingly large and engaging. Game operators can increase revenue from online business applications through refined game operations. For example, they can recommend gift packs to players within the game application and generate additional revenue through their purchases.

[0027] However, not every refined operation project has a positive impact on business applications. For example, recommending a prop gift pack to players in a game may increase game revenue for a period of time, but due to excessive player churn, revenue will drop sharply in the subsequent period. Therefore, it is very necessary to evaluate the effectiveness of refined operation projects. Accurate evaluation results can enable operators to adjust refined operation strategies in a timely manner to achieve better operational results. User retention is an important indicator for evaluating the effectiveness of refined operation projects. User retention can be understood as "how many users stay." User retention fully reflects the quality of refined operation projects and their ability to retain users.

[0028] In the existing technology, there are mainly two evaluation methods for refined operation projects:

[0029] First, the AB experiment evaluation method, that is, evaluating the impact of refined operation projects on user retention through AB experiments. In the AB experiment, the observed users will be randomly divided into an experimental group and a control group, so that the users in the experimental group and the control group can freely participate in business applications. The difference is that refined operation projects are launched to the users in the experimental group. After a period of operation, the indicator data (such as retention data) of the users in the experimental group and the control group are collected, and then the effect of the refined operation project is evaluated by comparing the indicator data of the two (for example, the impact of the refined operation project on user retention is evaluated). When using AB experiments to evaluate the effect of refined operation projects, there is often no obvious difference between the experimental group data and the control group data. This is mainly because the number of users in the experimental group who participate in the refined operation project is too small, which dilutes the impact of the refined operation project on user retention.

[0030] Second, the experimental group self-evaluation method, that is, only the data of the experimental group users in the AB experiment is used to evaluate the impact of the refined operation project on user retention. In other words, by comparing the data of users in the experimental group who participated in the refined operation project with the data of users who did not participate in the refined operation project, the effectiveness of the refined operation project is evaluated. This method generally results in better retention of users who participated in the refined operation project, but because there are large individual differences between users who participated in the refined operation project and users who did not participate in the refined operation project (for example, users who participated in the refined operation project have stronger purchasing power), the comparability between the two is not strong, and therefore the conclusion lacks accuracy.

[0031] In summary, the existing evaluation methods for refined operations are unscientific and lack accuracy due to the small number of users participating in refined operations projects and the large individual differences between users.

[0032] In response to the above problems, the present application provides a more scientific evaluation method, which uses the user data of the experimental group users in the AB experiment to evaluate the impact of the event to be evaluated on user retention. The method calculates the probability value of each user in the user set participating in the event to be evaluated, and then obtains the weight value of each user in the evaluation of the event to be evaluated through the probability value of each user participating in the event to be evaluated and the actual value of each user participating in the event to be evaluated. The weight value is used as a factor in the subsequent evaluation process, which solves the problem of inaccurate evaluation results caused by individual differences between users.

[0033] The evaluation method, device, electronic device, and computer-readable storage medium described in this application are further described in detail below with reference to specific embodiments and accompanying drawings.

[0034] Figure 1 This is an application system diagram of an evaluation method provided in an embodiment of the present application. Figure 1As shown, the system includes a user terminal 101 and a server terminal 102. The user terminal 101 is connected to the server terminal 102 for communication via a network. The user terminal 101 can be multiple, and is a terminal device for users to participate in business applications and refined operation projects. Each of the user terminals 101 can be a touch terminal, such as a smart phone, tablet computer, personal digital assistant (PDA) and other devices; it can also be a computer terminal, such as a laptop computer, desktop computer and other devices. The server terminal 102 is used to deploy business applications and refined operation projects for the user terminal 101, and multiple users participate in the business applications and the refined operation projects through multiple user terminals 101. The server terminal 102 is also deployed with the evaluation method provided by the present invention, and evaluates the impact of the refined operation project on user retention based on the user data sent by multiple user terminals 101. The server terminal 102 can be a device or electronic device deployed with the evaluation method, or it can be a server deployed with the evaluation method.

[0035] The first embodiment of the present application provides an evaluation method for evaluating the impact of a refined operation project on user retention.

[0036] Figure 2 This is a flow chart of the evaluation method provided in this embodiment. Figure 2 The evaluation method provided in this embodiment is described in detail. The embodiments described below are used to explain the technical solution of this application and are not intended to be limiting for actual use.

[0037] like Figure 2 As shown, the evaluation method provided in this embodiment includes the following steps:

[0038] Step S201: Determine the probability value of each user in the first user set participating in the event to be evaluated based on the first data corresponding to each user in the first user set, wherein the first data is the data generated by each user in the first user set within a first preset time period for at least one influencing factor that affects the user's participation in the event to be evaluated.

[0039] The first user set can be understood as a set consisting of at least a portion of all users participating in the evaluation of the event to be evaluated. All users participating in the evaluation of the event to be evaluated can be considered an experimental group, with at least a portion of the users in the experimental group forming the first user set. For example, if the server launches a recommended gift package in a game involving N players, then these N players will constitute the experimental group of players evaluating the recommended gift package, and at least a portion of these players will constitute a player set.

[0040] The first data can be understood as the data generated by each user in the first user set within the first preset time period regarding the various influencing factors that affect the user's participation in the event to be evaluated. It is a multidimensional data group, and each dimension includes an influencing factor and the data generated by the user regarding the influencing factor within the first preset time period.

[0041] The first preset period is any period after the operator places the event to be evaluated into the business application. It is the time period for obtaining the user data required to evaluate the event to be evaluated. The length of this period can be one week, one month, etc., and is specifically pre-set by the operator based on the characteristics of the business application and the event to be evaluated. The longer the first preset period, the more user data will be obtained, and the more accurate the evaluation results will be. However, a longer period will affect the efficiency and effectiveness of the evaluation, hindering the operator from timely adjusting and refining its operational strategies. Therefore, it is very necessary to determine an appropriate period length based on the business application and the characteristics of the event to be evaluated. For example, if the event to be evaluated is a recommended gift package item placed in a game, and the game is updated once a week, the length of the first preset period can be set between 7 and 15 days. This will both obtain the required amount of data and shorten the evaluation cycle. If the game's top-up is heavily dependent on the recommended gift package item, and the recommended gift package item is preset to be active for 30 days, the length of the first preset period can be set to at least 30 days to obtain the required amount of data.

[0042] The events to be evaluated can be understood as refined operational projects designed by the operator for business applications. For example, gift pack recommendations for games, coupons for online shopping, and so on. These refined operational projects require an effectiveness evaluation before they are officially implemented for business applications. This allows the operator to optimize these refined operational projects based on the evaluation results, ultimately achieving the goal of refined operations.

[0043] Whether a user participates in a refined marketing campaign launched by an operator is not only related to the campaign itself, but also to the user itself. For example, whether a player purchases a recommended gift pack offered by a game operator depends not only on the package itself, such as whether the virtual items in the pack are appealing to the player and whether they increase their attack power, but also on the player's purchasing power, such as their purchasing power, recent player activity, and preferred game items. Because gift packs are designed by the game operator, their rationale is determined through evaluation and testing, and each player's purchasing power and activity level vary. Therefore, among the factors influencing a player's purchase of a gift pack, the gift pack factor is equivalent to a quantitative factor, while the player factor is equivalent to a variable. Player factors are a significant factor influencing whether a player purchases a gift pack. Due to individual differences, the data corresponding to player factors will vary, and their propensity to purchase gift packs will also vary.

[0044] This embodiment provides an optional method for calculating the propensity of different users to participate in the event to be evaluated, that is, the propensity score matching method. The propensity score matching method (PSM) is a type of statistical method that uses observational data to analyze the intervention effect, and is used to process data from observational studies. In observational studies, there are often many confounding variables, and the use of the propensity score matching method can reduce the impact of confounding variables so as to obtain more reasonable conclusions. In this embodiment, the influencing factors that affect the user's participation in the event to be evaluated are confounding variables, and there are a large number of them, which can include all factors that can affect the user's participation in the event to be evaluated. Based on the propensity score matching method, the server can obtain the propensity of different users to participate in the event to be evaluated according to the numerical values ​​of multiple influencing factors corresponding to different users. The use of the propensity score matching method is equivalent to replacing multiple confounding variables with one propensity score variable, which simplifies the expression of individual differences between users.

[0045] In this embodiment, the numerical value corresponding to the user's propensity to participate in the event to be evaluated is defined as the probability value of the user's participation in the event to be evaluated. A larger probability value indicates a greater propensity for the user to participate in the event to be evaluated, and a smaller probability value indicates a greater propensity for the user not to participate in the event to be evaluated.

[0046] Propensity score matching includes a variety of implementation methods, such as logistic regression, probabilistic regression, and machine learning, all of which can be used to calculate the probability of a user participating in the event to be evaluated. Regardless of the method, all involve the construction of a propensity model. Logistic regression, probabilistic regression, and machine learning can provide a model framework for the propensity model, but building a usable propensity model requires a certain amount of training data. Therefore, before determining the probability of each user in the first user set participating in the event to be evaluated based on the first data corresponding to each user in the first user set, a series of steps are involved, including obtaining training data and establishing a propensity model.

[0047] Based on this, this embodiment provides a specific implementation method for obtaining the probability value of a user participating in an event to be evaluated. Figure 3 This is a flowchart of obtaining the probability value of a user participating in an event to be evaluated, provided by this embodiment.

[0048] like Figure 3 As shown, the method for obtaining the probability value of a user participating in an event to be evaluated provided by this embodiment includes at least the following steps:

[0049] Step S201 - 1 : placing the event to be evaluated into a business application in a third preset time period, where the third preset time period is earlier than the first preset time period.

[0050] During the third preset period, the server will place the event to be evaluated into the business application. After placement, users can choose to participate in the event to be evaluated. For example, during the third preset period, the server may place a recommended gift package item into the game. After placement, players can choose to purchase the gift package.

[0051] The third preset period is a period set by the operator to put the event to be evaluated into the business application. The server puts the event to be evaluated into the business application so that users can independently participate in the event to be evaluated. Therefore, the third preset period must be earlier than the first preset period.

[0052] In reality, the server delivers the pending event at a specific point in time, not a specific time period. However, the operator can pre-set a time period for the pending event and set a delivery condition for the business application. Within that time period, the server can deliver the pending event if it determines the delivery condition is met. For example, if the number of users participating in a business application is a delivery condition, the server will immediately deliver the pending event to the business application if the number of users participating in the business application reaches the set delivery condition within the third pre-set time period.

[0053] Step S201 - 2 : Divide at least part of the users participating in the business application into the first user set and the second user set according to a preset ratio.

[0054] After the server places the event to be evaluated into the business application, it can use all or part of the users participating in the business application as the experimental group for evaluating the event to be evaluated. The experimental group includes users who participated in the event to be evaluated and users who did not. Furthermore, the server can divide the users in the experimental group into two parts according to a pre-set ratio, with one part of the users forming a first user set and the other part forming a second user set. The user data of the first user set will be used to evaluate the event to be evaluated, while the user data of the second user set will be primarily used to train the propensity model. Therefore, the first user set can also be understood as the prediction group, and the second user set can also be understood as the training group.

[0055] Users can be divided randomly or based on certain characteristics. For example, if user gender is used as a characteristic to divide user sets, the male-female ratio in the first and second user sets will be relatively consistent. Another example is if user age is used as a characteristic to divide user sets, the age distribution of users in the first and second user sets will be relatively consistent. Clearly, dividing users based on certain characteristics can reduce certain differences between the two user sets to a certain extent. The specific division method is not limited here.

[0056] The user division ratio can generally be determined based on the number of users, as long as the amount of user data in the second user set is sufficient for training the propensity model. Therefore, when the total number of users is large enough, users can be divided according to a larger division ratio to ensure more user data is available for evaluating the subject to be evaluated. This embodiment provides an optional division ratio of 7:3, that is, the users in the experimental group are divided into the first user set and the second user set in a ratio of 7:3.

[0057] Step S201-3, collect the first data and the second data generated by each user in the first user set within the first preset time period, and the third data generated within the second preset time period, and establish a corresponding relationship between each user in the first user set and the first data, the second data and the third data.

[0058] After the experimental group users are divided according to a preset ratio, data generated by all users in each user set within a preset period needs to be collected. For the first user set, the user data to be collected includes the first data and the second data generated by the users within the first preset period.

[0059] As mentioned above, the first data refers to the data generated by each user in the first user set within the first preset time period regarding the various influencing factors that affect the user's participation in the event to be evaluated. It is a multidimensional data group, each dimension including an influencing factor and the data generated by the user regarding the influencing factor within the first preset time period.

[0060] The second data can be understood as the data generated by each user in the first user set during the first preset time period regarding participation in the event to be evaluated. The data can be represented in the form of either 0 or 1, where 0 can be used to indicate that the user did not participate in the event to be evaluated during the first preset time period, and 1 can be used to indicate that the user participated in the event to be evaluated during the first preset time period.

[0061] In addition, for the first user set, the user data that needs to be collected also includes third data generated by the users within the second preset time period.

[0062] The second preset period refers to the period after the first preset period and adjacent to the first preset period. It can be understood as a stage in which users may churn after a refined operation project is launched into a business application and users have participated in the business application under refined operation for a period of time. The length of the second preset period is set in advance by the operator based on the characteristics of the business application. Generally, the length of the second preset period is set to 7 days or 15 days, that is, the 7-day churn of users or the 15-day churn of users is used as an indicator for evaluating the event to be evaluated. Assume that the first preset period is 30 days, the starting time is September 1, and the second preset period is 7 days. Then, the first preset period is from September 1 to September 31, and the second preset period is from October 1 to October 7. Setting the second preset period to a longer time will affect the efficiency of the evaluation on the one hand, and on the other hand, it will cause inaccurate evaluation results due to more interference factors.

[0063] The second preset time period mainly obtains data on whether the user has churned. Therefore, the third data can be understood as the churn data generated by each user in the first user set during the second preset time period. The data can also be expressed in the form of either 0 or 1, where 0 can be used to indicate that the user has not churned during the second preset time period, and 1 can be used to indicate that the user has churned during the second preset time period.

[0064] After data collection, it is necessary to establish a correspondence between each data and the user, for example, the user identifier or user number of each user in the first user set and each data are recorded as a data table. This embodiment provides an optional way to establish a correspondence between data and users, as shown in Table 1:

[0065] Table 1 User data table of the first user set

[0066]

[0067]

[0068] Each row in the user data table records all the data corresponding to a user, including the data group of the first data and the second data generated by the user in the first preset time period, and the third data generated by the user in the second preset time period. The server can call any data in the user data table according to the subsequent evaluation process.

[0069] Step S201-4, collect the fourth data and fifth data generated by each user in the second user set within the first preset time period, and establish a correspondence between each user in the second user set and the fourth data and the fifth data, wherein the fourth data is the data generated by each user in the second user set within the first preset time period for at least one of the influencing factors, and the fifth data is the data indicating whether each user in the second user set participated in the event to be evaluated within the first preset time period.

[0070] In addition to collecting user data from each user in the first user set, user data from each user in the second user set also needs to be collected. Since the second user set is a training set, this user data is used to train the propensity model. As previously mentioned, the data required for training the propensity model primarily involves data generated by users during the first preset period regarding various factors influencing their participation in the event to be evaluated, as well as data generated by users during the first preset period regarding whether they participated in the event to be evaluated. Therefore, for the second user set, only the data generated by each user during the first preset period can be collected. The data generated by each user in the second user set regarding multiple factors during the first preset period is defined as fourth data, and the data on whether each user in the second user set participated in the event to be evaluated during the first preset period is defined as fifth data. In other words, for the second user set, the fourth and fifth data generated by each user during the first preset period need to be collected. The data type of the fourth data is the same as that of the first data: a multidimensional data set, with each dimension consisting of an influencing factor and the data generated by the user regarding that factor during the first preset period. The data type of the fifth data is the same as that of the second data, and can be represented in the form of either 0 or 1, where 0 can be used to indicate that the user did not participate in the event to be evaluated within the first preset time period, and 1 can be used to indicate that the user participated in the event to be evaluated within the first preset time period.

[0071] Similarly, after data collection, it is necessary to establish a correspondence between each data and the user. For example, the user identifier or user number of each user in the second user set and each data are recorded as a data table, as shown in Table 2:

[0072] Table 2 User data table of the second user set

[0073]

[0074] Each row in the user data table records all data corresponding to a user, including a data group of fourth data and fifth data generated by the user in the first preset time period.

[0075] The fourth data and the first data are data generated by users in the second user set and users in the first user set for the same multiple influencing factors. The fifth data and the second data are both data generated for whether the user participated in the event to be evaluated. Therefore, the fourth data and the fifth data corresponding to each user in the second user set are used as training data. The tendency model obtained by training is a model that can be used by the user data of the first user set.

[0076] Step S201 - 5 , establishing a first model using the fourth data and the fifth data as training data.

[0077] The first model is a propensity model, which is used to calculate the probability value of each user in the first user set participating in the event to be evaluated.

[0078] This embodiment provides a method for constructing a first model, that is, using a logistic regression function as a basic framework of the first model and using the fourth data and the fifth data as training data to construct the first model.

[0079] The expression of the logistic regression function is as follows:

[0080] Y=β0+β1X1+β2X2+……+β n X n

[0081] Among them, X1, X2, ..., X n represents the cause variable, Y represents the result variable, n represents the number of cause variables, β1, β2, ..., β n Indicates the weight value of each cause variable.

[0082] In the method provided in this embodiment, the server retrieves the fourth and fifth data corresponding to each user in the second user set, uses the fourth data as the cause variable, and the fifth data as the result variable, and inputs them into the above expression according to the corresponding relationship to calculate the weight value corresponding to each cause variable in the above expression. In other words, the β0, β1, β2, ..., β in the above expression are calculated using the fourth and fifth data. n , thus obtaining a function that can get Y by inputting X.

[0083] Of course, the first model can also be established based on a probabilistic regression function or a neural network model, and the specific implementation method is not limited here.

[0084] Step S201-6: Input the first data corresponding to each user in the first user set into the first model in sequence, obtain the sixth data corresponding to each user in the first user set output by the first model in sequence, and use the sixth data as the probability value of each user in the first user set participating in the event to be evaluated.

[0085] After obtaining the first model, the server can call out the first data corresponding to each user in the first user set, use the first data as input data, input it into the first model, and obtain a data output by the first model. In this embodiment, the data output from the first model is defined as the sixth data, which is the probability value of each user in the first user set participating in the event to be evaluated.

[0086] Still taking the above logistic regression function as an example, the β0, β1, β2, ..., β in the logistic regression function expression are calculated by the fourth data and the fifth data. n The server calls the first data corresponding to each user in the first user set and inputs it into the function as X to obtain the Y value corresponding to each user. The Y value corresponding to each user is the probability value of each user in the first user set participating in the event to be evaluated.

[0087] Through the above steps, individual differences among different users, that is, differences among different users in multiple influencing factors, can be converted into probability values ​​of users participating in the event to be evaluated.

[0088] Step S202: Determine the weight value of each user in the first user set in the evaluation of the event to be evaluated based on the probability value of each user in the first user set participating in the event to be evaluated and the second data corresponding to each user in the first user set, wherein the second data is data indicating whether each user in the first user set participated in the event to be evaluated within the first preset time period.

[0089] Through step S201, the probability value of each user in the first user set participating in the event to be evaluated is obtained, and the second data is data indicating whether each user in the first user set participated in the event to be evaluated within the first preset time period. It can be understood that the second data is the true value of each user in the first user set participating in the event to be evaluated.

[0090] Based on the difference between the true value and the probability value, we can calculate each user's influence in the evaluation of the event to be evaluated, that is, the user's weight in the evaluation of the event to be evaluated. In fact, calculating each user's weight in the evaluation of the event to be evaluated by using the true value of each user's participation in the event to be evaluated and the probability value of each user's participation in the event to be evaluated is a process of re-collecting user samples, expanding the sample of users with high influence and shrinking the sample of users with low influence.

[0091] This embodiment provides an optional method for calculating user weights, namely, inverse probability weighting (IPW). The inverse probability weighting (IPW) method is a method used to account for missing and selection bias caused by non-randomly selected observations or non-random missing population information. This method can modify the analysis by weighting the observations to give them a probability of being selected.

[0092] In this embodiment, the expression of the inverse probability weighting method can be shown as follows:

[0093]

[0094] Among them, IPW represents the user weight, bought represents the actual value of the user's participation in the event to be evaluated, and propensity represents the probability value of the user's participation in the event to be evaluated.

[0095] The actual value of the user's participation in the event to be evaluated is the second data, and the probability value of the user's participation in the event to be evaluated is the data obtained in step S201. The server inputs these two data into the above expression to output the user's weight in the evaluation of the event to be evaluated. The specific process is as follows: the server inputs the probability value and the second data corresponding to each user in the first user set into the first function in sequence, obtains the seventh data corresponding to each user in the first user set, and uses this seventh data as the weight value of each user in the first user set in the evaluation of the event to be evaluated.

[0096] The first function is the function corresponding to the inverse probability weighting method, which can be expressed by the above expression.

[0097] The seventh data can be understood as the data output by the first function after the probability value and the second data are input into the first function, that is, the weight value of the user in evaluating the event to be evaluated.

[0098] From the expression of the first function above, we can see that when the bought value corresponding to user A is 1 (i.e., user A participated in the event to be evaluated), if the propensity corresponding to user A is relatively high (i.e., user A is more likely to participate in the event to be evaluated), then the IPW value corresponding to user A will be relatively small, that is, user A's influence in evaluating the event to be evaluated will be relatively small. If the propensity corresponding to user B is relatively low (i.e., user B is less likely to participate in the event to be evaluated), then the IPW value corresponding to user B will be relatively large, that is, user B's influence in evaluating the event to be evaluated will be relatively large.

[0099] The weight of the user in evaluating the event to be evaluated can be understood as follows:

[0100] If the probability value of the user participating in the event to be evaluated is large, and the user actually participates in the event to be evaluated, then the user's influence in evaluating the event to be evaluated will be relatively small, and its weight value will be relatively small.

[0101] If the probability value of a user participating in the event to be evaluated is large, but the user did not actually participate in the event to be evaluated, then the possibility that the user did not participate in the event to be evaluated due to the event to be evaluated itself will be relatively large. The user should have a greater influence in the evaluation of the event to be evaluated, and its weight value will be larger.

[0102] If the probability value of the user participating in the event to be evaluated is small, and the user has not actually participated in the event to be evaluated, then the user's influence in evaluating the event to be evaluated will be relatively small, and its weight value will be relatively small.

[0103] If the probability value of a user participating in the event to be evaluated is small, but the user actually participates in the event to be evaluated, then the possibility that the user participates in the event to be evaluated because of the event to be evaluated itself will be relatively high, and the user should have a greater influence in the evaluation of the event to be evaluated, and its weight value will be larger.

[0104] Users with larger weights will account for a larger proportion of subsequent evaluations, while users with smaller weights will account for a smaller proportion. For example, if User A's IPW is 0.5, then User A will only act as half a user in the evaluation of the event being evaluated. If User B's IPM is 2, then User B will act as two users in the evaluation of the event being evaluated. This approach balances user bias and addresses the issue of inaccurate evaluation results caused by individual user differences.

[0105] Step S203: Determine the evaluation result of the event to be evaluated based on the weight value of each user in the first user set in the evaluation of the event to be evaluated, the second data corresponding to each user in the first user set, and the third data corresponding to each user in the first user set, wherein the third data is data indicating whether each user in the first user set has been lost within a second preset time period, the second preset time period is a time period adjacent to the first preset time period, and the first preset time period is earlier than the second preset time period.

[0106] Through steps S201 and S202, the weight value of each user in the first user set in the evaluation of the event to be evaluated is obtained. Then, based on the weight value, the second data corresponding to the user, and the third data corresponding to the user, the evaluation result of the event to be evaluated can be obtained. Because the second data is the data corresponding to whether the user participated in the event to be evaluated, and the third data is the data corresponding to whether the user retained, the evaluation result of the event to be evaluated is the impact of the event to be evaluated on user retention. Specifically, it includes the following steps:

[0107] Step S203 - 1 : Input the weight value, the second data, and the third data corresponding to each user in the first user set into a second model to obtain eighth data output by the second model.

[0108] The second model can also be a model corresponding to methods such as logistic regression, probabilistic regression, and machine learning. The input of this model is the weight value, second data, and third data corresponding to each user in the first user set, and the output is an impact value, which is the impact of the event to be evaluated on user retention. The server extracts the weight value, second data, and third data corresponding to each user in the first user set and inputs them into the second model. The second model then outputs a data. In this embodiment, the data output by the second model is defined as the eighth data, which is used to evaluate the impact of the event to be evaluated on user retention.

[0109] Step S203-2: determining an evaluation result of the event to be evaluated based on the eighth data, specifically: if the eighth data is positive and statistically significant, then the event to be evaluated has a negative impact on user retention.

[0110] Since in the second data, 1 indicates that the user participated in the event to be evaluated, and 0 indicates that the user did not participate in the event to be evaluated, in the third data, 1 indicates user churn, and 0 indicates that the user did not churn, and the eighth data output by the second model is data for evaluating the impact of the event to be evaluated on user retention, therefore, if the eighth data is positive, it means that the event to be evaluated will cause user churn, and the larger the eighth data is, the more serious the user churn caused by the event to be evaluated will be. If the eighth data is negative, it means that the event to be evaluated will not cause user churn.

[0111] Statistical significance refers to the level of risk associated with rejecting the null hypothesis if it is true; that is, the probability level or significance level. It can also be understood as the probability of error in estimating a population parameter falling within a certain interval. Statistically significant means that the result is genuine and not due to chance, and can be measured by the p-value. In this example, a p-value of 0.05 or less indicates that the eighth data is statistically significant and reliable.

[0112] The operator can adjust the refined operation project based on the actual situation of Eighth Data so that the project can increase the revenue of business applications without having a negative impact on user retention.

[0113] The above-mentioned first embodiment provides a method for evaluating the impact of the event to be evaluated on user retention. The method calculates the probability value of each user in the first user set participating in the event to be evaluated, and then obtains the weight of each user in the evaluation of the event to be evaluated through the probability value of each user participating in the event to be evaluated and the actual value of each user participating in the event to be evaluated. The weight is used as a factor in the subsequent evaluation process, avoiding the problem of inaccurate evaluation results caused by individual differences between users.

[0114] The method provided in this embodiment is described in detail in various possible implementations in the first embodiment. Of course, the examples provided in this embodiment are only for facilitating understanding of the method described herein and are not intended to be limiting. The evaluation method provided in this application includes but is not limited to the implementation provided in the first embodiment of this application.

[0115] The second embodiment of the present application provides an evaluation method for evaluating the impact of refined operation projects in game applications on player retention.

[0116] In this embodiment, the event to be evaluated includes the event of recommending a gift package to a player in a game application. The evaluation method includes:

[0117] First, based on the first data corresponding to each user in the first user set, determine the probability value of each user in the first user set purchasing the gift package, wherein the first data is the data generated by each user in the first user set within a first preset time period for at least one influencing factor that affects the player's purchase of the gift package.

[0118] Second, based on the probability value of each user in the first user set purchasing the gift package and the second data corresponding to each user in the first user set, determine the weight value of each user in the first user set in evaluating the recommended gift package event, wherein the second data is data indicating whether each user in the first user set purchases the gift package within the first preset time period.

[0119] Third, the evaluation result of the recommended gift package event is determined based on the weight value of each user in the first user set in the evaluation of the recommended gift package event, the second data corresponding to each user in the first user set, and the third data corresponding to each user in the first user set, wherein the third data is data indicating whether each user in the first user set has been lost within a second preset time period, the second preset time period is a time period adjacent to the first preset time period, and the first preset time period is earlier than the second preset time period.

[0120] The following describes in detail the method for evaluating the response of the recommended gift package project to player retention provided by this embodiment using a specific implementation method. Figure 4 is a flow chart of the evaluation method provided in this embodiment.

[0121] like Figure 4 As shown, the evaluation method provided in this embodiment includes the following steps:

[0122] Step S401: Place the "recommended gift package" item in the game.

[0123] The game operator designs a "recommended gift package" project for the game. The server will release the "recommended gift package" project into the game based on the release time set by the game operator. After the release, players can choose whether to purchase the gift package.

[0124] Step S402: Divide at least part of the players participating in the game into a first player group and a second player group according to a preset ratio.

[0125] The players who participate in the game include players who have participated in the game before the "recommended gift package" project is launched and players who participate in the game after the "recommended gift package" project is launched.

[0126] The server will use all or a portion of the players participating in the game as an experimental group for evaluating the "recommended gift pack" project. This experimental group includes players who have purchased the gift pack and those who have not. The server can further divide the players in the experimental group into two groups according to a pre-set ratio: one group will form the first player group, and the other group will form the second player group. The player data of the first player group will be used to evaluate the "recommended gift pack" project, while the player data of the second player group will be primarily used to train the model. Therefore, the first player group can also be considered the prediction group, and the second player group can also be considered the training group.

[0127] Players can be divided randomly or based on certain characteristics. For example, if gender is used as a characteristic to divide the player groups, the male-female ratio in the first and second player groups will be relatively consistent. Another example is if age is used as a characteristic to divide the player groups, the age distribution of players in the first and second player groups will be relatively consistent. Obviously, dividing players based on certain characteristics can reduce certain differences between the two player groups to a certain extent. The specific division method is not limited here.

[0128] The player division ratio can generally be determined based on the player base. It's sufficient for the player data in the second player group to meet the model training requirements. Therefore, when the player base is large enough, players can be divided according to a larger division ratio to ensure more player data is available for evaluating the "recommended gift package" project. This embodiment provides an optional division ratio of 7:3, that is, the players in the experimental group are divided into a prediction group (the first player group) and a training group (the second player group) in a ratio of 7:3.

[0129] Step S403 collects first data and second data generated by each player in the first player group during the first preset time period, as well as third data generated during the second preset time period, and establishes a correspondence between each player in the first player group and the first data, the second data, and the third data. The first data is data generated by each player in the first player group during the first preset time period for at least one factor influencing the purchase of the gift package by the player; the second data is data indicating whether each player in the first player group purchased the gift package during the first preset time period; and the third data is data indicating whether each player in the first player group churned during the second preset time period.

[0130] The first preset period is any period of time after the game operator launches the "Recommended Gift Pack" into the game. This period can range from one week to one month, and is specifically set by the game operator based on the characteristics of the game and the "Recommended Gift Pack." A longer first preset period allows for more player data to be collected, resulting in more accurate evaluation results. However, this will undoubtedly reduce evaluation efficiency and impact the effectiveness of the results, hindering the game operator's ability to adjust and refine operational strategies based on the evaluation results. Therefore, determining an appropriate period length is crucial.

[0131] The second preset period refers to the period after and adjacent to the first preset period. This can be understood as the period after players have participated in the game under the "recommended gift package" after the "recommended gift package" item is released, during which player churn is counted. The length of the second preset period is set in advance by the game operator based on the characteristics of the game, and is generally set to 7 or 15 days. Setting the second preset period longer can affect evaluation efficiency and lead to inaccurate evaluation results due to more interference factors.

[0132] The data collected during the first preset time period includes data generated by players regarding factors influencing the players to purchase the gift package, that is, first data.

[0133] Factors influencing a player's purchase of the gift pack may include player attributes and purchasing power, such as the player's preferred class, preferred game mode or dungeon, player activity level, preferred game items, and other factors. Another example is the player's total top-up amount, frequency of top-up, and recent top-up history. These player attributes and purchasing power will influence whether a player purchases the gift pack. For example, if a player's preferred game item is a rocket launcher, but the recommended gift pack doesn't include one, the player is unlikely to purchase the gift pack. Another example is if a player hasn't topped up recently and their top-up balance is low, the player's likelihood of purchasing the gift pack is also low. Therefore, by using the data corresponding to these influencing factors, it is possible to determine the probability of each player purchasing the gift pack.

[0134] Undoubtedly, a player's top-up is the most important condition for purchasing a gift pack; players who haven't topped up cannot purchase a gift pack. Based on this, this embodiment provides an optional implementation of the first data, namely, the first data includes at least one of the following: the player's top-up frequency within the first preset period, the player's total top-up amount within the first preset period, and the time of the player's last top-up within the first preset period.

[0135] The data collected during the first preset time period also includes real data on whether the player purchases the gift package, that is, the second data.

[0136] The second data is data indicating whether each player in the first player group purchased the gift package within the first preset time period. This embodiment provides an optional implementation form of the second data, that is, the second data is 0 or 1, wherein 0 indicates that the player did not purchase the gift package within the first preset time period, and 1 indicates that the player purchased the gift package within the first preset time period.

[0137] The data collected during the second preset period includes real data on whether the player has churned, ie, the third data.

[0138] The third data indicates whether each player in the first player group has churned within the second preset time period. This embodiment provides an optional implementation of the third data, that is, the third data is 0 or 1, where 0 indicates that the user has not churned within the second preset time period, and 1 indicates that the user has churned within the second preset time period.

[0139] After data collection, it is necessary to establish a corresponding relationship between each piece of data and the player to facilitate its retrieval and use in subsequent calculations. This embodiment provides an optional method for establishing a corresponding relationship between data and players. That is, the player identifier or player number of each player in the first player group and the player data of the player are recorded as a row in the data table in the form of a data table, thereby establishing a corresponding binding relationship between the player and the player data. The data table is shown in Table 3:

[0140] Table 3 Player data table of the first player group

[0141]

[0142]

[0143] As shown in Table 3, taking player number 3 as an example, the data table is explained. The first data generated by player number 3 during the first preset period includes: a total of 3 recharges during the first preset period, with a total recharge amount of 500 yuan, and the last recharge occurred on the last day of the first preset period. The second data generated by player number 3 during the first preset period is 1, indicating that the player purchased the gift package during the first preset period. The third data generated by player number 3 during the second preset period is 1, indicating that the player churned during the second preset period.

[0144] The server can call any data of any player from the data table according to the player number in the subsequent evaluation process.

[0145] Step S404: Collect the fourth data and fifth data generated by each player in the second player group during the first preset time period, and establish a correspondence between each player in the second player group and the fourth data and the fifth data, wherein the fourth data is the data generated by each player in the second player group during the first preset time period for at least one of the influencing factors, and the fifth data is the data indicating whether each player in the second player group purchased the gift package during the first preset time period.

[0146] The server collects player data of each player in the second player group during the first preset time period, including data generated by factors influencing the players to purchase the gift package, i.e., the fourth data, and also includes real data on whether the players purchase the gift package, i.e., the fifth data.

[0147] The fourth data has the same data type as the first data and is a multi-dimensional data set, where each dimension includes an influencing factor and the data generated by the player in response to the influencing factor during the first preset time period. The fifth data has the same data type as the second data and can be represented by 0 or 1, where 0 indicates that the player did not purchase a gift pack during the first preset time period, and 1 indicates that the player did purchase a gift pack during the first preset time period.

[0148] Similarly, after data collection, it is necessary to establish a correspondence between each data and the player. For example, the player ID or player number of each player in the second player group and the player data corresponding to the player are recorded in a data table, as shown in Table 4:

[0149] Table 4 Player data table of the second player group

[0150]

[0151] A row in the data table records all data corresponding to a player, including a data group of fourth data and fifth data generated by the player in a first preset time period.

[0152] Step S405 , using the fourth data and the fifth data as training data to establish a tendency model.

[0153] The propensity model is a model used to calculate the player's purchasing propensity value for gift packages (i.e., the possibility of purchasing gift packages). The model can be a neural network model, or a logistic regression model, a probabilistic regression model, or any other model.

[0154] Step S403 collects the first and second data generated by each player in the first player group during the first preset period. Step S404 collects the fourth and fifth data generated by each player in the second player group during the first preset period. Because the fourth and first data are generated based on the same multiple influencing factors, and the fifth and second data both represent whether a player purchases a gift package, using the fourth and fifth data corresponding to each player in the second player group as training data, a propensity model capable of calculating the likelihood of each player in the first player group purchasing a gift package can be trained.

[0155] In this embodiment, a logistic regression model is used as an example to illustrate the tendency model.

[0156] The expression of the logistic regression model is as follows:

[0157] Y=β0+β1X1+β2X2+β3X3

[0158] Among them, X1, X2, and X3 respectively represent the recharge frequency, total recharge amount, and last recharge time of each player in the second player group during the first preset period. Y represents whether each player in the second player group purchased a gift package during the first preset period. β1, β2, and β3 represent the weight values ​​of the three influencing factors.

[0159] When training the above model, X1, X2, X3 and Y are input data, β1, β2, β3 are output data, X1, X2, X3 correspond to the fourth data in Table 4, and Y corresponds to the fifth data in Table 4. After repeated training with player data corresponding to multiple player samples, the proportion of each influencing factor in calculating the player's tendency to purchase gift packages can be obtained.

[0160] Step S406: Determine the probability value of each player in the first player group purchasing the gift package based on the first data corresponding to each player in the first player group.

[0161] After obtaining the propensity model, the server calls the first data in Table 3 and inputs the first data corresponding to each player in the first player group into the propensity model in sequence. This obtains multiple data output by the propensity model, which are the probability values ​​of each player in the first player group purchasing the gift package.

[0162] Continuing to use the aforementioned logistic regression model as the propensity model, the fourth and fifth data are used to calculate β0, β1, β2, and β3 in the logistic regression model expression. The server then retrieves the first data corresponding to each player in the first player group from Table 3 and inputs it as X into the propensity model to obtain the Y value corresponding to each player. Each Y value is the probability of each player in the first player group purchasing the gift package.

[0163] Through the above steps, the differences among different players in multiple influencing factors can be converted into differences in the likelihood of players purchasing gift packages.

[0164] Step S407 , determining a weight of each player in the first player group in evaluating the recommended gift package item based on the probability value of each player in the first player group purchasing the gift package and the second data corresponding to each player in the first player group.

[0165] The probability value of each player in the first player group purchasing the gift package is obtained through step S406, and the second data can be understood as the actual value of each player in the first player group purchasing the gift package.

[0166] Based on the difference between the true value and the probability value, we can calculate each player's influence in the "Recommended Gift Pack" evaluation, that is, the player's weight in the "Recommended Gift Pack" evaluation. In fact, calculating each player's weight in the "Recommended Gift Pack" evaluation by comparing the true value of each player's gift pack purchase with the probability value of each player's gift pack purchase is a process of re-collecting the player sample, that is, expanding the sample of players with great influence and shrinking the sample of players with less influence.

[0167] In this embodiment, the inverse probability weighting method is used to calculate the player's weight value, and the calculation formula is as follows:

[0168]

[0169] Among them, IPW represents the player weight, bought represents the actual value of the player purchasing the gift package, and propensity represents the probability value of the player purchasing the gift package.

[0170] The actual value of the player purchasing the gift package is the second data, and the probability value of the player purchasing the gift package is the data obtained through the above steps. The server calls the second data and the probability value in the player data table shown in Table 3, and inputs these two data into the calculation formula of the inverse probability weighted method to output the player's weight value in the evaluation of the "recommended gift package" project.

[0171] From the above expression, we can see that the greater the difference between the actual value of a player purchasing a gift package and the probability value of the player purchasing the gift package, the greater the player's weight will be, and the smaller the difference between the actual value of a player purchasing a gift package and the probability value of the player purchasing the gift package, the smaller the player's weight will be. Specifically, it can include the following situations:

[0172] First, if the probability value of a player purchasing a gift package is high and the player actually purchases the gift package, then the player's influence in evaluating the "recommended gift package" project will be relatively small, and its weight value will be smaller.

[0173] Second, if the probability value of a player purchasing a gift package is high, but the player does not actually purchase the gift package, then the player's influence in evaluating the "recommended gift package" project will be relatively large, and its weight value will be relatively large.

[0174] Third, if the probability value of a player purchasing a gift package is small, and the player does not actually purchase the gift package, then the player's influence in evaluating the "recommended gift package" project will be relatively small, and its weight value will be relatively small.

[0175] Fourth, if the probability of a player purchasing a gift package is small, but the player actually purchases the gift package, then the player's influence in evaluating the "recommended gift package" project will be greater, and its weight value will be larger.

[0176] Players with larger weights will have a larger proportion in the subsequent evaluation calculations, while players with smaller weights will have a smaller proportion in the subsequent evaluation calculations. This approach can balance the problem of player bias and solve the problem of inaccurate evaluation results caused by individual differences among players.

[0177] Step S408: Determine the impact of the recommended gift package on player retention based on the weight of each player in the first player group in evaluating the recommended gift package item, the second data corresponding to each player in the first player group, and the third data corresponding to each player in the first player group.

[0178] In this embodiment, the server calls the second data and third data corresponding to each player in the first player group in the player data table shown in Table 3, builds a model through a logistic regression model, and adjusts the proportion of each player's player data in the total player data according to the player weight value obtained in the above steps, so as to calculate the impact data of the recommended gift package on player retention.

[0179] Since in the second data, 1 indicates that the player purchased the gift box and 0 indicates that the player did not purchase the gift box, and in the third data, 1 indicates that the player churned and 0 indicates that the player did not churn, therefore, if the calculated impact data is positive and statistically significant, it means that the "recommended gift package" project will cause user churn and will have a negative impact on user retention.

[0180] Game operators can adjust the gift package content or gift package gameplay based on the evaluation results, so that the recommended gift package will increase game revenue without having a negative impact on player retention.

[0181] The above-mentioned second embodiment provides a method for evaluating the impact of refined operation projects in game applications on player retention. This method converts the differences among players in multiple influencing factors into differences in their propensity to purchase gift packages, and further converts the differences in the propensity of each player to purchase gift packages into the weight of the player in the evaluation. In this way, the player's own factors are incorporated into the evaluation of refined operation projects, the player attributes are balanced, and the problem of inaccurate evaluation results caused by individual differences between players is avoided.

[0182] The method provided in this embodiment is described in detail in a variety of possible implementations in the second embodiment. Of course, the examples provided in this embodiment are only for facilitating understanding of the method described in this application and are not intended to be limiting. The evaluation method provided in this application includes but is not limited to the implementation provided in the second embodiment of this application.

[0183] A third embodiment of the present application provides an evaluation device. Figure 5 Schematic diagram of the structure of the evaluation device provided in this embodiment.

[0184] like Figure 5 As shown, the evaluation device provided in this embodiment includes: a probability value determination unit 501, a weight value determination unit 502, and an evaluation result determination unit 503;

[0185] The probability value determination unit 501 is used to determine the probability value of each user in the first user set participating in the event to be evaluated based on the first data corresponding to each user in the first user set, wherein the first data is the data generated by each user in the first user set within a first preset time period for at least one influencing factor that affects the user's participation in the event to be evaluated.

[0186] Optionally, before the step of determining, based on the first data corresponding to each user in the first user set, the probability value of each user in the first user set participating in the event to be evaluated, the method is further configured to:

[0187] The event to be evaluated is placed in a business application in a third preset time period, where the third preset time period is earlier than the first preset time period.

[0188] Optionally, after the step of placing the event to be evaluated in a business application during a third preset time period, the method is further configured to:

[0189] dividing at least part of the users participating in the business application into the first user set and the second user set according to a preset ratio;

[0190] collecting the first data and the second data generated by each user in the first user set within the first preset time period, and the third data generated within the second preset time period, and establishing a correspondence between each user in the first user set and the first data, the second data, and the third data;

[0191] Collect the fourth data and fifth data generated by each user in the second user set within the first preset time period, and establish a correspondence between each user in the second user set and the fourth data and the fifth data, wherein the fourth data is the data generated by each user in the second user set within the first preset time period for at least one of the influencing factors, and the fifth data is the data indicating whether each user in the second user set participated in the event to be evaluated within the first preset time period.

[0192] Optionally, before the step of determining, based on the first data corresponding to each user in the first user set, the probability value of each user in the first user set participating in the event to be evaluated, the method is further configured to:

[0193] A first model is established using the fourth data and the fifth data as training data.

[0194] Optionally, determining, based on the first data corresponding to each user in the first user set, a probability value of each user in the first user set participating in the event to be evaluated includes:

[0195] The first data corresponding to each user in the first user set is input into the first model in sequence, and sixth data corresponding to each user in the first user set output in sequence by the first model is obtained, and the sixth data is used as the probability value of each user in the first user set participating in the event to be evaluated.

[0196] The weight value determination unit 502 is used to determine the weight value of each user in the first user set in the evaluation of the event to be evaluated based on the probability value of each user in the first user set participating in the event to be evaluated and the second data corresponding to each user in the first user set, wherein the second data is data indicating whether each user in the first user set participated in the event to be evaluated within the first preset time period.

[0197] Optionally, determining a weight value of each user in the first user set in evaluating the event to be evaluated based on the probability value of each user in the first user set participating in the event to be evaluated and the second data corresponding to each user in the first user set includes:

[0198] The inverse probability weighting method is used to determine the weight of each user in the first user set in evaluating the event to be evaluated based on the probability value corresponding to each user in the first user set and the second data, specifically:

[0199] The probability value and the second data corresponding to each user in the first user set are sequentially input into a first function to obtain seventh data corresponding to each user in the first user set outputted sequentially by the first function, and the seventh data is used as the weight value of each user in the first user set in evaluating the event to be evaluated, wherein the first function is a function corresponding to the inverse probability weighted method.

[0200] The evaluation result determination unit 503 is used to determine the evaluation result of the event to be evaluated based on the weight value of each user in the first user set in the evaluation of the event to be evaluated, the second data corresponding to each user in the first user set, and the third data corresponding to each user in the first user set, wherein the third data is data indicating whether each user in the first user set has been lost within a second preset time period, the second preset time period is a time period adjacent to the first preset time period, and the first preset time period is earlier than the second preset time period.

[0201] Optionally, determining an evaluation result of the event to be evaluated based on a weight value of each user in the first user set in evaluating the event to be evaluated, the second data corresponding to each user in the first user set, and the third data corresponding to each user in the first user set includes:

[0202] Inputting the weight value corresponding to each user in the first user set, the second data, and the third data into a second model to obtain eighth data output by the second model;

[0203] An evaluation result of the event to be evaluated is determined according to the eighth data.

[0204] Optionally, determining an evaluation result of the event to be evaluated according to the eighth data includes:

[0205] If the eighth data is positive and statistically significant, then the event to be evaluated has a negative impact on user retention.

[0206] Optionally, the event to be evaluated includes an event of recommending a gift package to a player in a game application, and the apparatus is further configured to:

[0207] Determining, based on first data corresponding to each user in the first user set, a probability value of each user in the first user set purchasing the gift package, wherein the first data is data generated by each user in the first user set within a first preset time period for at least one influencing factor that influences a player to purchase the gift package;

[0208] Determining a weight value of each user in the first user set in evaluating the gift package recommendation event based on the probability value of each user in the first user set purchasing the gift package and second data corresponding to each user in the first user set, wherein the second data is data indicating whether each user in the first user set purchased the gift package within the first preset time period;

[0209] The evaluation result of the recommended gift package event is determined based on the weight value of each user in the first user set in the evaluation of the recommended gift package event, the second data corresponding to each user in the first user set, and the third data corresponding to each user in the first user set, wherein the third data is data indicating whether each user in the first user set has been lost within a second preset time period, the second preset time period is a time period adjacent to the first preset time period, and the first preset time period is earlier than the second preset time period.

[0210] Optionally, the first data includes at least one of the following data: the recharge frequency of the user in the first preset period, the total recharge amount of the user in the first preset period, and the time of the last recharge of the user in the first preset period.

[0211] Optionally, the second data is 0 or 1, where 0 indicates that the user did not purchase the gift package within the first preset time period, and 1 indicates that the user purchased the gift package within the first preset time period.

[0212] Optionally, the third data is 0 or 1, wherein 0 indicates that the user has not been lost within the second preset time period, and 1 indicates that the user has been lost within the second preset time period.

[0213] A fourth embodiment of the present application provides an electronic device. Figure 6 Schematic diagram of the structure of the electronic device provided in this embodiment.

[0214] like Figure 6 As shown, the electronic device provided by this embodiment includes: a memory 601 and a processor 602.

[0215] The memory 601 is used to store computer instructions for executing the evaluation method.

[0216] The processor 602 is configured to execute computer instructions stored in the memory 601 to perform the following operations:

[0217] Determining, based on first data corresponding to each user in the first user set, a probability value of each user in the first user set participating in the event to be evaluated, wherein the first data is data generated by each user in the first user set within a first preset time period for at least one influencing factor that affects the user's participation in the event to be evaluated;

[0218] Determining a weight value of each user in the first user set in evaluating the event to be evaluated based on the probability value of each user in the first user set participating in the event to be evaluated and second data corresponding to each user in the first user set, wherein the second data is data indicating whether each user in the first user set participated in the event to be evaluated within the first preset time period;

[0219] An evaluation result of the event to be evaluated is determined based on a weight value of each user in the first user set in evaluating the event to be evaluated, the second data corresponding to each user in the first user set, and the third data corresponding to each user in the first user set, wherein the third data is data indicating whether each user in the first user set has churned within a second preset time period, the second preset time period is a time period adjacent to the first preset time period, and the first preset time period is earlier than the second preset time period.

[0220] Optionally, before the step of determining, based on the first data corresponding to each user in the first user set, the probability value of each user in the first user set participating in the event to be evaluated, the method further includes:

[0221] The event to be evaluated is placed in a business application in a third preset time period, where the third preset time period is earlier than the first preset time period.

[0222] Optionally, after the step of placing the event to be evaluated in a business application during a third preset time period, the method further includes:

[0223] dividing at least part of the users participating in the business application into the first user set and the second user set according to a preset ratio;

[0224] collecting the first data and the second data generated by each user in the first user set within the first preset time period, and the third data generated within the second preset time period, and establishing a correspondence between each user in the first user set and the first data, the second data, and the third data;

[0225] Collect the fourth data and fifth data generated by each user in the second user set within the first preset time period, and establish a correspondence between each user in the second user set and the fourth data and the fifth data, wherein the fourth data is the data generated by each user in the second user set within the first preset time period for at least one of the influencing factors, and the fifth data is the data indicating whether each user in the second user set participated in the event to be evaluated within the first preset time period.

[0226] Optionally, before the step of determining, based on the first data corresponding to each user in the first user set, the probability value of each user in the first user set participating in the event to be evaluated, the method further includes:

[0227] A first model is established using the fourth data and the fifth data as training data.

[0228] Optionally, determining, based on the first data corresponding to each user in the first user set, a probability value of each user in the first user set participating in the event to be evaluated includes:

[0229] The first data corresponding to each user in the first user set is input into the first model in sequence, and sixth data corresponding to each user in the first user set output in sequence by the first model is obtained, and the sixth data is used as the probability value of each user in the first user set participating in the event to be evaluated.

[0230] Optionally, determining a weight value of each user in the first user set in evaluating the event to be evaluated based on the probability value of each user in the first user set participating in the event to be evaluated and the second data corresponding to each user in the first user set includes:

[0231] The inverse probability weighting method is used to determine the weight of each user in the first user set in evaluating the event to be evaluated based on the probability value corresponding to each user in the first user set and the second data, specifically:

[0232] The probability value and the second data corresponding to each user in the first user set are sequentially input into a first function to obtain seventh data corresponding to each user in the first user set outputted sequentially by the first function, and the seventh data is used as the weight value of each user in the first user set in evaluating the event to be evaluated, wherein the first function is a function corresponding to the inverse probability weighted method.

[0233] Optionally, determining an evaluation result of the event to be evaluated based on a weight value of each user in the first user set in evaluating the event to be evaluated, the second data corresponding to each user in the first user set, and the third data corresponding to each user in the first user set includes:

[0234] Inputting the weight value corresponding to each user in the first user set, the second data, and the third data into a second model to obtain eighth data output by the second model;

[0235] An evaluation result of the event to be evaluated is determined according to the eighth data.

[0236] Optionally, determining an evaluation result of the event to be evaluated according to the eighth data includes:

[0237] If the eighth data is positive and statistically significant, then the event to be evaluated has a negative impact on user retention.

[0238] Optionally, the event to be evaluated includes an event of recommending a gift package to a player in a game application, and the method includes:

[0239] Determining, based on first data corresponding to each user in the first user set, a probability value of each user in the first user set purchasing the gift package, wherein the first data is data generated by each user in the first user set within a first preset time period for at least one influencing factor that influences a player to purchase the gift package;

[0240] Determining a weight value of each user in the first user set in evaluating the gift package recommendation event based on the probability value of each user in the first user set purchasing the gift package and second data corresponding to each user in the first user set, wherein the second data is data indicating whether each user in the first user set purchased the gift package within the first preset time period;

[0241] The evaluation result of the recommended gift package event is determined based on the weight value of each user in the first user set in the evaluation of the recommended gift package event, the second data corresponding to each user in the first user set, and the third data corresponding to each user in the first user set, wherein the third data is data indicating whether each user in the first user set has been lost within a second preset time period, the second preset time period is a time period adjacent to the first preset time period, and the first preset time period is earlier than the second preset time period.

[0242] Optionally, the first data includes at least one of the following data: the recharge frequency of the user in the first preset period, the total recharge amount of the user in the first preset period, and the time of the last recharge of the user in the first preset period.

[0243] Optionally, the second data is 0 or 1, where 0 indicates that the user did not purchase the gift package within the first preset time period, and 1 indicates that the user purchased the gift package within the first preset time period.

[0244] Optionally, the third data is 0 or 1, wherein 0 indicates that the user has not been lost within the second preset time period, and 1 indicates that the user has been lost within the second preset time period.

[0245] A fifth embodiment of the present application provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed by a processor, they are used to implement the methods described in each embodiment of the present application.

[0246] It should be noted that relational terms such as "first" and "second" in this document are used only to distinguish one entity or operation from another entity or operation, and do not require or imply any actual relationship or order between these entities or operations. In addition, the words "include," "have," "include," and "includes" and other similar forms are synonymous in meaning, and the ending of any one or more items following any of the above words is open-ended, and none of the above terms indicates that the one or more items are exhaustive or limited to the one or more items listed.

[0247] As used herein, unless expressly stated otherwise, the term "or" includes all possible combinations, except those that are infeasible. For example, if a statement states that a database may include A or B, then unless otherwise specified or infeasible, it may include databases A, B, or A and B. As a second example, if a statement states that a database may include A, B, or C, then unless otherwise specified or infeasible, it may include databases A, B, or C, or A and B, or A and C, or B and C, or A, B, and C.

[0248] It is worth noting that the above embodiments can be implemented by hardware or software (program code), or a combination of hardware and software. If implemented by software, it can be stored in the above-mentioned computer-readable medium. When the software is executed by a processor, it can execute the above-mentioned disclosed method. The computing unit and other functional units described in this disclosure can be implemented by hardware or software, or a combination of hardware and software. Those of ordinary skill in the art will also understand that the above-mentioned multiple modules / units can be combined into one module / unit, and each of the above-mentioned modules / units can be further divided into multiple sub-modules / sub-units.

[0249] In the above detailed description, the embodiments have been described with reference to many specific details, which may vary depending on the implementation. Certain adaptations and modifications may be made to the embodiments. For those skilled in the art, other embodiments will be readily apparent from the specific embodiments disclosed herein. This description and examples are for illustrative purposes only, and the true scope and nature of this application are described in the claims. The order of steps shown in the figures is also for illustrative purposes only and is not intended to be limiting to any particular steps or order. Therefore, those skilled in the art will appreciate that these steps may be performed in different orders when implementing the same method.

[0250] In the figures and detailed description of this application, exemplary embodiments are disclosed. However, many variations and modifications may be made to these embodiments. Accordingly, although specific terms are used, these terms are used in a general and descriptive sense only and not for purposes of limitation.

Claims

1. A method for evaluating a refined operation project, characterized in that: The method comprises: Determining, based on first data corresponding to each user in a first user set, a probability value of each user in the first user set participating in an event to be evaluated, wherein the first data is data generated by each user in the first user set within a first preset time period for at least one influencing factor that affects the user's participation in the event to be evaluated, and the event to be evaluated is a refined operation project designed for a business application; An inverse probability weighting method is used to determine the weight of each user in the first user set in the evaluation of the event to be evaluated based on the probability value of each user in the first user set participating in the event to be evaluated and the second data corresponding to each user in the first user set, wherein the second data is data indicating whether each user in the first user set participated in the event to be evaluated within the first preset time period. The expression of the inverse probability weighting method is as follows: Wherein, IPW represents the weight of the user in evaluating the event to be evaluated, bought represents the data of whether the user participates in the event to be evaluated, and propensity represents the probability value of the user participating in the event to be evaluated; Based on the second model, the evaluation result of the event to be evaluated is determined according to the weight value of each user in the first user set in the evaluation of the event to be evaluated, the second data corresponding to each user in the first user set, and the third data corresponding to each user in the first user set, wherein the third data is data indicating whether each user in the first user set has been lost within a second preset time period, the second preset time period is a time period adjacent to the first preset time period, and the first preset time period is earlier than the second preset time period, the second model is a model corresponding to the machine learning method, and the evaluation result is the impact of the event to be evaluated on user retention.

2. The method according to claim 1, characterized in that Before the step of determining, based on the first data corresponding to each user in the first user set, the probability value of each user in the first user set participating in the event to be evaluated, the method further includes: The event to be evaluated is placed in a business application in a third preset time period, where the third preset time period is earlier than the first preset time period.

3. The method according to claim 2, characterized in that After the step of placing the event to be evaluated in a business application during a third preset time period, the method further includes: Dividing at least part of the users participating in the business application into the first user set and the second user set according to a preset ratio; collecting the first data and the second data generated by each user in the first user set within the first preset time period, and the third data generated within the second preset time period, and establishing a correspondence between each user in the first user set and the first data, the second data, and the third data; Collect the fourth data and fifth data generated by each user in the second user set within the first preset time period, and establish a correspondence between each user in the second user set and the fourth data and the fifth data, wherein the fourth data is the data generated by each user in the second user set within the first preset time period for at least one of the influencing factors, and the fifth data is the data indicating whether each user in the second user set participated in the event to be evaluated within the first preset time period.

4. The method according to claim 3, characterized in that The determining, based on the first data corresponding to each user in the first user set, a probability value of each user in the first user set participating in the event to be evaluated includes: Using the fourth data and the fifth data as training data, establishing a first model, wherein the first model is established based on any one of a probabilistic regression function and a neural network model; The first data corresponding to each user in the first user set is input into the first model in sequence, and sixth data corresponding to each user in the first user set output in sequence by the first model is obtained, and the sixth data is used as the probability value of each user in the first user set participating in the event to be evaluated.

5. The method according to claim 1, wherein The inverse probability weighting method is used to determine the weight of each user in the first user set in evaluating the event to be evaluated based on the probability value of each user in the first user set participating in the event to be evaluated and the second data corresponding to each user in the first user set, including: The probability value and the second data corresponding to each user in the first user set are sequentially input into a first function to obtain seventh data corresponding to each user in the first user set outputted sequentially by the first function, and the seventh data is used as the weight value of each user in the first user set in evaluating the event to be evaluated, wherein the first function is a function corresponding to the inverse probability weighted method.

6. The method according to claim 1, characterized in that The determining, based on the second model, an evaluation result of the event to be evaluated according to a weight value of each user in the first user set in evaluating the event to be evaluated, the second data corresponding to each user in the first user set, and the third data corresponding to each user in the first user set, includes: Inputting the weight value corresponding to each user in the first user set, the second data, and the third data into the second model to obtain eighth data output by the second model; An evaluation result of the event to be evaluated is determined according to the eighth data.

7. The method according to claim 6, characterized in that Determining the evaluation result of the event to be evaluated according to the eighth data includes: If the eighth data is positive and statistically significant, then the event to be evaluated has a negative impact on user retention.

8. The method according to claim 1, characterized in that The event to be evaluated includes an event of recommending a gift package to a player in a game application, and the method includes: Determining, based on first data corresponding to each user in the first user set, a probability value of each user in the first user set purchasing the gift package, wherein the first data is data generated by each user in the first user set within a first preset time period for at least one influencing factor that influences a player to purchase the gift package; Determining a weight value of each user in the first user set in evaluating the gift package recommendation event based on the probability value of each user in the first user set purchasing the gift package and second data corresponding to each user in the first user set, wherein the second data is data indicating whether each user in the first user set purchased the gift package within the first preset time period; The evaluation result of the recommended gift package event is determined based on the weight value of each user in the first user set in the evaluation of the recommended gift package event, the second data corresponding to each user in the first user set, and the third data corresponding to each user in the first user set, wherein the third data is data indicating whether each user in the first user set has been lost within a second preset time period, the second preset time period is a time period adjacent to the first preset time period, and the first preset time period is earlier than the second preset time period.

9. The method according to claim 8, characterized in that The first data includes at least one of the following data: the recharge frequency of the user in the first preset period, the total recharge amount of the user in the first preset period, and the time of the last recharge of the user in the first preset period.

10. The method according to claim 8, characterized in that The second data is 0 or 1, wherein 0 indicates that the user has not purchased the gift package within the first preset time period, and 1 indicates that the user has purchased the gift package within the first preset time period.

11. The method according to claim 8, characterized in that The third data is 0 or 1, wherein 0 indicates that the user has not been lost within the second preset time period, and 1 indicates that the user has been lost within the second preset time period.

12. An evaluation device for a refined operation project, characterized in that: The device comprises: a probability value determination unit, a weight value determination unit, and an evaluation result determination unit; The probability value determination unit is configured to determine, based on first data corresponding to each user in the first user set, a probability value of each user in the first user set participating in the event to be evaluated, wherein the first data is data generated by each user in the first user set within a first preset time period for at least one influencing factor that affects the user's participation in the event to be evaluated, and the event to be evaluated is a refined operation project designed for a business application; The weight value determination unit is configured to determine, using an inverse probability weighting method, a weight value of each user in the first user set in evaluating the event to be evaluated based on the probability value of each user in the first user set participating in the event to be evaluated and second data corresponding to each user in the first user set, wherein the second data is data indicating whether each user in the first user set participated in the event to be evaluated within the first preset time period; the expression for the inverse probability weighting method is as follows: Wherein, IPW represents the weight of the user in evaluating the event to be evaluated, bought represents the data of whether the user participates in the event to be evaluated, and propensity represents the probability value of the user participating in the event to be evaluated; The evaluation result determination unit is used to determine the evaluation result of the event to be evaluated based on a second model, according to the weight value of each user in the first user set in the evaluation of the event to be evaluated, the second data corresponding to each user in the first user set, and the third data corresponding to each user in the first user set, wherein the third data is data indicating whether each user in the first user set has been lost within a second preset time period, the second preset time period is a time period adjacent to the first preset time period, and the first preset time period is earlier than the second preset time period, the second model is a model corresponding to the machine learning method, and the evaluation result is the impact of the event to be evaluated on user retention.

13. An electronic device, characterized in that: include: memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to implement the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having one or more computer instructions stored thereon, characterized in that: The instruction is executed by a processor to implement the method according to any one of claims 1 to 11.

Images