Method, device, electronic device and readable storage medium for determining a tag

Abstract

The embodiments of the present application provide a method, device, electronic device and readable storage medium for determining labels to improve the accuracy of determining labels. The method includes: determining the estimated behavior generation probability of the user according to the historical behavior data of the user; inputting the estimated behavior generation probability of the user into a calibration model to calibrate the estimated behavior generation probability of the user, wherein the calibration model is a model obtained by training a first preset model with the mapping relationship between multiple groups of estimated behavior generation probability sample values ​​and actual behavior generation probability sample values ​​as input; determining the label of the user according to the calibrated estimated behavior generation probability of the user.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular, to a method, device, electronic device, and readable storage medium for determining a tag. Background Art

[0002] Accurately determining user tags is of great significance for information push. Based on user tags, appropriate information can be accurately pushed to users.

[0003] The related art provides a method for determining user tags based on user portraits. Using this method, first, the user's user portrait is determined. The information used to determine a user's user portrait includes the user's basic information, such as gender, age, zodiac sign, permanent residence, etc., and also includes the user's behavior frequency, such as information click frequency, information browsing frequency, etc. Then, the user's tag is determined based on the user's user portrait.

[0004] However, the above method for determining user labels only relies on user portraits, and the information used to determine user portraits is not necessarily true, resulting in insufficient accuracy of user portraits, which in turn directly affects the accuracy of user labels. Therefore, the accuracy of the method for determining user labels in the related art needs to be improved. Summary of the invention

[0005] Embodiments of the present application provide a method, device, electronic device, and readable storage medium for determining a tag to improve the accuracy of determining a user's tag.

[0006] A first aspect of an embodiment of the present application provides a method for determining a user's tag, the method comprising:

[0007] Determine the estimated probability of the user's behavior based on the user's historical behavior data;

[0008] Inputting the user's estimated behavior generation probability into a calibration model to calibrate the user's estimated behavior generation probability, wherein the calibration model is a model obtained by training a first preset model with a mapping relationship between multiple groups of estimated behavior generation probability sample values ​​and actual behavior generation probability sample values ​​as input;

[0009] Determining the user's label according to the calibrated estimated behavior generation probability. Optionally, the method further includes:

[0010] Obtain the target parameters set by the information publisher;

[0011] Collect the historical behavior data of the user associated with the target parameter.

[0012] Optionally, after determining the tag of the user, the method further includes:

[0013] Determine the target audience based on the target parameters;

[0014] The information released by the information publisher is pushed to the target audience.

[0015] Optionally, the calibration model is obtained by following the steps:

[0016] sorting the plurality of sample data according to the magnitude of the estimated behavior generation probability of each of the plurality of sample data;

[0017] Divide the sorted multiple sample data into multiple groups of sample data;

[0018] For each group of sample data in the plurality of groups of sample data, averaging the estimated behavior generation probability of the group of sample data to obtain a sample value of the estimated behavior generation probability of the group;

[0019] According to whether the mark carried by each sample data in the group of sample data is a preset mark, determining the proportion of sample data in the group of sample data whose mark carried is the preset mark, and determining the proportion as the probability sample value of the group of actual behavior generation;

[0020] The first preset model is trained based on the mapping relationship between multiple groups of estimated behavior generation probability sample values ​​and actual behavior generation probability sample values ​​as input to obtain the calibration model.

[0021] Optionally, determining the estimated behavior generation probability of the user according to the historical behavior data of the user includes:

[0022] Obtaining the user's on-site historical behavior data and off-site historical behavior data;

[0023] The user's on-site historical behavior data and off-site historical behavior data are input into a behavior generation probability estimation model to determine the user's estimated behavior generation probability.

[0024] Optionally, determining the estimated behavior generation probability of the user according to the historical behavior data of the user includes:

[0025] The user's historical behavior data is input into a behavior generation probability prediction model to determine the user's estimated behavior generation probability, wherein the behavior generation probability prediction model is a model obtained by training a second preset model with multiple historical behavior sample data carrying marks as input, wherein each mark carried by the historical behavior data represents whether the user has performed actual behavior.

[0026] A second aspect of an embodiment of the present application provides a device for determining a tag, the device comprising:

[0027] A first determination module, used to determine the estimated behavior generation probability of the user according to the historical behavior data of the user;

[0028] A calibration module, used for inputting the estimated behavior generation probability of the user into a calibration model to calibrate the estimated behavior generation probability of the user, wherein the calibration model is a model obtained by training a first preset model with a mapping relationship between multiple groups of estimated behavior generation probability sample values ​​and actual behavior generation probability sample values ​​as input;

[0029] A second determination module is used to determine a label for the user according to the calibrated estimated behavior generation probability. Optionally, the device further includes:

[0030] The first acquisition module is used to obtain the target parameters set by the information publisher;

[0031] A collection module is used to collect the user's historical behavior data associated with the target parameter.

[0032] Optionally, the device further comprises:

[0033] A third determination module is used to determine the target audience according to the target parameter after determining the tag of the user;

[0034] The push module is used to push the information released by the information publisher to the target audience after determining the tag of the user.

[0035] Optionally, the device further comprises:

[0036] A sorting module, used for sorting the plurality of sample data according to the magnitude of the estimated behavior generation probability of each of the plurality of sample data;

[0037] A division module, used for dividing the sorted plurality of sample data into a plurality of groups of sample data;

[0038] A calculation module, configured to average the estimated behavior generation probability of each group of sample data in the plurality of groups of sample data to obtain a sample value of the estimated behavior generation probability of the group;

[0039] The fourth determination module is used to determine the proportion of sample data in the group of sample data that carry a preset mark based on whether the mark carried by each sample data in the group of sample data is a preset mark, and determine the proportion as the probability sample value generated by the group of actual behaviors.

[0040] The training module is used to train the first preset model based on the mapping relationship between multiple groups of estimated behavior generation probability sample values ​​and actual behavior generation probability sample values ​​as input to obtain the calibration model.

[0041] Optionally, the first determining module includes:

[0042] A second acquisition module is used to obtain the user's on-site historical behavior data and off-site historical behavior data;

[0043] The input module is used to input the user's on-site historical behavior data and off-site historical behavior data into the behavior generation probability estimation model to determine the user's estimated behavior generation probability.

[0044] Optionally, the first determining module includes:

[0045] An input submodule is used to input the user's historical behavior data into a behavior generation probability prediction model to determine the user's estimated behavior generation probability, wherein the behavior generation probability prediction model is a model obtained by training a second preset model with multiple historical behavior sample data carrying marks as input, wherein each mark carried by the historical behavior data represents whether the user has performed actual behavior.

[0046] A third aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method for determining a label as described in the first aspect of the present application.

[0047] A fourth aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes, the steps of the method for determining a label described in the first aspect of the present application are implemented.

[0048] A method for determining labels provided by an embodiment of the present application is used to pre-establish a calibration model for calibrating the estimated behavior generation probability of a user. After obtaining the estimated behavior generation probability of multiple users, the user's label is not determined directly based on the estimated behavior generation probability of multiple users, but the estimated behavior generation probability of multiple users is calibrated using the pre-established calibration model, and then the user's label is determined based on the calibrated estimated behavior generation probability of multiple users. Since the determination of the user's label is based on the calibrated estimated behavior generation probability, that is, the user's label is determined using a more accurate estimated behavior generation probability, the accuracy of the determined user's label is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0050] Figure 1 is a flow chart of a method for determining a label proposed in an embodiment of the present application;

[0051] Figure 2 is a flow chart of a method for obtaining historical behavior data proposed in an embodiment of the present application;

[0052] Figure 3 is a flow chart of a method for recommending information to a user proposed in an embodiment of the present application;

[0053] Figure 4 This is a flow chart of determining the probability of a user's estimated behavior generation proposed in an embodiment of the present application;

[0054] Figure 5 is a flow chart of a method for training a calibration model proposed in one embodiment of the present application;

[0055] Figure 6 is another flow chart of a method for collecting historical behavior data provided by another embodiment of the present application;

[0056] Figure 7 is a flow chart of an advertisement delivery method provided by another embodiment of the present application;

[0057] Figure 8 is a schematic diagram of a device for determining a tag proposed in an embodiment of the present application;

[0058] Fig. 9 It is a schematic diagram of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] refer to Figure 1 , Figure 1 : is a flow chart of a method for determining a tag proposed in one embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0061] Step S11: Determine the estimated behavior generation probability of the user based on the user's historical behavior data.

[0062] In this embodiment, the user's historical behavior data includes but is not limited to: the user's on-site historical behavior data and off-site historical behavior data. Among them, "on-site" and "off-site" are relative to a specific platform (for example, an e-commerce platform). The user's historical behavior data includes but is not limited to the following two types of information: the first type of information is the user's on-site historical behavior data, such as: the user's browsing behavior, collection behavior, click behavior, login behavior, etc.; the second type of information is the user's off-site historical behavior data, such as: the name of the client installed by the user, the model of the electronic device used by the user, the advertisements clicked by the user, etc. Optionally, the user's historical behavior data can also be combined with the user's basic information to facilitate the subsequent determination of the user's estimated behavior generation probability. Among them, the user's basic information includes: gender, age, constellation, permanent residence, etc.

[0063] The "behavior" in the estimated behavior generation probability refers to the behavior that users take in response to information recommended by a specific platform. For example, when the recommended information is an advertisement, users can click, collect, share, etc. on the advertisement. The estimated behavior generation probability refers to the pre-estimation of the probability of users taking various actions on the recommended information before the recommended information is recommended to the user, for example: the pre-estimation of the probability of users clicking on an advertisement or collecting an advertisement.

[0064] Based on the historical behavior data of the user, the estimated behavior generation probability of the user can be determined, which means: the historical behavior data of the user is input into a behavior generation probability estimation model (the concept of the behavior generation probability estimation model is explained below).

[0065] Multiple pieces of sample data of multiple users can be obtained based on the historical behavior data of the user. In various embodiments of the present application, all information recommended to the user can form sample data for a certain user, and the user does not necessarily generate user behavior for all sample data. For example, the user views (or does not view) a piece of recommended information but does not perform user behavior such as clicking, collecting or sharing. However, the user can also form a piece of sample data for the recommended information.

[0066] The sample data of a user can be about a specific user behavior, such as sample data about click behavior, sample data about collection behavior, or sample data about sharing behavior. Sample data about click behavior means: sample data of a selected part of users, if the click rate of each of these users is to be calculated based on this sample data, then this part of sample data is used as sample data about click behavior. Similarly, sample data of a selected part of users, if the collection rate of each of these users is to be calculated based on this sample data, then this part of sample data is used as sample data about collection behavior.

[0067] A piece of sample data of a user may include: the user's ID, data generated based on the user's feedback (whether the user has taken a user action or not taken a user action in response to the recommended information), and additional information (the additional information may be arbitrarily determined from the user's historical behavior data and basic information, such as gender, age, occupation, zodiac sign, etc., or may be determined based on the requirements of the information publisher). For example, a piece of sample data about click behavior may specifically be: a piece of sample data generated when a user clicks on a piece of recommended information, or a piece of sample data generated when a user does not click on a piece of recommended information.

[0068] For the same recommended information, multiple different users can generate multiple different sample data. Each sample data is input into the behavior generation probability estimation model, which can predict the probability of each user corresponding to the sample data taking a user action for the recommended information. For all information recommended by a specific platform, the same user can generate multiple sample data. Each sample data is input into the behavior generation probability estimation model, which can predict the probability of the user taking a user action for each recommended information.

[0069] Step S12: Input the user's estimated behavior probability into a calibration model to calibrate the user's estimated behavior probability, wherein the calibration model is a model obtained by training a first preset model with the mapping relationship between multiple groups of estimated behavior probability sample values ​​and actual behavior probability sample values ​​as input.

[0070] In this embodiment, in order to ensure that the user's estimated behavior generation probability obtained by the behavior generation probability estimation model is closer to the actual probability, the user's estimated behavior generation probability is calibrated using a calibration model.

[0071] A set of estimated behavior generation probability sample values ​​is the probability of a user performing a user behavior in response to a piece of recommended information, for example, when a set of estimated behavior generation probability sample values ​​is about click behavior, the higher the sample value, the higher the predicted click rate of the user, and the lower the sample value, the lower the predicted click rate of the user. Corresponding to a set of estimated behavior generation probability sample values ​​is a set of actual behavior generation probability sample values, which indicates the probability that the user actually performs a user behavior in response to a piece of recommended information. One user corresponds to a set of estimated behavior generation probability sample values ​​and a set of actual behavior generation probability sample values, and multiple users correspond to multiple sets of estimated behavior generation probability sample values ​​and multiple sets of actual behavior generation probability sample values. The mapping relationship between multiple sets of estimated behavior generation probability sample values ​​and multiple sets of actual behavior generation probability sample values ​​is used as input to train the first preset model, thereby obtaining a calibration model.

[0072] The specific process of training the calibration model based on the estimated behavior generation probability sample values ​​and the actual behavior generation probability sample values ​​will be described below.

[0073] Step S13: Determine the user's label according to the calibrated estimated behavior generation probability.

[0074] In this embodiment, the higher the calibrated predicted behavior generation probability for a user, the higher the probability that the user will perform a user behavior on the recommended information. The calibrated predicted behavior generation probabilities of multiple users are divided into multiple groups according to a certain order, and user tags are added to each group, so that users can be managed conveniently. According to the user's tag, the user behavior performed by the user on the recommended information can be viewed.

[0075] Taking the case where the user behavior is click behavior as an example, the multiple calibrated estimated behavior generation probabilities of click behaviors are divided into multiple groups according to a certain rule, for example, they are arranged in descending order, and users with probability values ​​between 80% and 100% are divided into Group A, users with probability values ​​between 60% and 80% are divided into Group B, users with probability values ​​between 40% and 60% are divided into Group C, and so on, to obtain multiple groups. Then, user labels are added to the multiple groups obtained by the division in sequence, for example: the user label added to Group A is "user group with a very high probability of clicking on the recommended information", the user label added to Group B is "user group with a high probability of clicking on the recommended information", and the user label added to Group C is "user group with a low probability of clicking on the recommended information".

[0076] The same user can generate multiple different user behaviors, and the same user can belong to multiple different tags according to different user behaviors. For example, in the above embodiment, when the user behavior is a click behavior, a tag about the user of the click behavior is formed. When the user behavior is other types, such as a collection behavior, the user can belong to a tag about the collection behavior. Or, when the user behavior is a sharing behavior, the user can belong to a tag about the sharing behavior.

[0077] Under different tags, the probability of a user performing the user behavior corresponding to the tag may be different. For example, for the same user, the probability of clicking on the recommended information may be between 80% and 100%, the probability of collecting it may be between 60% and 80%, the probability of sharing it may be between 40% and 60%, and so on.

[0078] The following is a detailed description of the above steps S11 to S13 using a specific embodiment. Taking the final label obtained as an example, which is a label about click behavior, the specific process is as follows:

[0079] The first step is to obtain multiple groups of sample data on click behavior (a group of sample data on click behavior is a group of sample data generated based on part or all of a user's recommended information. For example, for all historical recommended information recommended to the user, a group of sample data on click behavior belonging to the user can be obtained). Then, in each group of sample data on click behavior, each sample data on click behavior is input into a behavior generation probability estimation model to determine an estimated behavior generation probability of the user corresponding to the sample data on click behavior (before each recommended information is recommended to the user, the probability of each user clicking on the recommended information, i.e., the estimated behavior generation probability, is usually measured in advance).

[0080] In the second step, multiple groups of estimated behavior generation probabilities about click behaviors are input into the calibration model to calibrate each group of estimated behavior generation probabilities about click behaviors, so as to obtain an estimated behavior generation probability for each user that is close to the true probability value.

[0081] The third step is to generate probabilities based on the estimated behaviors of multiple users that are close to the true probability values ​​obtained in the second step, divide the multiple users into groups, and set labels for each group, wherein, based on each label, the probability of users belonging to the label clicking on the recommended information can be obtained.

[0082] If the label finally obtained is a label about collection behavior, multiple sets of sample data about collection behavior must be obtained first; if the label finally obtained is a label about sharing behavior, multiple sets of sample data about sharing behavior must be obtained first, and so on.

[0083] A method for determining labels provided by an embodiment of the present application is used to pre-establish a calibration model for calibrating the estimated behavior generation probability of a user. After obtaining the estimated behavior generation probability of multiple users, the user's label is not determined directly based on the estimated behavior generation probability of multiple users, but the estimated behavior generation probability of multiple users is calibrated using the pre-established calibration model, and then the user's label is determined based on the calibrated estimated behavior generation probability of multiple users. Since the determination of the user's label is based on the calibrated estimated behavior generation probability, that is, the user's label is determined using a more accurate estimated behavior generation probability, the accuracy of the determined user's label is improved.

[0084] refer to Figure 2 , Figure 2 1 is a flow chart of a method for obtaining historical behavior data proposed in an embodiment of the present application. Before executing steps S11 to S13, it is necessary to obtain historical behavior data. Figure 2 As shown, the method comprises the following steps:

[0085] Step S21: Obtaining target parameters set by the information publisher;

[0086] Step S22: Collecting the user's historical behavior data associated with the target parameter.

[0087] First, in step S21, the target parameter is a condition set by the information publisher to achieve a certain information recommendation effect. The target parameter can be a condition for user behavior, such as: the expected user's click rate, collection rate, sharing rate, conversion rate, etc.; or a restrictive condition for the user's basic information, such as: the expected target user's gender, age group, occupation, etc.; or other conditions, such as: a specific time period. The purpose of setting the target parameter is to improve the effect of the final information recommendation.

[0088] For step S22, since the user has a large amount of historical behavior data, it is not necessary to train all the historical behavior data at once, but the historical behavior data can be used in a targeted manner. How to use it in a targeted manner requires screening all the historical behavior data according to the target parameters set by the user.

[0089] For example, when the target parameter is a range of click-through rates, the historical behavior data associated with the target parameter includes at least: a portion of historical recommendation information recommended to users under normal circumstances (for example, the user has not logged out of the account, has not failed to use a specific platform for a long time, etc.); when the target parameter is the user's gender, the historical behavior data associated with the target parameter includes at least: a portion of historical recommendation information recommended to female users under normal circumstances.

[0090] In this embodiment, not all of the user's historical behavior data is directly used as sample data. Instead, the historical behavior data is filtered through target parameters and then used as the user's sample data. This allows the model subsequently trained using the filtered sample data to be more targeted and more flexible in responding to various requirements from the information publisher.

[0091] refer to Figure 3 , Figure 3 1 is a flow chart of a method for recommending information to a user according to an embodiment of the present application. After executing steps S11 to S13, information can also be recommended to the user. Figure 3 As shown, the method comprises the following steps:

[0092] Step S31: determining the target audience according to the target parameters;

[0093] Step S32: Pushing the information released by the information publisher to the target audience.

[0094] First, in step S31, since the target parameter is a condition set by the information publisher to achieve a certain information recommendation effect, the ideal audience group selected by the information publisher for the recommended information can be obtained according to the target parameter. Then, the user tags obtained in steps S11 to S13 are matched with the target parameter to select the ideal audience group.

[0095] For example, if the target parameters set by the information publisher are: female, 18-24 years old, and click-through rate of 80%-100%, then among the users included in the tags representing users with a click-through rate of 80% or above, users who are female and aged 18-24 years old will be identified as the target audience.

[0096] Then, in step S32, the information released by the information publisher is pushed to the target audience. For example, in the above step S31, after female users who are aged 18-24 and have a click rate of more than 80% for the recommended information are selected as the target audience, the message that the publisher needs to release is recommended to the target audience. Finally, users who belong to the target audience can view the released information, while users who do not belong to the target audience cannot view the released information.

[0097] In this embodiment, on the one hand, selecting target audiences that meet the target parameters based on user tags can help the information publisher to determine the target audience better and faster; on the other hand, in the process of determining the target audience, since the essence of the target parameters involves multiple restrictive conditions, the selected target audience has a great probability of performing user behavior on the recommended information; in addition, since the target parameters can be set flexibly, the needs of the information publisher for various types of target audiences can be better met.

[0098] Figure 4 1 is a flow chart of determining the probability of a user's estimated behavior according to an embodiment of the present application. Figure 4 As shown, step S11 includes the following steps:

[0099] Step S111: Obtaining the user's on-site historical behavior data and off-site historical behavior data;

[0100] Step S112: Inputting the user's on-site historical behavior data and off-site historical behavior data into a behavior generation probability estimation model to determine the user's estimated behavior generation probability.

[0101] First, in step S111, since the concepts of on-site historical behavior data and off-site historical behavior data have been explained in the embodiment corresponding to step S11, they will not be repeated here. In combination with step S21, the on-site historical behavior data and off-site historical behavior data obtained are the on-site historical behavior data and off-site historical behavior data that are screened according to the target parameters of the information publisher.

[0102] Then in step S112, the filtered on-site historical behavior data and off-site historical behavior data can form multiple sample data on specific user behaviors of multiple users. Each sample data is input into the behavior generation probability estimation model to obtain the estimated behavior generation probability for each sample data.

[0103] In this embodiment, in the process of determining the user's label, multiple data sources are referenced: the user's historical behavior data on the site and the historical behavior data outside the site. Compared with the related art that only refers to the user's user portrait, referring to multiple data sources can better determine the user's label.

[0104] In addition, the user's on-site historical behavior data and off-site historical behavior data respectively characterize the effect of information recommendation (e.g., advertising display) from different perspectives, such as whether the user clicked on the recommended information, and whether the user purchased the product in the recommended information. Combining the user's on-site historical behavior data and off-site historical behavior data to determine the user's label can more realistically reflect the user's feedback on the recommended information, which is more suitable for some information recommendation scenarios, such as advertising scenarios, and can provide advertisers with more accurate reference information for advertising.

[0105] In another implementation, step S11 may further include the following steps:

[0106] The user's historical behavior data is input into a behavior generation probability prediction model to determine the user's estimated behavior generation probability, wherein the behavior generation probability prediction model is a model obtained by training a second preset model with multiple historical behavior sample data carrying marks as input, wherein the mark carried by each historical behavior sample data represents whether the user has performed actual behavior.

[0107] The historical behavior sample data is marked, for example: positive sample data (sample data in which the user has taken a user action on the recommended information) is marked with one symbol, and negative sample data (sample data in which the user has not taken a user action on the recommended information) is marked with another different symbol. For example: positive sample data is marked with the symbol "1", and negative sample data is marked with the symbol "0".

[0108] There are multiple types of behavior generation probability prediction models, such as a click behavior generation probability prediction model that predicts the probability of click behavior, or a collection behavior generation probability prediction model that predicts the probability of collection behavior, or a behavior generation probability prediction model that predicts the probability of other types of user behavior. For different types of behavior generation probability prediction models, the types of historical behavior sample data required are different, but consistent with the type of model, for example: for a click behavior generation probability prediction model, the type of historical behavior sample data required is sample data about click behavior; for a collection behavior generation probability prediction model, the type of historical behavior sample data required is sample data about collection behavior.

[0109] The second preset model can be a machine learning model, such as a naive Bayes model, a decision tree model, etc. Taking the final label as an example, if the label about click behavior is marked "1" to indicate that the user clicked on the recommended information, and the label "0" to indicate that the user did not click on the recommended information, then the positive sample data carrying the label "1" and the negative sample data carrying the label "0" are input into the second preset model for training until the trained preset model has the function of predicting the probability of a single user's click behavior. The trained preset model is the click behavior probability estimation model.

[0110] In this embodiment, a behavior generation probability prediction model is established in advance. For each user among multiple users, after obtaining the user's historical behavior data, the user's historical behavior data can be input into the pre-established behavior generation probability prediction model. The value output by the behavior generation probability prediction model is the estimated behavior generation probability of the user. Therefore, when the number of users is large, the estimated behavior generation probability of the user can be determined conveniently and quickly.

[0111] refer to Figure 5 , Figure 5 : is a flow chart of a method for training a calibration model proposed in one embodiment of the present application. Figure 5 As shown, the method comprises the following steps:

[0112] Step S41: sorting the plurality of sample data according to the magnitude of the estimated behavior generation probability of each of the plurality of sample data.

[0113] When sorting the estimated behavior generation probabilities, multiple estimated behavior generation probabilities are for the same specific user behavior. That is, for click behavior, multiple estimated behavior generation probabilities for click behavior are sorted, and for favorite behavior, multiple estimated behavior generation probabilities for favorite behavior are sorted. The sorting method can be ascending order according to the size of the probability value, or descending order according to the size of the probability value.

[0114] Step S42: Divide the sorted plurality of sample data into a plurality of groups of sample data.

[0115] Each piece of sample data about a specific user behavior corresponds to a user, and the sorted sample data is divided into corresponding users, so that each user corresponds to a sample data group, wherein each sample data group contains no less than a preset number of sample data about specific user behaviors. For example: a sample data group of a user contains multiple sample data about click behaviors.

[0116] Step S43: for each group of sample data in the plurality of groups of sample data, averaging the estimated behavior generation probability of the group of sample data to obtain the sample value of the estimated behavior generation probability of the group.

[0117] Take the average of all the estimated behavior generation probabilities of a sample data group to get the sample value of the estimated behavior generation probability of the users in the group. For example, take the average of the estimated behavior generation probability of the click behavior of a sample data group to get the sample value of the estimated behavior generation probability of the click behavior of the users in the group.

[0118] Step S44: Determine the proportion of sample data in the group of sample data that carry a preset mark according to whether the mark carried by each sample data in the group of sample data is a preset mark, and determine the proportion as the probability sample value generated by the group of actual behaviors.

[0119] Multiple sample data about specific user behaviors in each sample data group all carry a mark. When the mark is a preset mark, it means that the user corresponding to the sample data has made the corresponding user behavior. For example: a sample data about click behavior carries the mark "1" ("1" is a preset mark), indicating that the user has made a click behavior on the recommended information in the sample data. If a sample data about click behavior does not carry the mark "1", it means that the user has not made a click behavior on the recommended information in the sample data. The number of sample data carrying the preset mark in the sample data group is used as the numerator, and the number of all sample data in the sample data group is used as the denominator. The ratio of the two is the actual behavior generation probability sample value. For example: the ratio of the sample data carrying the mark "1" to all sample data in the above sample data group is the actual behavior generation probability sample value about the click behavior of the user corresponding to the group.

[0120] Step S45: training the first preset model based on the mapping relationship between the multiple groups of estimated behavior generation probability sample values ​​and the actual behavior generation probability sample values ​​as input to obtain the calibration model.

[0121] Specifically, step S45 includes:

[0122] Determine a mapping relationship (e.g., a functional relationship) between the estimated behavior generation probability sample values ​​and the actual behavior generation probability sample values ​​of the plurality of sample users according to the respective estimated behavior generation probability sample values ​​and the actual behavior generation probability sample values ​​of the plurality of sample users;

[0123] The mapping relationship between the estimated behavior generation probability sample values ​​and the actual behavior generation probability sample values ​​of the multiple sample users is input into the first preset model (which can be a machine learning model, such as a naive Bayes model, a decision tree model, etc.), and the first preset model is trained to obtain the calibration model.

[0124] The following embodiments provide specific application processes for advertising delivery scenarios.

[0125] Figure 6 FIG. 1 is another flow chart of a method for collecting historical behavior data provided by another embodiment of the present application. Figure 6 As shown, before step S11 to step S13, the step of collecting historical behavior data includes:

[0126] Step S10a: Obtaining the marketing target set by the advertiser;

[0127] Step S10b: Collecting historical behavior data of each of the multiple users associated with the marketing target.

[0128] In this embodiment, the marketing goal set by the advertiser refers to the promotion effect that the advertiser expects to achieve. For example, the marketing goal may be: high click-through rate, high conversion rate, high UV (Unique Visitor) volume, etc.

[0129] Considering the marketing goal set by the advertiser, the historical behavior data of multiple users associated with the marketing goal can be collected, and then steps S11 to S13 are performed to obtain user tags from the dimension corresponding to the marketing goal set by the advertiser. For example, the marketing goal set by the advertiser is a high click-through rate, and the user tag results obtained by performing the above steps are divided according to the high and low click-through rates.

[0130] Figure 7 FIG. 1 is a flow chart of an advertisement delivery method provided by another embodiment of the present application. Figure 7 As shown, after step S11 to step S13, the following steps are also included:

[0131] Step S14: determining an advertisement audience from the plurality of users according to the marketing goal;

[0132] Step S15: Pushing the marketing advertisement placed by the advertiser to the advertisement audience.

[0133] After determining the advertising audience according to the marketing goals set by the advertiser, marketing ads can be delivered in a targeted manner to reduce the proportion of people who are misdirected. Specifically, according to the marketing goals set by the advertiser, the advertising audience is determined from multiple users. The advertising audience is the user who matches the marketing goals set by the advertiser. Then, the marketing ads delivered by the advertiser are pushed to the advertising audience, which optimizes the effect of advertising delivery. For non-advertising audiences, the marketing ads delivered by the advertiser can be not pushed, which reduces the cost of advertising delivery.

[0134] If the marketing goal set by the advertiser is a high click-through rate, the users included in the tags representing a high click-through rate are selected as the advertising audience, and the marketing ads placed by the advertiser are pushed to the advertising audience. For another example, if the marketing goal set by the advertiser is a high UV volume, multiple users can be selected as the advertising audience, and the marketing ads placed by the advertiser are pushed to the advertising audience.

[0135] In combination with the above embodiments, another embodiment of the present application provides a method for advertising. In addition to step S11 to step S13, the method further includes the following steps:

[0136] According to the tag of the target user, a marketing advertisement corresponding to the tag is pushed to the target user, and the target user is any user among the multiple users.

[0137] In this embodiment, after determining the user's tag, targeted marketing advertisements can also be delivered. Specifically, marketing advertisements corresponding to the tags can be delivered to the users with different tags, that is, different marketing advertisements can be delivered to users with different tags, so as to increase the user's click-through rate and conversion rate, optimize the advertising delivery effect, and control the advertising delivery cost.

[0138] Based on the same inventive concept, an embodiment of the present application provides a device for determining a tag. Figure 8 , Figure 8 FIG. 1 is a schematic diagram of a device for determining a tag provided in an embodiment of the present application. Figure 8 As shown, the device comprises:

[0139] A first determination module 801 is used to determine the estimated behavior generation probability of the user according to the historical behavior data of the user;

[0140] A calibration module 802 is used to input the user's estimated behavior generation probability into a calibration model to calibrate the user's estimated behavior generation probability, wherein the calibration model is a model obtained by training a first preset model with a mapping relationship between multiple groups of estimated behavior generation probability sample values ​​and actual behavior generation probability sample values ​​as input;

[0141] The second determination module 803 is used to determine a label for the user according to the calibrated estimated behavior generation probability.

[0142] Optionally, the device further comprises:

[0143] The first acquisition module is used to obtain the target parameters set by the information publisher;

[0144] A collection module is used to collect the historical behavior data of the user associated with the target parameter.

[0145] Optionally, the device further comprises:

[0146] A third determination module is used to determine the target audience according to the target parameter after determining the tag of the user;

[0147] The push module is used to push the information released by the information publisher to the target audience after determining the tag of the user.

[0148] Optionally, the device further comprises:

[0149] A sorting module, used for sorting the plurality of sample data according to the magnitude of the estimated behavior generation probability of each of the plurality of sample data;

[0150] A partitioning module, used for partitioning the sorted plurality of sample data into a plurality of groups of sample data;

[0151] A calculation module, configured to average the estimated behavior generation probability of each group of sample data in the plurality of groups of sample data to obtain a sample value of the estimated behavior generation probability of the group;

[0152] The fourth determination module is used to determine the proportion of sample data in the group of sample data that carry a preset mark based on whether the mark carried by each sample data in the group of sample data is a preset mark, and determine the proportion as the probability sample value generated by the group of actual behaviors.

[0153] The training module is used to train the first preset model based on the mapping relationship between multiple groups of estimated behavior generation probability sample values ​​and actual behavior generation probability sample values ​​as input to obtain the calibration model.

[0154] Optionally, the first determining module includes:

[0155] A second acquisition module is used to obtain the user's on-site historical behavior data and off-site historical behavior data;

[0156] The input module is used to input the user's on-site historical behavior data and off-site historical behavior data into the behavior generation probability estimation model to determine the user's estimated behavior generation probability.

[0157] Optionally, the first determining module includes:

[0158] An input submodule is used to input the user's historical behavior data into a behavior generation probability prediction model to determine the user's estimated behavior generation probability, wherein the behavior generation probability prediction model is a model obtained by training a second preset model with multiple historical behavior sample data carrying marks as input, wherein each mark carried by the historical behavior data represents whether the user has performed actual behavior.

[0159] Based on the same inventive concept, another embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the method for determining a label as described in any of the above embodiments of the present application are implemented.

[0160] Based on the same inventive concept, another embodiment of the present application provides an electronic device. Fig. 9 , Fig. 9 is a schematic diagram of an electronic device provided by an embodiment of the present application. Fig. 9 As shown, the electronic device 100 includes: a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus communication. A computer program is stored in the memory 110. The computer program can be executed on the processor 120 to implement the steps in the method for determining a tag described in any of the above embodiments of the present application.

[0161] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0162] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0163] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0167] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0168] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0169] The above is a detailed introduction to a method, device, storage medium and electronic device for determining a label provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for determining a label, characterized in that: The method comprises: determining the estimated behavior generation probability of the user according to the historical behavior data of the user; inputting the estimated behavior generation probability of the user into a calibration model to calibrate the estimated behavior generation probability of the user, wherein the calibration model is a model obtained by training a first preset model with a mapping relationship between multiple groups of estimated behavior generation probability sample values ​​and actual behavior generation probability sample values ​​as input; determining the label of the user according to the calibrated estimated behavior generation probability; The calibration model is obtained according to the following steps: sorting the multiple sample data according to the size of their respective estimated behavior generation probabilities; dividing the sorted multiple sample data into multiple groups of sample data; for each group of sample data in the multiple groups of sample data, averaging the estimated behavior generation probabilities of the group of sample data to obtain the sample values ​​of the estimated behavior generation probabilities of the group; determining the proportion of sample data in the group of sample data whose marks are preset marks according to whether the marks carried by each sample data in the group of sample data are preset marks, and determining the proportion as the sample values ​​of the actual behavior generation probabilities of the group; training the first preset model according to the mapping relationship between the multiple groups of estimated behavior generation probabilities and the actual behavior generation probabilities as input to obtain the calibration model; Determining the estimated behavior generation probability of the user based on the historical behavior data of the user includes: obtaining the historical behavior data of the user on the site and the historical behavior data of the user off the site; inputting the historical behavior data of the user on the site and the historical behavior data of the user off the site into the behavior generation probability estimation model to determine the estimated behavior generation probability of the user; Determining the estimated behavior probability of the user based on the user's historical behavior data includes: inputting the user's historical behavior data into a behavior probability estimation model to determine the user's estimated behavior probability, wherein the behavior probability estimation model is a model obtained by training a second preset model with multiple historical behavior sample data carrying tags as input, wherein the tag carried by each of the historical behavior sample data represents whether the user has performed actual behavior.

2. The method according to claim 1, characterized in that The method further includes: obtaining target parameters set by the information publisher; and collecting historical behavior data of the user associated with the target parameters.

3. The method according to claim 2, characterized in that After determining the tag of the user, the method further includes: determining a target audience according to the target parameter; and pushing the information released by the information publisher to the target audience.

4. A device for determining a label, characterized in that: The device comprises: a first determination module, used to determine the estimated behavior generation probability of the user according to the historical behavior data of the user; a calibration module, used to input the estimated behavior generation probability of the user into a calibration model to calibrate the estimated behavior generation probability of the user, wherein the calibration model is a model obtained by training a first preset model with the mapping relationship between multiple groups of estimated behavior generation probability sample values ​​and actual behavior generation probability sample values ​​as input; a second determination module, used to determine a label for the user according to the calibrated estimated behavior generation probability; the calibration model is obtained by the following steps: according to the large and small values ​​of the estimated behavior generation probability of each of the multiple sample data small, sorting the plurality of sample data; dividing the sorted plurality of sample data into a plurality of groups of sample data; for each group of sample data in the plurality of groups of sample data, averaging the estimated behavior generation probability of the group of sample data to obtain the group of estimated behavior generation probability sample values; determining the proportion of sample data in the group of sample data whose mark is a preset mark according to whether the mark carried by each sample data in the group of sample data is a preset mark, and determining the proportion as the group of actual behavior generation probability sample values; training the first preset model according to the mapping relationship between the plurality of groups of estimated behavior generation probability sample values ​​and the actual behavior generation probability sample values ​​as input to obtain the calibration model; Determining the estimated behavior generation probability of the user based on the historical behavior data of the user includes: obtaining the historical behavior data of the user on the site and the historical behavior data of the user off the site; inputting the historical behavior data of the user on the site and the historical behavior data of the user off the site into the behavior generation probability estimation model to determine the estimated behavior generation probability of the user; Determining the estimated behavior probability of the user based on the user's historical behavior data includes: inputting the user's historical behavior data into a behavior probability estimation model to determine the user's estimated behavior probability, wherein the behavior probability estimation model is a model obtained by training a second preset model with multiple historical behavior sample data carrying tags as input, wherein the tag carried by each of the historical behavior sample data represents whether the user has performed actual behavior.

5. The device according to claim 4, characterized in that The device also includes: a first acquisition module, used to obtain the target parameters set by the information publisher; and a collection module, used to collect the historical behavior data of the user associated with the target parameters.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in any one of the methods described in claims 1 to 3 are implemented.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When executed, the processor implements the steps of the method according to any one of claims 1 to 3.

Images