Sample generation method, machine learning model training method, and information evaluation method

Abstract

The present disclosure provides a sample generation method, a machine learning model training method, an information evaluation method, an apparatus, a device, a storage medium and a program product, relates to the technical field of computer and data processing, and particularly relates to the technical field of artificial intelligence, machine learning, big data, data mining and the like. The specific implementation scheme is as follows: for any one content publishing subject, determining target information according to the publishing content of the content publishing subject; determining a label for each content publishing subject according to the target information; and for any one content publishing subject, generating a subject sample according to the label.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer technology and data processing, in particular to the technical field of artificial intelligence, machine learning, big data, data mining, and the like, and specifically relates to a sample generation method, a machine learning model training method, an information evaluation method, an apparatus, a device, a storage medium, and a program product. BACKGROUND

[0002] The development of computer technology and Internet technology has led to a large number of specific business needs in various scenarios, which has made it a technical problem to use automated methods to solve specific businesses. SUMMARY

[0003] The present disclosure provides a sample generation method, a machine learning model training method, an information evaluation method, an apparatus, a device, a storage medium, and a program product.

[0004] According to an aspect of the present disclosure, a sample generation method is provided, comprising: determining target information for any one content publishing subject according to the publishing content of the content publishing subject; determining a label for each content publishing subject according to the target information; and generating a subject sample for any one content publishing subject according to the label.

[0005] According to another aspect of the present disclosure, a machine learning model training method is provided, comprising: inputting a subject sample into an initial machine learning model to obtain an output result of the subject sample; determining a feedback value according to the output result and a loss function; and adjusting the model parameters of the initial machine learning model according to the feedback value to obtain a target machine learning model, wherein the subject sample is obtained by using the above-mentioned sample generation method.

[0006] According to another aspect of the present disclosure, an information evaluation method is provided, comprising: inputting a to-be-processed content associated with a content publishing subject to be evaluated into a target machine learning model to obtain a processing result, wherein the processing result represents an evaluation result for the content publishing subject; and the target machine learning model is trained by the above-mentioned machine learning model training method.

[0007] According to another aspect of the present disclosure, a sample generation apparatus is provided, comprising: a target information determination module configured to determine target information for any one content publishing subject according to the publishing content of the content publishing subject; a label determination module configured to determine a label for each content publishing subject according to the target information; and a subject sample generation module configured to generate a subject sample for any one content publishing subject according to the label.

[0008] According to another aspect of the present disclosure, a training device of a machine learning model is provided, comprising: an output result determination module configured to input a subject sample into an initial machine learning model to obtain an output result of the subject sample; a feedback value determination module configured to determine a feedback value according to the output result and a loss function; and a target machine learning model determination module configured to adjust model parameters of the initial machine learning model according to the feedback value to obtain a target machine learning model, wherein the subject sample is obtained by using the sample generation device.

[0009] According to another aspect of the present disclosure, an information evaluation device is provided, comprising: a processing result determination module configured to input to-be-processed content associated with a content publishing subject to be evaluated into a target machine learning model to obtain a processing result, wherein the processing result represents an evaluation result for the content publishing subject; and the target machine learning model is trained by using the training device of the machine learning model.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory connected to the at least one processor in communication. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the embodiments of the present disclosure.

[0011] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, and the computer instructions are used to enable a computer to perform the method of the embodiments of the present disclosure.

[0012] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program stored in at least one of a readable storage medium and an electronic device, and the computer program is stored in at least one of the readable storage medium and the electronic device, and the computer program is executed by a processor to implement the method of the embodiments of the present disclosure.

[0013] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings are used to better understand the present scheme, and do not limit the present disclosure. Among them:

[0015] Figure 1 The system architecture diagram of the sample generation method, the training method of the machine learning model, the information evaluation method and device according to the embodiments of the present disclosure is schematically shown;

[0016] Figure 2A flowchart of a sample generation method according to an embodiment of the present disclosure is schematically shown;

[0017] Figure 3 A schematic diagram of a sample generation method according to another embodiment of the present disclosure is schematically shown;

[0018] Figure 4 A flowchart of a machine learning model training method according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 5 A schematic diagram of a machine learning model training method according to yet another embodiment of the present disclosure is schematically shown;

[0020] Figure 6 A flowchart of an information evaluation method according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 7 A block diagram of a sample generation apparatus according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 8 A block diagram of a machine learning model training apparatus according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 9 A block diagram of an information evaluation apparatus according to an embodiment of the present disclosure is schematically shown; and

[0024] Figure 10 A block diagram of an electronic device that can implement a sample generation method, a machine learning model training method, and an information evaluation method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0025] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are included to provide a thorough understanding of embodiments of the present disclosure by a person of ordinary skill in the art, and should not be construed as limiting the present disclosure to particular embodiments. Thus, it will be apparent to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, the description below includes detailed explanations of well-known functions and structures, which are omitted in the interest of clarity and conciseness.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the term "includes" and / or "including", as well as "comprises" and / or "comprising", means "including, but not limited to".

[0027] All terms used herein (including technical and scientific terms) have the meaning commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of the specification, and should not be interpreted in an idealized or overly formal manner.

[0028] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should be generally interpreted as including any one of A, B, or C, etc. (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C, etc.).

[0029] The development of computer technology and Internet technology has led to a large number of specific business needs in various scenarios, which has also made it a technical problem how to use an automated way to solve specific businesses.

[0030] For example, with the development of self-media platforms, network users can publish content such as text, video, live broadcast, and dynamic on the self-media platform as content publishers. In this specific business scenario, the self-media platform has the need to evaluate the content publisher. For example, the self-media platform can support the content publisher to obtain economic gains and the like through the published content, and the evaluation of the content publisher's ability to obtain economic gains can be used to assist the maintenance and operation of the self-media platform.

[0031] In some embodiments, the content publisher is evaluated by artificial evaluation, and such an evaluation method has low accuracy and low efficiency.

[0032] Figure 1 The system architecture of the sample generation method, the training method of the machine learning model, and the information evaluation method and device according to an embodiment of the present disclosure is schematically shown. It should be noted that, Figure 1 The shown is only an example of the system architecture to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0033] As Figure 1 shown, the system architecture 100 in the embodiments of the present disclosure can include a terminal 101 for sample generation, a terminal 102 for machine learning model training, and a terminal 103 for information evaluation.

[0034] In the embodiments of the present disclosure, the terminal 101 can be configured to generate a subject sample by performing a corresponding sample generation method according to the publishing content of the content publishing subject. The terminal 102 can be configured to perform a corresponding machine learning model training method to obtain a target machine learning model according to the subject sample obtained by the terminal 101. The terminal 103 can be configured to perform information evaluation on the to-be-processed content associated with the content publishing subject to be evaluated based on the target machine learning model obtained by the terminal 102, and obtain an evaluation result.

[0035] It should be noted that any one or more of the information evaluation, the training of the machine learning model, and the sample generation can be implemented on the same terminal, or can be implemented on different terminals.

[0036] The terminal 101, the terminal 102, and the terminal 103 can be a server, and can also be a server cluster.

[0037] It should be understood that Figure 1 The number of the terminal 101, the terminal 102, and the terminal 103 is only illustrative. According to the needs of implementation, there can be any number of the terminal 101, the terminal 102, and the terminal 103.

[0038] It should be noted that in the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solutions comply with the relevant legal regulations and do not violate public order and good customs.

[0039] In the technical solutions of the present disclosure, the authorization or consent of the user is obtained before the user personal information is acquired or collected.

[0040] The embodiments of the present disclosure provide a sample generation method, which will be described below in combination with Figure 1 the system architecture, and with reference to Figures 2-3 the sample generation method according to the exemplary embodiments of the present disclosure. The sample generation method of the embodiments of the present disclosure can be performed by the terminal 101 as shown in Figure 1 for example.

[0041] Figure 2 A flowchart of a sample generation method according to an embodiment of the present disclosure is illustratively shown.

[0042] As shown in Figure 2 the sample generation method 200 of the embodiments of the present disclosure can include operation S210 to operation S230, for example.

[0043] In operation S210, for any one content publishing subject, target information is determined according to the publishing content of the content publishing subject.

[0044] Exemplarily, the content publishing subject may, for example, include an author publishing content on a platform such as a self-media platform. The published content may, for example, include image content, text content, video content, and the like.

[0045] In operation S220, a label for each content publishing subject is determined according to the target information.

[0046] In operation S230, a subject sample is generated for any one content publishing subject according to the label.

[0047] Exemplarily, the subject sample may be used for model training, for example, may be used for training of a machine learning model, and the trained machine learning model may, for example, be used to perform a specific task. The subject sample with the label as input for model training may be used for supervised learning for the specific task.

[0048] According to the sample generation method of the embodiments of the present disclosure, the target information used to determine the label may be mined according to the published content of the content publishing subject. The subject sample generated according to the label may be adapted to the related downstream task using the label for supervised learning. For example, in the case of training of a machine learning model according to the subject sample, the related task may be completed automatically, and the accuracy and efficiency of task execution may be improved. The task may, for example, include an evaluation task of the content publishing subject.

[0049] The above-mentioned generation of the subject sample may be understood as a process of outputting the subject sample through computer processing.

[0050] Exemplarily, according to the sample generation method of another embodiment of the present disclosure, the following embodiments may, for example, be used to determine the target information according to the published content of the content publishing subject: determining the total published content of the content publishing subject in a preset time period, the target published content with gain, and the gain value of each target published content according to the published content of the content publishing subject.

[0051] It should be noted that in the preset time period, all the published content published by a certain content publishing subject may be regarded as the total published content, and the total published content may include published content with gain and published content without gain. The target published content may be understood as the published content with gain in the total published content.

[0052] The gain may be understood as the benefit brought by the published content, for example, may include economic benefit.

[0053] For example, at least one of the total published content, the target published content, and the gain value of each target published content may be used as the target information.

[0054] According to the sample generation method provided in the embodiments of the present disclosure, target information strongly related to the evaluation task of the content publishing subject can be determined from the published content of the content publishing subject, so that an accurate label can be determined according to the target information, and a model trained by using the subject sample with the label has higher evaluation accuracy.

[0055] For example, according to the sample generation method provided in another embodiment of the present disclosure, a specific example of determining a label for each content publishing subject according to target information can be implemented by using the following embodiment: determining a total gain number and a gain average of each target published content according to the gain values of the target published content of the content publishing subject in a preset time period; determining a content gain ratio according to the number of target published content and the number of total published content of the content publishing subject in the preset time period; and determining the label according to the total gain number, the gain average and the content gain ratio.

[0056] For example, the preset time period can be two months.

[0057] For example, the number of target published content of the content publishing subject in the preset time period and the gain value of each target published content can be determined.

[0058] For example, the gain values of each target published content can be added to obtain the total gain number. The gain average of each target published content can also be determined according to the total gain number and the number of target published content. The content gain ratio can also be determined according to the number of target published content and the number of total published content.

[0059] For example, the positive sample label and the negative sample label can be determined according to the total gain number, the gain average and the content gain.

[0060] For example, taking the gain as the economic benefit, a threshold for comparing with the total gain number, the gain average and the content gain ratio can be set for the preset time period, respectively. For example, the corresponding content publishing subject with the total gain number, the gain average and the content gain greater than or equal to the corresponding threshold can be determined as a positive sample, and the corresponding content publishing subject with the total gain number, the gain average and the content gain less than the corresponding threshold can be determined as a negative sample.

[0061] For example, the content publishing subject with higher gain has higher ability to obtain economic benefit. According to the sample generation method provided in the embodiments of the present disclosure, the label representing the ability of the content publishing subject to obtain the gain can be accurately determined according to the total gain number, the gain average and the content gain ratio. The model trained according to the subject sample with the label can be used to accurately evaluate the gain obtaining ability of the content publishing subject.

[0062] Exemplarily, Figure 3 A schematic diagram of a sample generation method according to another embodiment of the present disclosure is shown schematically.

[0063] As Figure 3 shown, for example, the content publisher publishing the target published content can be determined as the target published content publisher 302 according to the content publisher 301 in a predetermined time period, and the gain evaluation (for example, gain evaluation by gain total, gain average, content gain ratio) of the published content published by the target published content publisher can be performed to determine the positive sample 303 and the negative sample 304, and the content publisher not publishing the target published content in the predetermined time period can be determined as the content publisher to be evaluated 305.

[0064] Exemplarily, the sample generation method according to still another embodiment of the present disclosure, the label includes a positive sample label and a negative sample label, and the subject sample includes a positive sample and a negative sample.

[0065] The subject sample with the positive sample label is the positive sample, and the subject sample with the negative sample label is the negative sample.

[0066] The sample generation method according to still another embodiment of the present disclosure may, for example, further include: in a case where a sample ratio between the number of positive samples and the number of negative samples is less than or equal to a preset threshold, expanding the number of positive samples according to the positive samples to obtain expanded positive samples.

[0067] The sample ratio between the number of expanded positive samples and the number of negative samples is greater than the preset threshold.

[0068] The preset threshold represents the sample ratio between the number of positive samples and the number of negative samples. Exemplarily, the preset threshold may, for example, be 1:1.

[0069] Exemplarily, for example, the positive samples can be data enhanced to obtain new positive samples, so as to realize the specific example of expanding the number of positive samples according to the positive samples to obtain expanded positive samples.

[0070] For example, in a case where the number of positive samples and the number of negative samples are greatly different, that is, in a case of sample imbalance, it is easier to train an inaccurate model using the positive samples and the negative samples, and the inaccuracy is reflected in that the model has a greater probability of outputting a result corresponding to a larger number of samples.

[0071] According to the sample generation method of the embodiment of the present disclosure, in the case of an imbalance between the number of positive samples and the number of negative samples in an actual business scenario (for example, the number of positive samples is large and the number of negative samples is small), the number of positive samples is expanded according to the positive samples to obtain expanded positive samples, so that the samples can be balanced. The accuracy of the model trained based on the balanced samples for performing a task is also higher, and the performance is better.

[0072] The embodiment of the present disclosure provides a training method of a machine learning model. The training method of the machine learning model of the embodiment of the present disclosure is described below with reference to the system architecture of Figure 1 Figure 4 The training method of the machine learning model according to the exemplary embodiment of the present disclosure is described below with reference to the system architecture of Figure 1 The training method of the machine learning model of the embodiment of the present disclosure can be performed by the terminal 102 shown in

[0073] Figure 4 The flowchart of the training method of the machine learning model according to an embodiment of the present disclosure is schematically shown.

[0074] As shown in Figure 4 The training method 400 of the machine learning model of the embodiment of the present disclosure can include operation S410 to operation S430, for example.

[0075] In operation S410, the subject sample is input into the initial machine learning model to obtain the output result of the subject sample.

[0076] In operation S420, the feedback value is determined according to the output result and the loss function.

[0077] The loss function can be used to evaluate the difference between the output result of the initial machine learning model of any one subject sample in the current training stage and the true value.

[0078] In operation S430, the model parameters of the initial machine learning model are adjusted according to the feedback value to obtain the target machine learning model.

[0079] The subject sample is obtained by the sample generation method of the above embodiment.

[0080] According to the training method of the machine learning model of the embodiment of the present disclosure, since the subject sample is obtained by the sample generation method of the above embodiment, the output result of the subject sample obtained by inputting the subject sample into the initial machine learning model, and the loss function according to the output result, the initial machine learning model can be supervised learning, and the target machine learning model determined thereby can automatically perform related tasks, improving the accuracy and efficiency of task execution. The task may, for example, include an evaluation task of a content publishing subject.

[0081] ​Exemplarily, the performance of the initial machine learning model can also be verified by the verification samples of the verification set during the training process of the initial machine learning model, and specifically, the probability threshold and the AUC can be used as the parameters for performance verification. The probability threshold can be set to 0.5, for example.

[0082] It should be noted that according to the sample generation method of the above embodiment, at least one of the full amount of published content of the content publishing subject within the preset time period, the target published content with the gain, and the gain value of each target published content can be used as the target information, and the accurate label can be determined according to the target information. Therefore, the target machine learning model obtained by training the model using the subject sample with the label has higher accuracy in performing related tasks, for example, can be used to evaluate the gain acquisition ability of the content publishing subject, and has higher accuracy and efficiency.

[0083] According to the sample generation method of the above embodiment, balanced positive samples and negative samples can be obtained, and the target machine learning model obtained by training the initial machine learning model using the balanced samples including the positive samples and the negative samples has higher accuracy in performing tasks.

[0084] Figure 5 A schematic diagram of a training method of a machine learning model according to yet another embodiment of the present disclosure is schematically shown.

[0085] As shown in Figure 5 According to the training method of the machine learning model of yet another embodiment of the present disclosure, the following embodiment can be used to implement a specific example of inputting the subject sample into the initial machine learning model to obtain the output result of the subject sample: inputting the subject sample 501 into the first level learner N1 of the initial machine learning model 505 to obtain a plurality of initial output results. The initial output results are input into the second level learner N2 of the initial machine learning model 505 to obtain the target output result 506.

[0086] The first level learner N1 includes a plurality of sub-learners, and each sub-learner outputs an initial output result.

[0087] Exemplarily, the plurality of sub-learners can include at least two of the following: a random forest model (RF), a neural network model, an extremely random tree model (ET), and an adaptive boosting model (AdaBoost).

[0088] Exemplarily, the neural network model can include a multilayer perceptron (ML), for example.

[0089] The second-level learner is used to assign corresponding weights to multiple initial outputs, and the target output is obtained based on the multiple initial outputs and their corresponding weights.

[0090] For example, the second-level learner may include a gradient boosting decision tree (GBDT).

[0091] The output result of the main sample is the same as the target output result.

[0092] According to the training method of the machine learning model in the embodiments of this disclosure, by inputting the subject sample into the first-level learner of the initial machine learning model, multiple initial output results can be obtained. Multiple specific sub-learners can be integrated to combine the performance advantages of each seed learner. By inputting the initial output result of each sub-learner into the second-level learner of the initial machine learning model, more accurate target output results can be obtained quickly.

[0093] According to the training method of the machine learning model in the embodiments of this disclosure, by using a second learner to assign corresponding weights to multiple initial output results, the weights assigned to multiple initial output results can be automatically and learnably determined, thereby obtaining more accurate target output results.

[0094] For example, such as Figure 5 As shown, the training method of the machine learning model according to another embodiment of this disclosure may further include, for example, extracting features from the subject sample 501 to obtain subject sample features 502. Performing at least one of the following operations on the subject sample features 502: feature discretization, feature selection, and missing value imputation to obtain target features 504.

[0095] The target feature 504 is used as input to the initial machine learning model 505.

[0096] The process of performing feature discretization, feature selection, and missing value imputation on the main sample features to obtain the target features can be understood as the operation of feature engineering.

[0097] Feature discretization is a process for handling discrete data.

[0098] For example, feature selection can be based on the correlation between features.

[0099] For example, features can also be selected using machine learning models such as random forest models.

[0100] For example, missing discrete features can be filled with "None".

[0101] According to the method for training the machine learning model, the subject sample features are obtained by performing feature extraction on the subject samples, and the target features meeting the input requirements of the initial machine learning model and being more relevant to the tasks performed by the machine learning model are obtained by performing feature engineering on the subject sample features, so that the training of the machine learning model can be performed efficiently and quickly, and the target machine learning model obtained by the training has higher accuracy in performing the tasks and better performance.

[0102] For example, as shown in the figure, according to the method for training the machine learning model according to another embodiment of the present disclosure, the subject sample features may, for example, include at least one category of the following: basic attributes, identity attributes, and operation data of the content publishing subject. Figure 5

[0103] Hereinafter, the target machine learning model obtained by training the initial machine learning model will be taken as an example to perform an evaluation on the economic gain obtaining ability of the content publishing subject.

[0104] For example, the basic attributes of the content publishing subject may, for example, include but are not limited to the following dimensions: age, gender, registration time, education, city level, marital status, income level, and consumption level of the content publishing subject.

[0105] For example, the identity attributes of the content publishing subject may, for example, include but are not limited to the following dimensions: account type, whether a high-quality field creator, whether an original identity, whether supporting distribution of text, image, and video, author brand level, whether opening platform advertising, whether using brand advertising, whether a Ma Jia number, and whether a personalized author.

[0106] For example, the identity attributes of the content publishing subject may, for example, include but are not limited to the following dimensions:

[0107] Behavior interaction situation: comment times, message times, fan group times, automatic reply times, and total fan interaction times.

[0108] Sharing and promotion situation: article sharing times, author sharing times, number of times of putting, and total sharing and spreading times.

[0109] Account management situation: total content management times, home page configuration times, and total account management times.

[0110] Article publishing operation situation: manual content publishing times, content automatic synchronization times, task square article publishing times, publishing set, and content insertion special form times.

[0111] For example, the subject sample features may, for example, include at least one category of the following: publishing content data, content consumed by users, economic gain right usage data, and fan situation data of the content publishing subject.​

[0112] Exemplarily, the published content data may, for example, include but are not limited to the following dimensions:

[0113] Content type distribution: total number of published articles, number and proportion of published articles of each type (e.g., articles, short videos, small videos, live broadcasts, and moments), main type of published articles, proportion of main type of published articles, number of published article types, and the like.

[0114] Content vertical distribution: main verticals of each type (e.g., articles, short videos, small videos, and moments), number and proportion of published articles of main verticals of each type, number of published verticals, and the like.

[0115] Original content: number and proportion of original published articles, number and proportion of original content of each type (e.g., articles, short videos, small videos, and moments).

[0116] Hit content: number of hit articles in the last two months, hit rate in the last two months.

[0117] Exemplarily, the content consumption data may, for example, include but are not limited to the following dimensions: number of displays / distributions / point displays / durations of each type (e.g., articles, short videos, small videos, live broadcasts, and moments), number of displays / distributions / point displays / durations per article of each type, proportion of displays / distributions / point displays / durations of main verticals of each type, number and proportion of distributions of original content, article completion rate, video completion rate, and the like.

[0118] Exemplarily, the economic gain right usage data may, for example, include but are not limited to the following dimensions:

[0119] Right opening: whether column rights, ask-a-question rights, tipping rights, and content payment rights are opened.

[0120] Activity: whether the column is active, whether the paid Q&A is active, whether the paid circle is active, number of times of mounting paid content by the author, number of times of mounting tips by the author, number of times of mounting paid consultations by the author, and number of times of mounting paid circles by the author.

[0121] Exemplarily, the content publisher fan data may, for example, include but are not limited to the following dimensions: number of fans of the content publisher, fan conversion rate of the content publisher, proportion of fan distribution of the content publisher, fan activity, fan content preference, fan interaction rate, fan interaction interest, and the like.

[0122] The information evaluation method provided in the embodiments of the present disclosure is described below in combination with the system architecture of Figure 1 and the information evaluation method according to the exemplary embodiments of the present disclosure is described with reference to Figure 6 The information evaluation method of the embodiments of the present disclosure may, for example, be implemented byFigure 1 The terminal 103 shown is configured to perform.

[0123] Figure 6 A flowchart of an information evaluation method according to an embodiment of the present disclosure is schematically shown.

[0124] As shown in Figure 6 The information evaluation method 600 of the embodiment of the present disclosure may, for example, include operation S610 to operation S620.

[0125] In operation S610, input the to-be-processed content associated with the content publishing subject to be evaluated into a target machine learning model to obtain a processing result.

[0126] The processing result represents the evaluation result for the content publishing subject.

[0127] The target machine learning model is trained according to the training method of the machine learning model described above.

[0128] According to the information evaluation method of the embodiment of the present disclosure, since the target machine learning model is trained according to the training method of the machine learning model described above, the target machine learning model is more accurate when performing the evaluation task for the content publishing subject, and the information evaluation method of the embodiment of the present disclosure can accurately represent the evaluation result for the content publishing subject by inputting the to-be-processed content associated with the content publishing subject to be evaluated into the target machine learning model to obtain a processing result. The relevant principles and technical effects can be referred to the description of the above-mentioned sample generation method and the training method of the machine learning model, which will not be repeated here.

[0129] It should be noted that the information evaluation method according to the embodiment of the present disclosure can accurately evaluate the content publishing subject, for example, the ability of the content publishing subject to obtain gain, and through the accurate evaluation result, the maintenance and operation of the related platform can be facilitated, and the content publishing subject can publish content on the related platform.

[0130] Exemplarily, the information evaluation method according to another embodiment of the present disclosure may, for example, use the following embodiments to implement the specific example of inputting the to-be-processed content associated with the content publishing subject to be evaluated into the target machine learning model to obtain a processing result: input the to-be-processed content into a first level learner of the target machine learning model to obtain a plurality of initial output results, wherein the first level learner includes a plurality of sub-learners, each sub-learner outputs an initial output result; and input the initial output results into a second level learner of the target machine learning model to obtain a target output result, wherein the second level learner is used to assign corresponding weights to the plurality of initial output results, and the target output result is obtained according to the plurality of initial output results and the corresponding weights.

[0131] Exemplarily, the information evaluation method according to still another embodiment of the present disclosure may, for example, further include: performing feature extraction on the to-be-processed content to obtain initial features; and performing at least one of the following operations on the initial features: feature discretization, feature selection, and missing value filling, to obtain target features, wherein the target features are used for inputting the target machine learning model.

[0132] Exemplarily, the information evaluation method according to still another embodiment of the present disclosure may, for example, further include: performing feature extraction on the to-be-processed content to obtain initial features; and performing at least one of the following operations on the initial features: feature discretization, feature selection, and missing value filling, to obtain target features, wherein the target features are used for inputting the target machine learning model.

[0133] Exemplarily, the information evaluation method according to still another embodiment of the present disclosure may, for example, further include: performing feature extraction on the to-be-processed content to obtain initial features; and performing at least one of the following operations on the initial features: feature discretization, feature selection, and missing value filling, to obtain target features, wherein the target features are used for inputting the target machine learning model.

[0134] Exemplarily, the information evaluation method according to still another embodiment of the present disclosure may, for example, further include: performing feature extraction on the to-be-processed content to obtain initial features; and performing at least one of the following operations on the initial features: feature discretization, feature selection, and missing value filling, to obtain target features, wherein the target features are used for inputting the target machine learning model.

[0135] For example, taking two months as a preset time period, for the full amount of content publishing subjects in two months, the sample generation method according to the above embodiments may, for example, divide the relevant content publishing subjects with gain in the full amount of content publishing subjects in two months into positive samples and negative samples, and the relevant content publishing subjects without gain in the full amount of content publishing subjects are taken as the content publishing subjects to be evaluated.

[0136] According to the information evaluation method of the embodiments of the present disclosure, by taking the content publishing subjects to be evaluated and the subject samples to correspond to the same preset time period, the input of the target machine learning model and the subject samples used for training the target machine learning model can be synchronized in the time dimension, and the evaluation accuracy of the target machine learning model on the content publishing subjects to be evaluated for obtaining gain ability can be improved.

[0137] Figure 7 A block diagram of a sample generation apparatus according to an embodiment of the present disclosure is schematically shown.

[0138] As shown in Figure 7 The sample generation apparatus 700 according to the embodiments of the present disclosure may, for example, include a target information determination module 710, a label determination module 720, and a subject sample generation module 730.

[0139] The target information determination module 710 is configured to determine target information according to the publishing content of each content publishing subject.

[0140] The label determination module 720 is configured to determine a label for each content publishing subject according to the target information.

[0141] The subject sample generation module 730 is configured to generate a subject sample according to the label for any one content publishing subject.

[0142] For example, the target information determination module includes a target information determination sub-module configured to determine the total amount of publishing content of the content publishing subject in the preset time period, the target publishing content with the gain, and the gain value of each target publishing content according to the publishing content of the content publishing subject.

[0143] For example, the label determination module includes a gain determination sub-module configured to determine the total gain and the average gain of each target publishing content according to the gain value of the target publishing content of the content publishing subject in the preset time period, a content gain ratio determination sub-module configured to determine the content gain ratio according to the number of target publishing content and the number of total amount of publishing content of the content publishing subject in the preset time period, and a label determination sub-module configured to determine the label according to the total gain, the average gain, and the content gain ratio.

[0144] For example, the label includes positive sample labels and negative sample labels, and the subject sample includes positive samples and negative samples; the device further includes a sample expansion module configured to expand the number of positive samples according to the positive samples in a case where the sample ratio between the number of positive samples and the number of negative samples is less than or equal to a preset threshold, to obtain expanded positive samples, wherein the sample ratio between the number of expanded positive samples and the number of negative samples is greater than the preset threshold.

[0145] Figure 8 A block diagram of a training device of a machine learning model according to an embodiment of the present disclosure is shown schematically.

[0146] As shown in Figure 8 The training device 800 of the machine learning model of the embodiment of the present disclosure includes, for example, an output result determination module 810, a feedback value determination module 820, and a target machine learning model determination module 830.

[0147] The output result determination module 810 is configured to input the subject sample into the initial machine learning model to obtain the output result of the subject sample.

[0148] The feedback value determination module 820 is configured to determine the feedback value according to the output result and a loss function.

[0149] The target machine learning model determination module 830 is configured to adjust the model parameters of the initial machine learning model according to the feedback value to obtain the target machine learning model.

[0150] The subject sample is obtained by the sample generation device of the above embodiment.

[0151] Exemplarily, the output result determination module comprises: an initial output result first determination submodule, configured to input the subject sample into a first level learner of the initial machine learning model to obtain a plurality of initial output results, wherein the first level learner comprises a plurality of sub-learners, and each sub-learner outputs an initial output result; and a target output result second determination submodule, configured to input the initial output results into a second level learner of the initial machine learning model to obtain a target output result, wherein the second level learner is configured to assign corresponding weights to the plurality of initial output results, and the target output result is obtained according to the plurality of initial output results and the corresponding weights.

[0152] Exemplarily, the plurality of sub-learners comprise at least two of the following: a random forest model, a neural network model, an extreme random tree model, and an adaptive boosting model.

[0153] Exemplarily, the training device of the machine learning model according to another embodiment of the present disclosure can further comprise: a subject sample feature determination module, configured to perform feature extraction on the subject sample to obtain subject sample features; and a feature first processing module, configured to perform at least one of the following operations on the subject sample features: feature discretization, feature selection, and missing value filling, to obtain target features, wherein the target features are used to input the initial machine learning model.

[0154] Exemplarily, the subject sample features comprise at least one of the following categories: basic attributes, identity attributes, and operation data of the content publishing subject.

[0155] Figure 9 A block diagram of the information evaluation device according to an embodiment of the present disclosure is schematically shown.

[0156] As shown in Figure 9 The information evaluation device 900 of the embodiment of the present disclosure comprises, for example, a processing result determination module 910.

[0157] The processing result determination module 910 is configured to input the to-be-processed content associated with the content publishing subject to be evaluated into a target machine learning model to obtain a processing result, wherein the processing result represents an evaluation result for the content publishing subject; and the target machine learning model is trained by the training device of the machine learning model according to any one of claims 20-24.

[0158] Exemplarily, the processing result determination module comprises: an initial output result second determination submodule, configured to input the to-be-processed content into a first level learner of the target machine learning model to obtain a plurality of initial output results, wherein the first level learner comprises a plurality of sub-learners, and each sub-learner outputs an initial output result; and a target output result second determination submodule, configured to input the initial output results into a second level learner of the target machine learning model to obtain a target output result, wherein the second level learner is configured to assign corresponding weights to the plurality of initial output results, and the target output result is obtained according to the plurality of initial output results and the corresponding weights.

[0159] Exemplarily, the plurality of sub-learners comprise at least two of the following: a random forest model, a neural network model, an extreme random tree model, and an adaptive boosting model.

[0160] Exemplarily, the information evaluation device according to another embodiment of the present disclosure further comprises: an initial feature determination module, configured to perform feature extraction on the to-be-processed content to obtain initial features; and a feature second processing module, configured to perform at least one of the following operations on the initial features: feature discretization, feature selection, and missing value filling, to obtain target features, wherein the target features are used to input the target machine learning model.

[0161] Exemplarily, the initial features comprise at least one of the following categories: basic attributes, identity attributes, and operation data of the to-be-evaluated content publishing subject.

[0162] Exemplarily, the to-be-evaluated content publishing subject corresponds to the same preset time period as the subject sample.

[0163] It should be understood that the embodiments of the device part of the present disclosure correspond to the same or similar as the embodiments of the method part of the present disclosure, and the technical problems solved and the technical effects achieved are also the same or similar, which will not be repeated here.

[0164] According to embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0165] Figure 10A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0166] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0167] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0168] The computing unit 1001 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, and the like. The computing unit 1001 performs various methods and processes described above, such as the sample generation method, the training method of the machine learning model, the information evaluation method. For example, in some embodiments, the sample generation method, the training method of the machine learning model, the information evaluation method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded onto the RAM 1003 and executed by the computing unit 1001, one or more steps of the sample generation method, the training method of the machine learning model, the information evaluation method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 can be configured to perform the sample generation method, the training method of the machine learning model, the information evaluation method by any other appropriate means, such as by means of firmware.

[0169] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0170] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0171] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0172] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0173] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0174] The computer system can include clients and servers. This relationship can be

[0175] It should be understood that the procedures shown above can be re-ordered, added to, or removed from, while still being within the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the present disclosure are achieved, and are not limited herein.

[0176] The specific embodiments described above are not intended to be limiting, and persons skilled in the art will appreciate that various modifications, combinations, sub-combinations and alternatives can be made to the specific embodiments without departing from the spirit and scope of the disclosure. Any alternatives, modifications, equivalents, and the like, along with many apparent variations that would be apparent to one of ordinary skill in the art once given this disclosure, are intended to be included within the scope of the present disclosure.

Claims

1. A sample generation method, comprising: determining, for any one content publisher, target information according to published content of the content publisher, the target information comprising total published content of the content publisher in a preset time period, target published content with gain, and gain value of each of the target published content; determining a label for each of the content publisher according to the target information; and generating, for any one of the content publisher, a subject sample according to the label, wherein the determining the label for each of the content publisher according to the target information comprises: determining a total gain number and a gain average for each of the target published content according to the gain value of the target published content of the content publisher in the preset time period; determining a content gain ratio according to a number of the target published content and a number of the total published content of the content publisher in the preset time period; and determining the label according to the total gain number, the gain average, and the content gain ratio. The label comprises a positive sample label and a negative sample label, and the subject sample comprises a positive sample and a negative sample. The method further comprises: in a case where a sample ratio between a number of the positive sample and a number of the negative sample is less than or equal to a preset threshold, expanding the number of the positive sample according to the positive sample to obtain expanded positive samples, wherein a sample ratio between the number of the expanded positive sample and the number of the negative sample is greater than the preset threshold. 3.A training method of a machine learning model, comprising: inputting a subject sample into an initial machine learning model to obtain an output result of the subject sample; determining a feedback value according to the output result and a loss function; and adjusting a model parameter of the initial machine learning model according to the feedback value to obtain a target machine learning model, wherein the subject sample is obtained by the sample generation method of any one of claims 1-2. The inputting the subject sample into the initial machine learning model to obtain the output result of the subject sample comprises: inputting the subject sample into a first level learner of the initial machine learning model to obtain a plurality of initial output results, wherein the first level learner comprises a plurality of sub-learners, each of the sub-learners outputs one of the initial output results; and inputting the initial output results into a second level learner of the initial machine learning model to obtain a target output result, wherein the second level learner is configured to assign corresponding weights to the plurality of initial output results, and the target output result is obtained according to the plurality of initial output results and the corresponding weights. The plurality of sub-learners comprises at least two of the following: a random forest model, a neural network model, an extreme random tree model, and an adaptive boosting model. 6.The method of any one of claims 3-5, further comprising: performing feature extraction on the subject sample to obtain subject sample features. ​ ​ ​ ​ 2. The method of claim 1, wherein, ​ ​ ​ ​ ​ ​ ​ ​ 4. The method of claim 3, wherein, ​ ​ ​ 5. The method of claim 4, wherein, ​ ​ ​ At least one of the following operations is performed on the subject sample features: feature discretization, feature selection, and missing value filling, to obtain target features, wherein the target features are used to input the initial machine learning model.

7. The method of claim 6, wherein, The subject sample features include at least one of the following categories: basic attributes, identity attributes, and operation data of the content publishing subject.

8. An information evaluation method, comprising: inputting the to-be-processed content associated with the content publishing subject to be evaluated into a target machine learning model to obtain a processing result, wherein the processing result represents an evaluation result for the content publishing subject; and the target machine learning model is trained according to the machine learning model training method in any one of claims 3-7.

9. The method of claim 8, wherein, The inputting the to-be-processed content associated with the content publishing subject to be evaluated into a target machine learning model to obtain a processing result includes: inputting the to-be-processed content into a first-level learner of the target machine learning model to obtain a plurality of initial output results, wherein the first-level learner includes a plurality of sub-learners, and each sub-learner outputs one initial output result; and inputting the initial output results into a second-level learner of the target machine learning model to obtain a target output result, wherein the second-level learner is configured to assign corresponding weights to the plurality of initial output results, and the target output result is obtained according to the plurality of initial output results and the corresponding weights.

10. The method of claim 9, wherein, The plurality of sub-learners include at least two of the following: a random forest model, a neural network model, an extreme random tree model, and an adaptive boosting model.

11. The method of any one of claims 8-10, further comprising: extracting features from the to-be-processed content to obtain initial features; performing at least one of the following operations on the initial features: feature discretization, feature selection, and missing value filling, to obtain target features, wherein the target features are used to input the target machine learning model.

12. The method of claim 11, wherein, The initial features include at least one of the following categories: basic attributes, identity attributes, and operation data of the content publishing subject to be evaluated.

13. The method of any one of claims 8-10, wherein, The content publishing subject to be evaluated corresponds to the same pre-set time period as the subject sample.

14. A sample generation apparatus, comprising: a target information determination module configured to determine, for any one content publishing subject, target information according to publishing content of the content publishing subject, the target information including all publishing content of the content publishing subject within a pre-set time period, target publishing content with gain, and gain values of each target publishing content; a label determination module configured to determine, according to the target information, a label for each content publishing subject; and a subject sample generation module configured to generate, for any one content publishing subject, a subject sample according to the label, wherein the label determination module includes: a gain determination sub-module configured to determine a total gain and a gain average for each target publishing content according to gain values of the target publishing content of the content publishing subject within the pre-set time period. a content gain ratio determining sub-module, configured to determine a content gain ratio according to a quantity of the target publishing content and a quantity of the total publishing content of the content publishing subject in the preset time period; and a label determining sub-module, configured to determine the label according to the total gain number, the gain average value, and the content gain ratio.

15. The apparatus of claim 14, wherein, The label includes positive sample labels and negative sample labels, and the subject samples include positive samples and negative samples; the device further includes: a sample expansion module, configured to expand the quantity of the positive samples according to the positive samples in a case where a sample ratio between the quantity of the positive samples and the quantity of the negative samples is less than or equal to a preset threshold, to obtain expanded positive samples, wherein a sample ratio between the quantity of the expanded positive samples and the quantity of the negative samples is greater than the preset threshold.

16. A device for training a machine learning model, comprising: an output result determining module, configured to input a subject sample into an initial machine learning model to obtain an output result of the subject sample; a feedback value determining module, configured to determine a feedback value according to the output result and a loss function; and a target machine learning model determining module, configured to adjust model parameters of the initial machine learning model according to the feedback value to obtain a target machine learning model, wherein the subject sample is obtained by the sample generation device of any one of claims 14-15.

17. The apparatus of claim 16, wherein, The output result determining module includes: an initial output result first determining sub-module, configured to input the subject sample into a first level learner of the initial machine learning model to obtain a plurality of initial output results, wherein the first level learner includes a plurality of sub-learners, and each of the sub-learners outputs one of the initial output results; and a target output result second determining sub-module, configured to input the initial output results into a second level learner of the initial machine learning model to obtain a target output result, wherein the second level learner is configured to assign corresponding weights to the plurality of initial output results, and the target output result is obtained according to the plurality of initial output results and the corresponding weights.

18. The apparatus of claim 17, wherein, The plurality of sub-learners include at least two of the following: a random forest model, a neural network model, an extreme random tree model, and an adaptive boosting model.

19. The device of any one of claims 16-18, further comprising: a subject sample feature determining module, configured to perform feature extraction on the subject sample to obtain subject sample features; a feature first processing module, configured to perform at least one of the following operations on the subject sample features: feature discretization, feature selection, and missing value filling, to obtain target features, wherein the target features are used to input the initial machine learning model.

20. The apparatus of claim 19, wherein, The subject sample features include at least one of the following categories: basic attributes, identity attributes, and operation data of the content publishing subject.

21. An information evaluation device, comprising: The processing result determination module is configured to input the to-be-processed content associated with the content publisher to be evaluated into a target machine learning model to obtain a processing result, where the processing result represents an evaluation result for the content publisher; and the target machine learning model is trained by the training device of the machine learning model according to any one of claims 16-20.

22. The apparatus of claim 21, wherein, The processing result determination module includes: An initial output result second determination submodule configured to input the to-be-processed content into a first level learner of the target machine learning model to obtain a plurality of initial output results, where the first level learner includes a plurality of sub-learners, and each of the sub-learners outputs one of the initial output results; and A target output result second determination submodule configured to input the initial output results into a second level learner of the target machine learning model to obtain a target output result, where the second level learner is configured to assign corresponding weights to the plurality of initial output results, and the target output result is obtained according to the plurality of initial output results and the corresponding weights.

23. The apparatus of claim 22, wherein, The plurality of sub-learners include at least two of the following: a random forest model, a neural network model, an extreme random tree model, and an adaptive boosting model.

24. The apparatus of any one of claims 21-23, further comprising: An initial feature determination module configured to perform feature extraction on the to-be-processed content to obtain initial features; A feature second processing module configured to perform at least one of the following operations on the initial features: feature discretization, feature selection, and missing value filling, to obtain target features, where the target features are used to input the target machine learning model.

25. The apparatus of claim 24, wherein, The initial features include at least one of the following categories: basic attributes, identity attributes, and operation data of the content publisher to be evaluated.

26. The apparatus of any one of claims 21-23, wherein, The content publisher to be evaluated and the subject sample correspond to a same preset time period.

27. An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-13.

28. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-13.

29. A computer program product, comprising a computer program stored on at least one of a readable storage medium and an electronic device, and the computer program, when executed by a processor, implements the method of any one of claims 1-13.

Images