Multi-label model selection and effect evaluation method, system, equipment and medium

Abstract

The invention provides a multi-label model selection and effect evaluation method and system, equipment and a medium. The method comprises the steps of obtaining a corresponding service label based on published content of a user; obtaining a service label weight of a multi-label model based on the propagation area of the service label; obtaining an effect evaluation index of the multi-label model based on the service label weight; and obtaining an effect evaluation result of the multi-label model based on the effect evaluation index. According to the method, the service label weight can be intelligently calculated, the inaccuracy of manually defining the model label weight can be avoided, the weights of dozens of hundreds of service labels can be efficiently and intelligently adjusted according to the change of the service form, and then the effect of the multi-label model is efficiently evaluated. Moreover, the method can accurately judge the advantages and disadvantages of the effects between different multi-label models through the effect evaluation indexes, and enables the capability of the model selected in the re-iterative optimization process to better meet the requirements of services.

Description

Technical Field

[0001] The present application belongs to the technical field of model evaluation, and in particular relates to a method, system, device and medium for selecting and evaluating the effect of a multi-label model. Background Art

[0002] As business changes and the collected training data evolve, models need to be iteratively optimized from time to time to ensure that their capabilities better meet business expectations. To this end, it's necessary to evaluate the performance of multi-label models to select the model that best meets business objectives.

[0003] At present, the existing multi-label model effect evaluation indicators include the model's recall rate, precision rate, f1-score indicator, mAP indicator (multi-label model), accuracy rate, etc. Generally speaking, the existing methods will directly judge the model effect based on one or more evaluation indicators, and use this to judge the performance of the models. For example, when the new model exceeds the old model in both recall rate and precision rate, the new model will be judged to be better than the old model. When the recall rate and precision rate are different and cannot be judged directly, the performance of the new and old models will be judged based on comprehensive indicators such as f1-score, mAP, and accuracy rate. The existing methods have more accurate effect judgment results when the performance gap between models is large or the number of labels is small.

[0004] However, in the content risk control business, existing methods find it difficult to accurately evaluate the effectiveness of multi-label models. On the one hand, some multi-label models have hundreds of business labels, and the weight of each business label, if defined manually, is often not accurate enough. On the other hand, with the continuous optimization and iteration of the model, the performance gap between the old and new models is narrowing, and different reasoning results may only appear on a small part of the content data, and different business content has different weights. For example, the risks of newly emerging risk content, current public opinion content, etc. have more important weights than those of outdated content. In the reasoning results of the business expectation model, it is hoped that more recall of higher-weighted content can be achieved. However, existing methods cannot perceive changes in business forms and cannot adjust the weights of business labels according to changes in business forms, resulting in inaccurate final judgment results.

[0005] Therefore, how to provide a multi-label model selection and effect evaluation method, system, equipment and medium so that the selected model capabilities can better meet business requirements has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0006] The purpose of this application is to provide a method, system, device and medium for selecting and evaluating the effects of a multi-label model, so as to make the selected model capabilities more in line with business requirements.

[0007] In a first aspect, the present application provides an effect evaluation method of a multi-label model, comprising:

[0008] obtaining a corresponding business label based on the published content of a user;

[0009] obtaining a business label weight of the multi-label model based on the propagation area of the business label;

[0010] obtaining an effect evaluation index of the multi-label model based on the business label weight;

[0011] obtaining an effect evaluation result of the multi-label model based on the effect evaluation index.

[0012] In an implementation form of the first aspect, the business label weight of the multi-label model is obtained based on the propagation area of the published content, comprising:

[0013] obtaining a browsing parameter of the published content;

[0014] obtaining a propagation area of the corresponding business label based on the browsing parameter of the published content;

[0015] obtaining the business label weight based on the propagation area of the business label.

[0016] In an implementation form of the first aspect, the browsing parameter of the published content comprises a browsing time period window, and a browsing frequency of the published content within the browsing time period window, a de-duplication quantity of browsing users.

[0017] In an implementation form of the first aspect, the propagation area of the corresponding business label based on the browsing parameter of the published content comprises:

[0018] obtaining a multiplication result of the browsing frequency and an adjustable parameter;

[0019] obtaining an addition result of the multiplication result and the de-duplication quantity of the browsing users;

[0020] comparing the addition result with the browsing time period window to obtain a unit time propagation area of the business label.

[0021] In an implementation form of the first aspect, the business label weight is obtained based on the propagation area of the business label, comprising: normalizing the unit time propagation area of the business label to obtain a normalized business label weight.

[0022] In an implementation form of the first aspect, the normalizing the unit time propagation area of the business label to obtain a normalized business label weight comprises:

[0023] obtaining a first propagation area and a second propagation area; the first propagation area is a maximum unit time propagation area of the business label; the second propagation area is a minimum unit time propagation area of the business label;

[0024] obtaining a subtraction result of the first propagation area and the second propagation area;

[0025] comparing the addition result and the subtraction result to obtain the normalized business label weight.

[0026] In an implementation form of the first aspect, obtaining an effect evaluation index based on the business label weight comprises:

[0027] obtaining a recall quantity of the business label;

[0028] obtaining the effect evaluation index based on the recall quantity of the business label and the business label weight.

[0029] In an implementation form of the first aspect, obtaining the effect evaluation index based on the recall quantity and the business label weight comprises:

[0030] obtaining a recall score; the recall score is a multiplication result of the recall quantity of each business label and the business label weight of each business label;

[0031] summing all the recall scores to obtain the effect evaluation index.

[0032] In an implementation form of the first aspect, obtaining an effect evaluation result of the multi-label model based on the effect evaluation index comprises: the effect evaluation index and the effect evaluation result are in a positive correlation relationship.

[0033] In a second aspect, the application provides a multi-label model selection method, comprising selecting a multi-label model meeting a business target through the effect evaluation result obtained by the method of the first aspect.

[0034] In a third aspect, the application provides a multi-label model effect evaluation system, comprising:

[0035] an obtaining module configured to obtain a corresponding business label based on the publishing content of a user;

[0036] a weight module configured to obtain a business label weight of a multi-label model based on a propagation area of the business label;

[0037] an evaluation module configured to obtain an effect evaluation index of the multi-label model based on the business label weight;

[0038] an evaluation module configured to obtain an effect evaluation result of the multi-label model based on the effect evaluation index

[0039] In a fourth aspect, the present application provides an electronic device comprising: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the electronic device performs any of the methods described above.

[0040] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements any of the methods described above when executed by a processor.

[0041] As described above, the multi-label model selection and effect evaluation method, system, device, and medium described in this application have the following beneficial effects:

[0042] (1) This application calculates the business tag weight based on the dissemination area of ​​the published content, avoiding the inaccuracy of artificially defined model tag weights.

[0043] (2) This application can efficiently and intelligently adjust the weights of multiple business labels according to changes in business forms, and thus efficiently evaluate the effectiveness of the multi-label model.

[0044] (3) This application designs an effect evaluation index for content security business based on the business label weight, so as to accurately judge the effectiveness of different multi-label models, so that the model capabilities selected in the iterative optimization process are more in line with business requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Shown is a flowchart of an embodiment of a method for evaluating the effectiveness of a multi-label model of the present application.

[0046] Figure 2 Shown is a flowchart of an embodiment of a method for evaluating the effectiveness of a multi-label model of the present application.

[0047] Figure 3 Shown is a flowchart of an embodiment of a method for evaluating the effectiveness of a multi-label model of the present application.

[0048] Figure 4 Shown is a flowchart of an embodiment of a method for evaluating the effectiveness of a multi-label model of the present application.

[0049] Figure 5 Shown is a flowchart of an embodiment of a method for evaluating the effectiveness of a multi-label model of the present application.

[0050] Figure 6 Shown is a flowchart of an embodiment of a method for evaluating the effectiveness of a multi-label model of the present application.

[0051] Figure 7Shown is a flowchart of a method for selecting a multi-label model in one embodiment of the present application.

[0052] Figure 8 Shown is a structural diagram of a key person name correction system in one embodiment of the present application.

[0053] Figure 9 Shown is a schematic structural diagram of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0054] The following describes the embodiments of the present application through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0055] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. Therefore, the illustrations only show components related to the present application and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0056] In addition, the descriptions of "first", "second", etc. in this application are for descriptive purposes only and should not be understood as indicating or implying their relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0057] Due to some differences in the model neural networks, commonly used image or text models are referred to as binary classification, multi-classification, and multi-label models. In the following embodiments of this application, since the multi-classification model is composed of multiple labels, the multi-classification and multi-label models are collectively referred to as multi-label models, and the binary classification model is a special model with only two labels.

[0058] The following embodiments of the application provide a multi-label model selection and effect evaluation method, system, device and medium, which can be applied in content security business. For example, in an online content platform, the more widely the content published by a user is spread, the more it will become the focus of review. Generally speaking, the business label corresponding to the content published by the user is obtained by content risk control model reasoning, and the reasoning effect of the model is crucial for content risk control. In addition, sometimes even special artificial additional review of hot content is required to prevent risk content from being widely spread. In this process, in order to save labor costs, only the data hit by the model is reviewed in real time, and all the data missed by the model are passed. However, due to the difficulty in deploying human resources, it is difficult to accurately review the model hit data in time. Therefore, the business requires that the data quantity hit by the content risk control model should be within a certain range, and in the case of limited hit quantity, it is the main goal of the content security business to try to improve the recall quantity of each business label. The technology provided in the application can accurately evaluate the recall effect of the model, so as to select a content risk control model that is more in line with the business goal, so as to accurately recall the business label of the content published by the user, achieve the content risk control goal, and reduce the cost and time of artificial review.

[0059] The technical solutions in the embodiments of the application will be described in detail below with reference to the drawings in the embodiments of the application.

[0060] Please refer to Figure 1 , which shows a flowchart of the effect evaluation method of the multi-label model of the application in an embodiment.

[0061] As shown in Figure 1 , the embodiment provides an effect evaluation method of a multi-label model, which includes the following steps S1 to S4.

[0062] S1: obtaining a corresponding business label based on the published content of a user.

[0063] Specifically, in an online content platform, various contents published by a plurality of users will be generated at the same time. Each content data will obtain a corresponding business label through business model reasoning or artificial review.

[0064] Taking a content security business as an example, the content published on an online content platform will be subjected to content risk control model reasoning to recall business safety labels, or real-time artificial review of the content data hit by the model to obtain corresponding business safety labels. In other embodiments, in order to save labor costs, other methods that are more cost-effective than artificial review will also be used to recall business labels. For example, based on existing rules, the content is reviewed, including screening user ID rules, so as to review the content data to obtain corresponding business labels.

[0065] It should be noted that if the corresponding tag does not appear, it can be regarded that the content included in the business tag has not been disseminated, and there is no need to perform subsequent dissemination area calculation.

[0066] It should be noted that a model hit refers to the output of a business label after model inference on content data. Model recall, on the other hand, refers to the output of a business label after model inference on content data, which is consistent with the actual label of the content data. The method provided in this application can accurately evaluate the effectiveness of multi-label models, thereby selecting a multi-label model that better meets business objectives, thereby increasing the model's recall of business labels, effectively achieving business objectives while reducing labor costs.

[0067] S2: Obtaining a service label weight of a multi-label model based on the propagation area of ​​the service label.

[0068] Specifically, such as Figure 2 As shown, step S2 includes steps S21 to S23.

[0069] S21: Obtain browsing parameters of the published content.

[0070] Specifically, the browsing parameters of the published content include a browsing time window, the number of browsing times of the published content within the browsing time window, and the number of duplicate browsing users.

[0071] In some embodiments, the unit of the browsing time period window is seconds. In other embodiments, the browsing time period can also be set to minutes. The unit of the browsing time period can be set according to business needs so that hot content can be counted more timely.

[0072] It should be noted that the deduplicated number of browsing users refers to the number of different browsing users, that is, the number of users who can actually browse the published content. By counting the deduplicated number of browsing users, the actual dissemination area of ​​the published content can be obtained more accurately.

[0073] In some embodiments, existing front-end embedding technologies may be used to count the number of visits, the number of duplicate browsing users, and the visit time window.

[0074] Front-end tracking is a technical term in the data field, primarily referring to the technologies and implementation processes for capturing, processing, and transmitting specific user behaviors or business processes. It is generally used for feedback in recommendation systems, monitoring and analyzing user behavior, and statistical analysis of the effectiveness of new features or operational activities. By capturing the number of visits and the deduplicated number of visiting users within a viewing time window through front-end tracking, we can accurately capture the visitor status of published content, providing a basis for subsequent calculation of dissemination area.

[0075] S22: Obtain the propagation area of the business tag corresponding to the publication content based on the browsing parameter of the publication content.

[0076] Specifically, as shown in FIG. 2, step S22 includes steps S221-S223. Figure 3

[0077] S221: Obtain the multiplication result of the browsing times and the adjustable parameter.

[0078] S222: Obtain the addition result of the multiplication result and the de-duplication number of the browsing users.

[0079] S223: Compare the addition result with the browsing time period window to obtain the unit time propagation area of the business tag.

[0080] Specifically, assuming that the unit time propagation area of the business tag is R, then

[0081]

[0082] wherein a is the browsing times, b is the de-duplication number of the browsing users, c is the time period window, and T is a hyperparameter.

[0083] It should be noted that T can be adjusted according to different business forms, and thus is referred to as an adjustable parameter in step S221.

[0084] In some embodiments, the browsing time period window is 60s, and thus the unit time propagation area of the business tag obtained according to the above formula is actually the propagation area of the business tag per second. In other embodiments, the browsing time period can also be in units of minutes, and thus the unit time propagation area of the business tag obtained according to the above formula is actually the propagation area of the business tag per minute. The unit of the browsing time period can be set according to business needs, so as to be able to more timely statistics of hot content.

[0085] S23: Obtain the business tag weight based on the propagation area of the business tag.

[0086] Specifically, in order to make the weight calculation of the business tag more accurate, step S23 performs normalization processing on the unit time propagation area of the business tag to obtain a normalized business tag weight.

[0087] ​Normalization refers to mapping data to a specified range, removing the dimensions and dimensional units of data of different dimensions to ensure comparability between data indicators. After data normalization, the original data is put into the same order of magnitude, making it suitable for comprehensive comparative evaluation. Therefore, step S23 normalizes the propagation area per unit time of the service tag, so that the service tag weight finally calculated is a proportional value of the propagation area of ​​the service tag, thereby more accurately obtaining the service tag weight calculated based on the propagation area.

[0088] Specifically, such as Figure 4 As shown, step S23 includes steps S231 to S233.

[0089] S231: Acquire a first propagation area and a second propagation area.

[0090] Specifically, the first propagation area is the largest propagation area per unit time of the service tag; and the second propagation area is the smallest propagation area per unit time of the service tag.

[0091] In some embodiments, within the same browsing time window, the online content platform may generate multiple user-published content. Each piece of content will actually obtain a corresponding service tag through model inference or manual review in step S1. Subsequently, step S22 is used to obtain the per-unit-time propagation area of ​​each service tag. In this case, the largest per-unit-time propagation area of ​​the service tag is used as the first propagation area, and the smallest per-unit-time propagation area of ​​the service tag is used as the second propagation area.

[0092] S232: Obtain a subtraction result between the first propagation area and the second propagation area.

[0093] S233: Compare the addition result and the subtraction result to obtain the normalized service label weight.

[0094] Specifically, assuming that the first propagation area is M1 and the second propagation area is M2, the addition result is obtained according to S221 and S222. Then, for a certain service label Li, its normalized service label weight Fi is:

[0095]

[0096] Where a is the number of visits; b is the number of duplicate browsing users; c is the time period window; and T is a hyperparameter.

[0097] It's important to note that M1 and M2 in the above formula are actually the service tag's propagation area per unit time, while the number of visits a and the number of duplicated users b are visit parameters within a specific visit time window, which also include time units. Therefore, in the above calculation formula, the normalized service tag weight does not actually include the visit time window (time unit), but directly compares the ratio of the service tag's propagation area.

[0098] Step S2 calculates the business tag weight based on the dissemination area of ​​the user-published content, thereby avoiding the inaccuracy of manually defined tag weights. It can also efficiently and intelligently adjust the weights of dozens or hundreds of business tags according to changes in business forms, providing a solid foundation for the effectiveness evaluation of the multi-tag model.

[0099] S3: Obtaining an effect evaluation index of the multi-label model based on the business label weight.

[0100] Specifically, such as Figure 5 As shown, step S3 includes steps S31 and S32.

[0101] S31: Obtain the recall quantity of the service tag.

[0102] S32: Acquire the effect evaluation index based on the recall quantity of the service tag and the service tag weight.

[0103] In some embodiments, regarding content security services, in order to save labor costs, real-time manual review is usually only performed on data that is hit by the content risk control model, and all data that is not hit by the model is passed, or recalled through other methods that are more cost-effective than manual review, thereby improving review efficiency while ensuring content security. It can be seen that the recall number of each business tag of the content risk control model is the main goal of the content security service. Therefore, if you want to evaluate the effectiveness of the content risk control model, you first need to obtain the recall number of the business tags of the content risk control model, and then obtain the effect evaluation index based on the recall number and the business tag weight to evaluate the model effect.

[0104] Specifically, such as Figure 6 As shown, step S32 includes step S321 and step S322.

[0105] S321: Obtain the recall score.

[0106] Specifically, the recall score is the product of the recall quantity of each service tag and the weight of each service tag.

[0107] In some embodiments, the content risk control model will infer the content data on the online content platform to hit or recall business tags. As mentioned above, the number of recalls of business tags of the content risk control model is the main goal of the content security business. Therefore, step S31 needs to calculate the recall score. Since the weight of each business tag has been calculated in step S2, the more important business tags (business tags with high weights) a content risk control model recalls, the higher the recall score will be, which means it is more in line with the goal of content security business.

[0108] Specifically, let the recall score of a business tag be Q i , then:

[0109] Q i =X i F i

[0110] Among them, X i is the recall number of business tags, F i Service label weight.

[0111] Wherein, i=1, 2...n, and n is the number of service labels.

[0112] S322: Sum all the recall scores to obtain the effect evaluation index.

[0113] In some embodiments, when a content risk control model recalls more important business tags (business tags with high weights), the recall score will be higher. At this time, the effect evaluation index score obtained by summing up all the recall scores in step S22 will also be higher, which means that the effect of the content risk control model is better and more in line with business expectations.

[0114] Specifically, assuming that the effect evaluation index is S, then:

[0115]

[0116] Where n is the number of service tags.

[0117] S4: Obtaining an effect evaluation result of the multi-label model based on the effect evaluation index.

[0118] Specifically, the effect evaluation index is positively correlated with the effect evaluation result.

[0119] As mentioned above, when a multi-label model recalls more important business tags (business tags with high weights), the effect evaluation index score obtained in step S3 will be higher, which means that the effect evaluation result of the model is better and more in line with the business goals.

[0120] Therefore, the business tag weights obtained by this application can avoid the inaccuracy of manually defined tag weights, and can efficiently and intelligently adjust the weights of dozens or hundreds of business tags according to changes in business forms. At the same time, this application designs an effect evaluation index based on the business tag weights, which can accurately evaluate the effect of the multi-tag model, so that the selected model can better meet the business requirements.

[0121] like Figure 7 As shown, the present application also provides a method for selecting a multi-label model. The method evaluates the effectiveness of the multi-label model in the business through the multi-label model effect evaluation method described above, so that a multi-label model that meets the business objectives can be selected during the model generation optimization process.

[0122] In some embodiments, regarding content security business, two new and old models (model one and model two) to be selected and a portion of content data are prepared, so that the effects of the two content risk control models are evaluated through the above-mentioned effect evaluation method, so as to select a content risk control model that is more in line with business security goals.

[0123] Specifically, the content data is derived from effective sampling in actual business, and the sampling method can be selected according to different businesses.

[0124] In some embodiments, a random sampling approach can be selected. This random sampling approach adheres to the principle of randomization, ensuring that each individual data point in the content data population has an independent, known, non-zero probability of being selected as an evaluation target. If the sample size of the content data is sufficiently large, the data is representative, and the randomization effect is good, the sampling results can be extrapolated to the population. Common random sampling methods include simple random sampling, systematic sampling, stratified sampling, cluster sampling, and multi-stage sampling, which can be selected based on the circumstances.

[0125] In other embodiments, bucket sampling can also be selected. Bucket sampling is also called bucket table sampling. For example, if the total number of buckets is 10, the sampling factor is set to 2, and the sampling is set to start from the first bucket, then it can be determined that the content data of five buckets needs to be retrieved for this sampling query.

[0126] Furthermore, after sampling content data, manual labeling or other methods are required to annotate the data. The annotated data is presented as samples and business tags. This allows for a clear correspondence between content data samples and business tags, enabling a better evaluation of the inference results of the content risk control model.

[0127] Specifically, after artificial labeling, the content data is input into two content risk control models respectively, and each piece of content data obtains inference results of the two models respectively. The inference results of the content risk control models are also presented in the form of business labels. Therefore, finally, the effect evaluation method of the content can be used to calculate the effect evaluation index based on the recall number of the business label and the business label weight, and the model with the optimal effect can be selected based on the high and low of the effect evaluation index.

[0128] The multi-label model selected by the multi-label model selection method provided in the application has better model effect and can better achieve the business goal.

[0129] The protection scope of the multi-label model effect evaluation method and / or the multi-label model selection method described in the embodiments of the application is not limited to the execution order of the steps listed in the embodiments. Any scheme realized by adding, replacing or replacing steps of the prior art according to the principle of the application is included in the protection scope of the application.

[0130] Please refer to Figure 8 , which shows the structural schematic diagram of the multi-label model effect evaluation system in an embodiment of the application.

[0131] As shown in Figure 8 , the embodiment provides a multi-label model effect evaluation system, which comprises:

[0132] The acquisition module 10 is configured to acquire corresponding business labels based on the published content of the user.

[0133] The weight module 20 is configured to acquire the business label weight of the multi-label model based on the propagation area of the business label.

[0134] The evaluation module 30 is configured to acquire the effect evaluation index of the multi-label model based on the business label weight.

[0135] The evaluation module 40 is configured to acquire the effect evaluation result of the multi-label model based on the effect evaluation index.

[0136] Specifically, the implementation manner of the acquisition module 10 acquiring the corresponding business label based on the published content of the user comprises: acquiring the corresponding business label of the published content of the user through business model inference or manual review.

[0137] Taking content security as an example, content published on online content platforms will be subject to content risk control model inference, recalling business security tags, or undergoing real-time manual review of content data matched by the model to obtain corresponding business security tags. In other embodiments, to save labor costs, business tags may be recalled through other more cost-effective methods compared to manual review. For example, content may be reviewed based on existing rules, including screening for user IDs, to review content data and obtain corresponding business tags.

[0138] It should be noted that if the corresponding tag does not appear, it can be regarded that the content included in the business tag has not been disseminated, and there is no need to perform subsequent dissemination area calculation.

[0139] Specifically, the implementation method of the weight module 20 obtaining the business tag weight of the multi-tag model based on the propagation area of ​​the business tag includes: obtaining the browsing parameters of the published content; obtaining the corresponding propagation area of ​​the business tag based on the browsing parameters of the published content; and obtaining the business tag weight based on the propagation area of ​​the business tag.

[0140] Specifically, the browsing parameters of the published content include a browsing time window, the number of browsing times of the published content within the browsing time window, and the number of duplicate browsing users.

[0141] Specifically, the weight module 20 acquires the corresponding propagation area of ​​the business tag based on the browsing parameters of the published content in the following manner:

[0142] Assume that the unit time propagation area of ​​the service label is R, then

[0143]

[0144] Where a is the number of visits; b is the number of duplicate browsing users; c is the time period window; and T is a hyperparameter.

[0145] It should be noted that T can be adjusted according to different business forms, so it can be called an adjustable parameter.

[0146] In some embodiments, if the browsing time window is 60 seconds, the area per unit time of the service tag calculated according to the above formula is actually the area per second of the service tag. In other embodiments, the browsing time window can also be in minutes, and the area per unit time of the service tag calculated according to the above formula is actually the area per minute of the service tag. The unit of the browsing time window can be customized according to business needs to enable more timely statistics of hot content.

[0147] Specifically, the weight module 20 acquires the service label weight based on the propagation area of ​​the service label by normalizing the propagation area per unit time of the service label to acquire a normalized service label weight.

[0148] Specifically, let the first propagation area be M1 and the second propagation area be M2. The first propagation area is the largest propagation area per unit time of the service tag; the second propagation area is the smallest propagation area per unit time of the service tag.

[0149] In some embodiments, within the same browsing time window, the online content platform generates content published by multiple users. Each piece of content actually obtains a corresponding business tag through model inference or manual review in step S1. Then, the unit time propagation area of ​​each business tag can be obtained in step S22. In this case, the unit time propagation area of ​​the largest business tag is used as the first propagation area, and the unit time propagation area of ​​the smallest business tag is used as the second propagation area. For a certain business tag Li, its normalized business tag weight Fi is:

[0150]

[0151] Where a is the number of visits; b is the number of duplicate browsing users; c is the time period window; and T is a hyperparameter.

[0152] It's important to note that M1 and M2 in the above formula are actually the service tag's propagation area per unit time, while the number of visits a and the number of duplicated users b are visit parameters within a specific visit time window, which also include time units. Therefore, in the above calculation formula, the normalized service tag weight does not actually include the visit time window (time unit), but directly compares the ratio of the service tag's propagation area.

[0153] Specifically, the evaluation module 30 obtains the effect evaluation index of the multi-label model based on the business label weight by: obtaining the recall number of the business label; and obtaining the effect evaluation index based on the recall number of the business label and the business label weight.

[0154] In some embodiments, regarding content security services, in order to save labor costs, real-time manual review is usually only performed on data that is hit by the content risk control model, and all data that is not hit by the model is passed, or recalled through other methods that are more cost-effective than manual review, thereby improving review efficiency while ensuring content security. It can be seen that the recall number of each business tag of the content risk control model is the main goal of the content security service. Therefore, if you want to evaluate the effectiveness of the content risk control model, you first need to obtain the recall number of the business tags of the content risk control model, and then obtain the effect evaluation index based on the recall number and the business tag weight to evaluate the model effect.

[0155] Specifically, let the recall score of a business tag be Q i , then:

[0156] Q i =X i F i

[0157] Among them, X i is the recall number of business tags, F i Service label weight.

[0158] Wherein, i=1, 2...n, and n is the number of service labels.

[0159] The evaluation module 30 sums up all the recall scores to obtain the effect evaluation index.

[0160] In some embodiments, when a content risk control model recalls more important business tags (business tags with high weights), the recall score will be higher. At this time, the effect evaluation index score obtained by summing up all the recall scores in step S22 will also be higher, which means that the effect of the content risk control model is better and more in line with business expectations.

[0161] Specifically, assuming that the effect evaluation index is S, then:

[0162]

[0163] Where n is the number of service tags.

[0164] Specifically, the evaluation module 40 obtains the performance evaluation result of the multi-label model based on the performance evaluation index in a manner that: the performance evaluation index is positively correlated with the performance evaluation result. That is, the more important business tags a multi-label model recalls, the higher the performance evaluation index score, indicating that the model is more consistent with business expectations, and the better the performance evaluation result.

[0165] It should be noted that the structure and principle of the above module correspond one by one to the steps in the effect evaluation method of the multi-label model, so part of the content is omitted.

[0166] The embodiment of the application provides a multi-label model effect evaluation system, which can realize the multi-label model effect evaluation method described in the application. However, the implementation device of the multi-label model effect evaluation method described in the application includes but is not limited to the structure of the multi-label model effect evaluation system listed in the embodiment. Any modification and replacement of the structure of the prior art according to the principle of the application are included in the protection scope of the application.

[0167] As shown in Figure 9 The application provides an electronic device, comprising a processor 50 and a memory 60.

[0168] The memory 60 is used for storing a computer program.

[0169] The processor 50 is used for executing the computer program stored in the memory 60, so that the electronic device executes the method described in any one of the above.

[0170] In several embodiments provided in the application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiment described above is only schematic, for example, the division of the module / unit is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, device or module or unit indirect coupling or communication connection, which can be electrical, mechanical or other forms.

[0171] The module / unit described as a separate component can be or can not be physically separated, and the component displayed as a module / unit can be or can not be a physical module, that is, it can be located in one place, or can be distributed to a plurality of network units. According to actual needs, part or all of the modules / unit can be selected to achieve the purpose of the embodiment of the application. For example, the functional modules / unit in each embodiment of the application can be integrated in a processing module, or each module / unit can be physically separated, or two or more modules / unit can be integrated in one module / unit.

[0172] Those skilled in the art should further understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in general terms. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0173] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The program is executed by a processor to implement the method described in any of the above embodiments. Those skilled in the art can understand that all or part of the steps of the method described in the above embodiments can be instructed by a program to complete by a processor. The program can be stored in a computer readable storage medium. The storage medium is a non-transitory medium, such as a random access memory, a read only memory, a flash memory, a hard disk, a solid state disk, a magnetic tape, a floppy disk, an optical disc and any combination thereof. The storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center and the like, which includes one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a digital video disc (DVD)) or a semiconductor medium (for example, a solid state disk (SSD)) and the like.

[0174] The embodiments of the present application can also provide a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in the embodiments of the present application are generated. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer or data center to another website, computer or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave and the like) mode.

[0175] When the computer program product is executed by a computer, the computer executes the method described in the above method embodiment. The computer program product can be a software installation package. When the above method is needed, the computer program product can be downloaded and executed on the computer.

[0176] The multi-label model selection and effect evaluation method, system, device and medium provided by this application can intelligently calculate the business label weights, which can not only avoid the inaccuracy of manually defining the model label weights, but also efficiently and intelligently adjust the weights of dozens or hundreds of business labels according to changes in business forms, thereby efficiently evaluating the effect of the multi-label model. In addition, this application can accurately distinguish the effectiveness of different multi-label models, so that the model capabilities selected in the iterative optimization process are more in line with business requirements.

[0177] The descriptions of the processes or structures corresponding to the above figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0178] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical concepts disclosed in this application shall be covered by the claims of this application.

Claims

1. A method for evaluating the effect of a multi-label model, characterized in that: include: Obtain corresponding business tags based on the user's published content; Obtaining a service label weight of a multi-label model based on a propagation area of ​​the service label; Obtaining an effect evaluation index of the multi-label model based on the business label weight; Obtain an effect evaluation result of the multi-label model based on the effect evaluation index.

2. The effect evaluation method of the multi-label model according to claim 1, characterized in that: Obtaining the service label weight of the multi-label model based on the propagation area of ​​the service label includes: Obtain browsing parameters of the published content; Acquire the corresponding propagation area of ​​the business tag based on the browsing parameters of the published content; The service label weight is obtained based on the propagation area of ​​the service label.

3. The effect evaluation method of the multi-label model according to claim 2, characterized in that: The browsing parameters of the published content include a browsing time window, the number of visits of the published content within the browsing time window, and the number of duplicate browsing users.

4. The effect evaluation method of the multi-label model according to claim 3, characterized in that: Acquiring the corresponding propagation area of ​​the business tag based on the browsing parameters of the published content includes: Obtaining a multiplication result of the number of visits and an adjustable parameter; Obtaining the sum of the multiplication result and the number of browsing users without duplicates; The addition result is compared with the tour time period window to obtain the propagation area per unit time of the service tag.

5. The effect evaluation method of the multi-label model according to claim 4, characterized in that: Acquiring the service label weight based on the propagation area of ​​the service label includes: Normalization is performed on the propagation area per unit time of the service label to obtain a normalized service label weight.

6. The effect evaluation method of the multi-label model according to claim 5, characterized in that: Normalizing the propagation area per unit time of the service label to obtain a normalized service label weight includes: Acquire a first propagation area and a second propagation area; the first propagation area is the largest propagation area per unit time of the service tag; the second propagation area is the smallest propagation area per unit time of the service tag; Obtaining a subtraction result between the first propagation area and the second propagation area; The addition result and the subtraction result are compared to obtain the normalized service label weight.

7. The effect evaluation method of the multi-label model according to claim 1, characterized in that: Acquiring effect evaluation indicators based on the service tag weights includes: Obtain the recall quantity of the business tag; The effect evaluation index is obtained based on the recall quantity of the service tag and the service tag weight.

8. The effect evaluation method of the multi-label model according to claim 7, characterized in that: Acquiring the effect evaluation index based on the recall quantity and the service tag weight includes: Obtaining a recall score; the recall score is the product of the number of recalls of each of the service tags and the weight of each of the service tags; All the recall scores are summed to obtain the effect evaluation index.

9. The effect evaluation method of the multi-label model according to claim 1, characterized in that: Obtaining the effect evaluation results of the multi-label model based on the effect evaluation index includes: The effect evaluation index is positively correlated with the effect evaluation result.

10. A method for selecting a multi-label model, characterized in that: The effect evaluation result obtained by the method according to any one of claims 1 to 9 selects a multi-label model that meets the business objectives.

11. A multi-label model effect evaluation system, characterized in that: include: The acquisition module is used to obtain the corresponding business tags based on the user's published content; A weight module, configured to obtain a service label weight of a multi-label model based on a propagation area of ​​the service label; An evaluation module, configured to obtain an effect evaluation index of the multi-label model based on the service label weight; An evaluation module is used to obtain an effect evaluation result of the multi-label model based on the effect evaluation index.

12. An electronic device, characterized in that: include: processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory, so that the electronic device executes the multi-label model effect evaluation method according to any one of claims 1 to 9 or the multi-label model selection method according to claim 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for evaluating the effect of a multi-label model according to any one of claims 1 to 9 or the method for selecting a multi-label model according to claim 10 is implemented.

Images